Image processing method and device, electronic equipment and computer readable storage medium

Abstract

The invention relates to an image processing method and device, electronic equipment, a storage medium and a computer program product. The method comprises the following steps: fusing a first image and at least one second image adjacent to the first image in time sequence to obtain a fused image; the boundary of the first image and the boundary of the at least one second image are both located in the fusion boundary of the fusion image; determining a target area in the fusion boundary based on inertial measurement data corresponding to the first image and the fusion boundary; and obtaining a target image according to the target area. By adopting the method, the image anti-shake accuracy can be improved.

Description

Technical Field

[0001] This application relates to camera technology, and particularly to an image processing method, apparatus, electronic device, and computer-readable storage medium. Background Art

[0002] When an electronic device such as a mobile phone takes a picture, there is usually external jitter to varying degrees, which affects the stability of the picture. Therefore, it is necessary to perform anti-shake during shooting to capture a clearer image. Existing anti-shake technologies usually use DIS (Electronic Image Stabilization) technology for anti-shake, that is, perform cropping and other processing on a single image to achieve anti-shake.

[0003] However, traditional image processing methods have the problem of low anti-shake accuracy. Summary of the Invention

[0004] Embodiments of this application provide an image processing method, apparatus, electronic device, computer-readable storage medium, and computer program product, which can improve the accuracy of image anti-shake.

[0005] In a first aspect, this application provides an image processing method. The method includes:

[0006] Fusing a first image and at least one second image adjacent to the first image in time sequence to obtain a fused image; the boundaries of the first image and the at least one second image are both located in the fusion boundary of the fused image;

[0007] Based on the inertial measurement data corresponding to the first image and the fusion boundary, determining a target area within the fusion boundary,

[0008] Obtaining a target image according to the target area.

[0009] In a second aspect, this application further provides an image processing apparatus. The apparatus includes:

[0010] An image fusion module, configured to fuse a first image and at least one second image adjacent to the first image in time sequence to obtain a fused image; the boundaries of the first image and the at least one second image are both located in the fusion boundary of the fused image;

[0011] A target area determination module, configured to determine a target area within the fusion boundary based on the inertial measurement data corresponding to the first image and the fusion boundary,

[0012] A target image determination module, configured to obtain a target image according to the target area.

[0013] In a third aspect, the present application also provides an electronic device. The electronic device includes a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0014] Fuse a first image and at least one second image that is temporally adjacent to the first image to obtain a fused image; the boundaries of the first image and the at least one second image are both located in the fusion boundary of the fused image;

[0015] Based on the inertial measurement data corresponding to the first image and the fusion boundary, determine a target area within the fusion boundary,

[0016] Obtain a target image according to the target area.

[0017] In a fourth aspect, the present application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0018] Fuse a first image and at least one second image that is temporally adjacent to the first image to obtain a fused image; the boundaries of the first image and the at least one second image are both located in the fusion boundary of the fused image;

[0019] Based on the inertial measurement data corresponding to the first image and the fusion boundary, determine a target area within the fusion boundary,

[0020] Obtain a target image according to the target area.

[0021] In a fifth aspect, the present application also provides a computer program product. The computer program product includes a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0022] Fuse a first image and at least one second image that is temporally adjacent to the first image to obtain a fused image; the boundaries of the first image and the at least one second image are both located in the fusion boundary of the fused image;

[0023] Based on the inertial measurement data corresponding to the first image and the fusion boundary, determine a target area within the fusion boundary,

[0024] Obtain a target image according to the target area.

[0025] The above image processing method, apparatus, electronic device, computer-readable storage medium, and computer program product fuse a first image and at least one second image temporally adjacent to the first image to obtain a fused image. The boundaries of the first image and at least one second image are both located in the fusion boundary of the fused image. That is, through image fusion in the time domain, the field of view range and image area in the spatial domain are broadened, and a fused image with a larger anti-shake range is obtained. Then, based on the inertial measurement data corresponding to the first image and the fusion boundary with a larger anti-shake range, the target area required for anti-shake can be more accurately determined within the fusion boundary. Furthermore, according to the target area of the fused image, a more accurate anti-shake target image can be obtained, improving the accuracy of anti-shake. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] To more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0027] Figure 1 It is a flowchart of an image processing method in an embodiment;

[0028] Figure 2 It is a schematic diagram of image fusion in an embodiment;

[0029] Figure 3 It is a schematic diagram of determining a target area within a fusion boundary in an embodiment;

[0030] Figure 4 It is a flowchart of an image processing method in another embodiment;

[0031] Figure 5 It is a flowchart of the step of determining a fusion boundary according to the boundary of the first image and the boundaries of at least one second image temporally adjacent to the first image in an embodiment;

[0032] Figure 6 It is a flowchart of the step of performing water ripple distortion correction on an intermediate image to obtain a first image in an embodiment;

[0033] Figure 7 It is a schematic diagram of distortion correction in an embodiment;

[0034] Figure 8 It is a system block diagram of image processing in an embodiment;

[0035] Figure 9 It is a structural block diagram of an image processing apparatus in an embodiment;

[0036] Figure 10 It is the internal structure diagram of an electronic device in an embodiment. Detailed implementation manners

[0037] In order to make the objectives, technical solutions and advantages of the present application clearer and more understandable, the present application will be further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0038] In one embodiment, as Figure 1 shown, an image processing method is provided. In this embodiment, an example is given where this method is applied to an electronic device. The electronic device may be a terminal or a server; it can be understood that this method can also be applied to a system including a terminal and a server and implemented through the interaction between the terminal and the server. Among them, the terminal may be, but is not limited to, various personal computers, laptop computers, smart phones, tablet computers, Internet of Things devices and portable wearable devices. The Internet of Things devices may be smart speakers, smart TVs, smart air conditioners, smart in-vehicle devices, smart cars, etc. The portable wearable devices may be smart watches, smart bracelets, head-mounted devices, etc. The server can be implemented by an independent server or a server cluster composed of multiple servers.

[0039] In this embodiment, the image processing method includes the following steps:

[0040] Step S102: Fuse a first image and at least one second image that is temporally adjacent to the first image to obtain a fused image; the boundary of the first image and the boundaries of at least one second image are both located in the fusion boundary of the fused image.

[0041] Among them, the first image is the image to be anti-shake, and the second image is the image adjacent to the first image in the time domain (temporal sequence) in the image stream.

[0042] Optionally, at least one second image adjacent to the first image includes at least one of the second image generated at a first time point and the second image generated at a second time point; the first time point is earlier than the generation time point of the first image, and the second time point is later than the generation time point of the first image. Among them, the first time point or the second time point is the time point when the image sensor generates the second image, and the generation time point of the first image is the time point when the image sensor generates the first image.

[0043] It can be understood that at least one second image including the second image generated at the first time point means that the second image generated at the first time point is the historical data of the first image; at least one second image including the second image generated at the second time point means that the second image generated at the second time point is the future data of the first image.

[0044] The fused image is an image obtained by fusing a first image and at least one second image. The fusion boundary is the boundary of the fused image. The boundaries of the first image and at least one second image are both located within the fusion boundary, that is, the area of the fused image is larger than the area of the first image or the second image. Part of the boundary of the first image is located on the fusion boundary, and the other part is located within the fusion boundary. Part of the boundary of at least one second image is located on the fusion boundary, and the other part is located within the fusion boundary. That is, the fusion boundary includes part of the boundary of the first image and part of the boundary of at least one second image.

[0045] Optionally, the image stream is an image stream in video mode or an image stream in preview mode. Each image is arranged in sequence in time series to form an image stream. The electronic device can obtain the first image and at least one second image adjacent to the first image in time series from the image stream.

[0046] Optionally, the electronic device determines the fusion weight of each second image, and based on the fusion weight of each second image, fuses the second image into the first image to obtain a fused image; wherein, the magnitude of the fusion weight of the second image is negatively correlated with the time domain distance; the time domain distance is the difference between the generation time points of the second image and the first image in the time domain.

[0047] Optionally, the electronic device aligns at least one second image adjacent to the first image to the first image; fuses the first image and the aligned at least one second image to obtain a fused image. Optionally, the electronic device determines the fusion weight of each aligned second image, and based on the fusion weight of each aligned second image, fuses the aligned second image into the first image to obtain a fused image.

[0048] Optionally, the electronic device registers each second image with the first image to obtain registration data; for each second image, based on the registration data corresponding to the second image, aligns the second image to the first image.

[0049] Optionally, for each aligned second image, the electronic device multiplies each pixel in the second image by the corresponding fusion weight to obtain a weighted second image; fuses each weighted second image into the first image to obtain a fused image.

[0050] Optionally, the electronic device performs image fusion on the first image and at least one second image adjacent to the first image using the following formula (1):

[0051] (1)

[0052] Wherein, is the fused image, is the fusion weight of the i-th frame image, is the i-th frame image, is an image set including the first image and each aligned second image.

[0053] Optionally, the electronic device performs distortion correction on a third image in the image stream to obtain the first image. Among them, the distortion correction includes lens distortion correction and moiré distortion correction.

[0054] Optionally, the electronic device obtains the first image and at least one second image adjacent to the first image from the image queue; fuses the first image and at least one second image adjacent to the first image to obtain a fused image.

[0055] Optionally, the electronic device obtains a third image through an image sensor, performs distortion correction on the third image to obtain the first image, and stores the first image in the image queue; if it is detected that a fourth image after distortion correction is stored in the image queue, the first image and at least one second image adjacent to the first image are obtained from the image queue; among them, the generation time point of the fourth image is later than the generation time point of the first image.

[0056] It can be understood that at least two images after distortion correction are stored in the image queue. To ensure the robustness of time-domain fusion of multiple frames of images for anti-shake, a semi-real-time processing method is adopted, that is, when the fourth image is stored in the image queue, the first image and at least one second image adjacent to the first image are obtained from the image queue.

[0057] Optionally, the electronic device uses the following formula (2) to obtain the first image and at least one second image adjacent to the first image from the image queue:

[0058] (2)

[0059] Among them, is an image set including the first image and at least one second image, is the i-th frame image in the image set, is the queue number of the first image, and the second images are M frames of images before and after the first image.

[0060] Optionally, the electronic device can also perform noise reduction processing on the image stream.

[0061] Step S104, based on the inertial measurement data corresponding to the first image and the fusion boundary, determine the target area within the fusion boundary.

[0062] Among them, the inertial measurement data is the information on the motion state and attitude of an object obtained by an Inertial Measurement Unit (IMU). The inertial measurement unit may include a gyroscope and an accelerometer. The three-axis attitude angular velocity (Gyro) is obtained through the gyroscope, and the translational acceleration (Acc) is obtained through the accelerometer. That is, the inertial measurement data includes translational acceleration and three-axis attitude angular velocity.

[0063] The target area is the area where the first image is located after image stabilization. The target area is within the fusion boundary.

[0064] Optionally, the electronic device determines the attitude of the camera module based on the inertial measurement data corresponding to the first image. The attitude includes the jitter direction and the jitter distance. Based on the attitude of the camera module, the image stabilization direction and the image stabilization distance of the first image are determined, and the area where the first image is located is adjusted according to the image stabilization direction and the image stabilization distance of the first image to obtain the target area.

[0065] Among them, the image stabilization direction of the first image is opposite to the jitter direction, and the image stabilization distance of the first image is less than or equal to the jitter distance.

[0066] Optionally, the electronic device determines the attitude of the camera module based on the inertial measurement data corresponding to the first image. The attitude includes the jitter direction and the jitter distance. In the current coordinate system, based on the attitude of the camera module, the image stabilization direction and the image stabilization distance of the fusion boundary are determined, the fusion boundary is adjusted according to the image stabilization direction and the image stabilization distance of the fusion boundary, and the area where the first image is located within the adjusted fusion boundary is used as the target area.

[0067] Among them, the image stabilization direction of the fusion boundary is the same as the jitter direction, and the image stabilization distance of the fusion boundary is less than or equal to the jitter distance.

[0068] As Figure 2 shown, the electronic device acquires the (N - 1)th frame, the Nth frame, and the (N + 1)th frame through an image sensor. The Nth frame image is the first image, and the (N - 1)th frame and the (N + 1)th frame are the second images adjacent to the Nth frame. The (N - 1)th frame, the Nth frame, and the (N + 1)th frame are fused to obtain a fused image. The fusion boundary in this fused image covers the boundaries of the (N - 1)th frame, the Nth frame, and the (N + 1)th frame, widening the cropping distances in the X direction (horizontal direction) and the Y direction (vertical direction). That is, the temporal information of the image stream is converted into spatial information. In Figure 2 it, the shooting object is a tree. Since jitter occurs, the tree is not included in the Nth frame image. Then, through temporal fusion of the (N - 1)th frame, the Nth frame, and the (N + 1)th frame to obtain a fused image, the jitter can be compensated for image stabilization. The target area can be determined in the fused image, and the target area contains the complete shooting object (the tree), improving the stability of the picture and the accuracy of image stabilization.

[0069] Optionally, the electronic device may also perform at least one of optical image stabilization (OIS) and digital image stabilization (DIS) on the first image to determine the target area.

[0070] Step S106, obtain a target image according to the target area.

[0071] Optionally, the electronic device obtains the image information where the target area is located from the fused image to obtain the target image.

[0072] The target image is the finally output image after performing anti-shake on the first image.

[0073] Optionally, the electronic device crops out the image information where the target area is located from the fused image to obtain the target image.

[0074] Optionally, the electronic device copies the image information where the target area is located from the fused image to obtain the target image.

[0075] Optionally, the electronic device obtains the image information where the target area is located from the fused image, and multiplies the image information by a target weight factor to obtain the target image. The target weight factor can be set as needed.

[0076] Optionally, the electronic device obtains the image information where the target area is located from the fused image, and renders the image information where the target area is located through a graphics processing unit (GPU) to obtain the target image.

[0077] In the above image processing method, the electronic device fuses the first image and at least one second image adjacent to the first image in time sequence to obtain a fused image. The boundary of the first image and the boundaries of at least one second image are both located in the fusion boundary of the fused image. That is, through image fusion in the time domain, the field of view range and image area in the spatial domain are broadened, and a fused image with a larger anti-shake range is obtained, enabling the anti-shake algorithm to have more anti-shake margins (cropping margins) during anti-shake; then, based on the inertial measurement data corresponding to the first image and the fusion boundary with a larger anti-shake range, the target area required for anti-shake can be more accurately determined within the fusion boundary, and then a more accurate anti-shake target image can be obtained according to the target area of the target image, improving the accuracy of anti-shake. Moreover, the field of view range of the fused image can be broadened on the premise of ensuring that the field of view angle output by the lens remains unchanged, thereby improving the anti-shake effect.

[0078] In one embodiment, determining a target area within a fusion boundary based on inertial measurement data and the fusion boundary corresponding to a first image includes: determining a target pose of the camera module based on the inertial measurement data and the fusion boundary corresponding to the first image; and determining the target area within the fusion boundary according to the target pose and the fusion boundary.

[0079] Wherein, the camera module is a module for taking pictures and includes multiple components. The camera module includes a lens, an image sensor, etc. The target pose of the camera module includes translation and rotation.

[0080] Optionally, determining the target pose of the camera module based on the inertial measurement data and the fusion boundary corresponding to the first image includes: determining an initial pose of the camera module based on the inertial measurement data corresponding to the first image; adjusting the initial pose based on the fusion boundary to determine the target pose of the camera module; the target pose is such that the boundary pixel points in the adjusted image are all located in the area included in the fusion boundary, and the similarity between the target pose and the initial pose is greater than a target similarity threshold, and the adjusted image is an image obtained by adjusting the first image with the target pose.

[0081] Wherein, the initial pose is a pose determined based on inertial measurement data. The boundary pixel points are the pixel points on the boundary in the first image. The target similarity threshold can be set as needed.

[0082] Optionally, the electronic device obtains the inertial measurement data corresponding to the first image through an inertial measurement unit, inputs the inertial measurement data corresponding to the first image into a pose filter, and determines the initial pose of the camera module through the pose filter. The initial pose includes the translation and rotation of the camera module. Wherein, the pose filter can be a pose filter of the inertial measurement unit or a pose filter based on image motion estimation.

[0083] It can be understood that the initial pose determined from the inertial measurement data may cause the image after anti-shake to cross the fusion boundary, that is, the image cropping may crop to an undefined area without pixels. Therefore, in order to improve the accuracy of anti-shake, the initial pose is adjusted to determine a more accurate target pose.

[0084] Optionally, the electronic device adjusts the initial pose to obtain an intermediate pose; the intermediate pose is such that the boundary pixel points in the first image adjusted with the intermediate pose are all located within the area included in the fusion boundary; for each intermediate pose, the intermediate pose and the initial pose are cross-multiplied to determine the cross-multiplication result; the minimum cross-multiplication result is determined from each cross-multiplication result, and the intermediate pose corresponding to the minimum cross-multiplication result is determined as the target pose. Wherein, the cross-multiplication result between the intermediate pose and the initial pose represents the similarity between the intermediate pose and the initial pose, and the cross-multiplication result and the similarity are negatively correlated.

[0085] The target pose ensures that the boundary pixel points in the adjusted image do not cross the fusion boundary, and the cross product result between the target pose and the initial pose is minimized. The cross product result between the target pose and the initial pose represents the similarity degree between the target pose and the initial pose. When the cross product result between the target pose and the initial pose is minimized, it can be considered that the similarity degree between the target pose and the initial pose is greater than the target similarity threshold.

[0086] Optionally, the electronic device determines the target pose of the camera module by using the following formula (3):

[0087] (3)

[0088] where is the initial pose, is the intermediate pose. When and the cross product result is minimized, is the target pose, and are both vectors; formula (3) needs to satisfy the constraint condition The constraint function is an out-of-bounds detection function. When the point is out of bounds, it outputs 1, otherwise it outputs 0; is the boundary pixel point of the first image, refers to the set of boundary pixel points of the first image; is the homography matrix of the rotation matrix transformed from , and is obtained through formula (4):

[0089] (4)

[0090] where is the transformed rotation matrix, is the homography matrix of is the camera internal parameter.

[0091] In another optional implementation manner, the electronic device randomly determines the cross product results less than the cross product threshold from each cross product result, and determines the intermediate pose corresponding to the cross product result less than the cross product threshold as the target pose. The cross product result corresponding to the target pose is less than the cross product threshold, indicating that the similarity degree between the target pose and the initial pose is greater than the target similarity threshold.

[0092] In other optional implementation manners, the electronic device can also use other methods to determine the target pose, which is not limited herein.

[0093] Optionally, according to the target pose and the fusion boundary, a target area is determined within the fusion boundary, including: determining an anti-shake direction and an anti-shake distance according to the target pose; the anti-shake direction and the anti-shake distance are such that the target area does not exceed the fusion boundary; moving the area where the first image is located according to the anti-shake direction and the anti-shake distance to obtain the target area.

[0094] Optionally, the electronic device may also use other methods to determine the target area, which is not limited herein.

[0095] In this embodiment, the electronic device determines the target pose of the camera module based on the inertial measurement data corresponding to the first image and the fusion boundary. According to the target pose and the fusion boundary, the target area can be more accurately determined within the fusion boundary. Further, the electronic device determines the initial pose of the camera module based on the inertial measurement data corresponding to the first image, and then adjusts the initial pose based on the fusion boundary to more accurately determine the target pose of the camera module. The target pose is such that the boundary pixel points in the adjusted image are all located within the area included in the fusion boundary, and the similarity between the target pose and the initial pose is greater than the target similarity threshold. That is, the adjusted target pose not only makes the boundary pixel points of the first image not cross the fusion boundary, but also is closer to the initial pose, and can more accurately perform anti-shake on the first image.

[0096] In one embodiment, according to the target pose and the fusion boundary, determining a target area within the fusion boundary includes: determining a first grid according to the fusion boundary; the fusion boundary is located within the area included in the first grid; adjusting the first grid based on the target pose to obtain a second grid; in the coordinate system of the second grid, determining the area where the first image is located within the fusion boundary as the target area.

[0097] Wherein, the first grid (mesh) can be a rectangular grid, a circular grid or a grid of other shapes, which is not limited herein. The second grid is the grid obtained after adjusting the first grid.

[0098] Optionally, the electronic device determines the area where the fusion boundary is located and divides the area where the fusion boundary is located into grids to obtain the first grid.

[0099] Optionally, determining the first grid according to the fusion boundary includes: determining the circumscribed rectangle of the fusion boundary; dividing the circumscribed rectangle into grids to obtain the first grid.

[0100] Wherein, the circumscribed rectangle is the smallest rectangle formed outside the fusion boundary.

[0101] Optionally, the electronic device divides the circumscribed rectangle into grids to obtain a uniformly divided first grid.

[0102] Optionally, the electronic device projects the first grid onto the output space where the target image is to be generated by using the target pose, and obtains a second grid; in the coordinate system of the second grid, the area where the first image is located is determined from the fusion boundary as the target area.

[0103] It can be understood that if the fusion boundary is located in the area included in the first grid, when the electronic device adjusts the first grid based on the target pose, the fusion boundary is adjusted simultaneously.

[0104] Optionally, the electronic device uses the target area as the target grid, and obtains the image information where the target grid is located from the fusion image for rendering to obtain the target image.

[0105] As Figure 3 shown, the electronic device determines the fusion boundary according to the (N-1)th frame, the Nth frame, and the (N+1)th frame, determines the circumscribed rectangle of the fusion boundary, performs grid division on the circumscribed rectangle to obtain the first grid; adjusts the first grid based on the target pose to obtain the second grid; in the coordinate system of the second grid, the target area where the first image is located is determined from within the fusion boundary as the target grid.

[0106] In this embodiment, the electronic device determines the first grid according to the fusion boundary, and the fusion boundary is located in the area included in the first grid; by adjusting the first grid based on the target pose, a more accurate second grid can be obtained; then, in the coordinate system of the second grid, the area where the first image is located can be used as the target area to more accurately determine the target area. Among them, the electronic device determines the circumscribed rectangle of the fusion boundary and performs grid division on the circumscribed rectangle to obtain an accurate first grid.

[0107] In one embodiment, before determining the target area within the fusion boundary based on the inertial measurement data corresponding to the first image and the fusion boundary, it further includes: determining the fusion boundary according to the boundary of the first image and the boundaries of at least one second image temporally adjacent to the first image.

[0108] Optionally, the electronic device obtains the boundary pixel points of the first image and the boundary pixel points of at least one second image adjacent to the first image; determines the fusion boundary according to the boundary pixel points of the first image and the boundary pixel points of each second image.

[0109] Optionally, the electronic device obtains the coordinates of the boundary pixel points of the first image and the coordinates of the boundary pixel points of at least one second image adjacent to the first image; determines the fusion boundary according to the coordinates of the boundary pixel points of the first image and the coordinates of the boundary pixel points of each second image.

[0110] It can be understood that considering that the actual image information is not required to be used when determining the target area subsequently, the fused image is not rendered, and the boundary pixels of the first image and the boundary pixels of each second image are obtained to determine the fusion boundary, which can save computer resources and determine the fusion boundary more quickly, so as to determine the target area more quickly.

[0111] In another embodiment, as Figure 4 shown, there is also provided an image processing method applied to an electronic device, and the image processing method includes the following steps:

[0112] Step S402, determine a fusion boundary according to the boundary of the first image and the boundaries of at least one second image that is temporally adjacent to the first image.

[0113] Step S404, based on the inertial measurement data corresponding to the first image and the fusion boundary, determine a target area within the fusion boundary.

[0114] Step S406, fuse the first image and at least one second image that is temporally adjacent to the first image to obtain a fused image; the boundaries of the first image and the boundaries of the at least one second image are both located in the fusion boundary of the fused image.

[0115] Step S408, obtain a target image according to the target area.

[0116] In this embodiment, the electronic device avoids the problem of slow image processing speed caused by first fusing and rendering the first image and the second image. That is, the fusion boundary is determined first, the target area can be quickly determined through this fusion boundary, then the first image and at least one second image adjacent to the first image are fused to obtain a fused image, and the target image is obtained from the fused image, which can improve the overall efficiency of image anti-shake.

[0117] In one embodiment, as Figure 5 shown, determining the fusion boundary according to the boundary of the first image and the boundaries of at least one second image that is temporally adjacent to the first image includes:

[0118] Step S502, align at least one second image that is temporally adjacent to the first image with the first image one by one.

[0119] Optionally, among at least one second image adjacent to the first image, for each second image, the electronic device registers the second image and the first image to determine the registration data of the second image; for each second image, the second image is aligned to the first image based on the registration data of the second image. Among them, the registration data may be a registration matrix (homography matrix).

[0120] Optionally, the electronic device uses a feature matching algorithm to register the second image with the first image and determine the registration data of the second image. Among them, the electronic device can use a feature matching algorithm, such as matching through inter-frame corner points or matching inter-frame points through optical flow method, etc., which is not limited here. The electronic device can select a feature extraction algorithm according to accuracy and performance, such as FAST feature extraction algorithm, ORB feature extraction algorithm, Harris feature extraction algorithm, etc., and select a suitable corner point extraction algorithm according to specific engineering requirements.

[0121] Optionally, the electronic device uses the following formula (5) to determine the registration data between the second image and the first image:

[0122] (5)

[0123] Among them, is the registration data set, and the registration data is a registration matrix (homography matrix), is the registration data corresponding to the th image, is the confidence of the registration data corresponding to the th image, is the queue number of the first image.

[0124] Optionally, the electronic device uses the matching point pairs between the second image and the first image to find the least squares solution; according to the reprojection error between the second image and the first image, calculate the confidence of the registration data corresponding to this second image. Among them, the electronic device encodes and normalizes the reprojection error, represents it as a decimal between 0 and 1, and obtains the confidence of the registration data corresponding to the second image.

[0125] Step S504, obtain the boundary of the first image and the boundary of each aligned second image.

[0126] Step S506, fuse the boundary of the first image and the boundaries of each aligned second image to determine the fused boundary.

[0127] Optionally, the electronic device combines the boundary of the first image and the boundaries of each aligned second image to determine the fused boundary.

[0128] Optionally, the electronic device obtains the boundary pixel points included in the boundary of the first image and the boundary pixel points included in the boundary of each aligned second image; combines the boundary pixel points included in the boundary of the first image and the boundary pixel points included in the boundary of each aligned second image to determine the fused boundary.

[0129] Optionally, the electronic device samples the boundary of the first image to obtain the boundary pixel points of the first image, and samples the boundaries of the aligned second images to obtain the boundary pixel points of the second images; the boundary pixel points of each first image and the boundary pixel points of the second images are fused to determine the fused boundary.

[0130] Optionally, the electronic device fuses the boundary pixel points of each first image and the boundary pixel points of the second images by using the following formulas (6) and (7):

[0131] (6)

[0132] (7)

[0133] Wherein, is the registration data corresponding to the th image, represents the boundary pixel points of the i-th image, is the boundary pixel points of the th image after registration, is the set of boundary pixel points, is the confidence of the registration data corresponding to the th image, is the queue number of the first image.

[0134] In this embodiment, the electronic device aligns at least one second image adjacent to the first image in time sequence with the first image one by one. Then, by fusing the boundary of the first image and the boundaries of the aligned second images, the fused boundary can be accurately determined.

[0135] In one embodiment, fusing the boundary of the first image and the boundaries of the aligned second images to determine the fused boundary includes: fusing the boundary of the first image and the boundaries of the aligned second images to obtain an intermediate boundary; the intermediate boundary includes each first boundary point; screening out second boundary points from each first boundary point; determining the fused boundary based on each second boundary point.

[0136] Optionally, the electronic device fuses the boundary pixel points of the first image and the boundary pixel points of the aligned second images to obtain an intermediate boundary; the intermediate boundary includes each first boundary point, and each first boundary point includes the boundary pixel points of the first image and the boundary pixel points of the second image.

[0137] It can be understood that since both the accuracy of the registration data and the movement amplitude of the camera module will affect the calculated first boundary points, it is necessary to screen the calculated first boundary points to obtain more accurate second boundary points.

[0138] Optionally, for each first boundary point, the electronic device detects the coordinates of the first boundary point; if the coordinates of the first boundary point are within the target coordinate range, the first boundary point is screened as a second boundary point; if the coordinates of the first boundary point are not within the target coordinate range, the first boundary point is discarded. The target coordinate range can be set as needed.

[0139] Optionally, for each first boundary point, the electronic device detects the confidence level of the registration data corresponding to the image to which the first boundary point belongs; if the confidence level is greater than the confidence level threshold, the first boundary point is screened as a second boundary point; if the confidence level is less than or equal to the confidence level threshold, the first boundary point is discarded. The confidence level threshold can be set as needed.

[0140] Optionally, screening out second boundary points from each of the first boundary points includes: for each first boundary point, if the first boundary point meets the target condition, the first boundary point is screened as a second boundary point; the target condition includes at least one of the confidence level of the registration data corresponding to the image to which the first boundary point belongs being greater than the confidence level threshold, there being a boundary intersection between the image to which the first boundary point belongs and an adjacent image, and there being a boundary intersection between the image to which the first boundary point belongs and the first image; the registration data corresponding to the image to which the first boundary point belongs is obtained by registering the image to which the first boundary point belongs with the first image.

[0141] Among them, the confidence level threshold can be set as needed.

[0142] It can be understood that the confidence level of the registration data corresponding to the image to which the first boundary point belongs being greater than the confidence level threshold can ensure that the registration data between the image to which the first boundary point belongs and the first image is highly credible; there being a boundary intersection between the image to which the first boundary point belongs and an adjacent image can ensure that the image to which the first boundary point belongs and the adjacent image are coherent; there being a boundary intersection between the image to which the first boundary point belongs and the first image, that is, the boundaries of the image to which the first boundary point belongs and the first image are connected, can ensure that the image to which the first boundary point belongs and the first image are related, thereby more accurately screening out the second boundary points.

[0143] Optionally, the second image adjacent to the first image includes at least one of the second image generated at the first time point and the second image generated at the second time point, where the first time point is earlier than the generation time point of the first image, and the second time point is later than the generation time point of the first image; there is a boundary intersection between the image to which the first boundary point belongs and the adjacent image, including: if the image to which the first boundary point belongs is the second image generated at the first time point, there is a boundary intersection between the image to which the first boundary point belongs and the adjacent image, and the generation time point of the adjacent image is later than the first time point; if the image to which the first boundary point belongs is the second image generated at the second time point, there is a boundary intersection between the image to which the first boundary point belongs and the adjacent image, and the generation time point of the adjacent image is earlier than the second time point.

[0144] It can be understood that the first time point is earlier than the generation time point of the first image. If the image to which the first boundary point belongs is the second image generated at the first time point, that is, the second image generated at the first time point is the historical image of the first image, then the adjacent image with a generation time point later than the first time point is determined, so that there is a boundary intersection between the adjacent image and the image to which the first boundary point belongs, that is, the adjacent image is closer to the first image in the time domain, and the second time point closer to the first image can be more accurately screened out.

[0145] Similarly, the second time point is later than the generation time point of the first image. If the image to which the first boundary point belongs is the second image generated at the second time point, that is, the second image generated at the second time point is the future image of the first image, then the adjacent image with a generation time point earlier than the second time point is determined, so that there is a boundary intersection between the adjacent image and the image to which the first boundary point belongs, that is, the adjacent image is closer to the first image in the time domain, and the second time point closer to the first image can be more accurately screened out.

[0146] Optionally, the electronic device uses the following formula (8) to screen out the second boundary point:

[0147] (8)

[0148] Where, is the set of second boundary points, is the second boundary point of the j-th frame image, is the set of first boundary points, is the confidence level of the registration data corresponding to the j-th frame image, is the confidence level threshold; indicates that if the j-th frame image is the future data of the first image, there is a boundary intersection between the boundary points of the j-th frame image and the boundary points of the (j + 1)-th frame image, and the generation time point of the (j + 1)-th frame image is earlier than the generation time point of the j-th frame image; It means that when the j-th frame image is the historical data of the first image, there is a boundary intersection between the boundary points of the j-th frame image and the boundary points of the (j - 1)-th frame image, and the generation time point of the (j - 1)-th frame image is later than the generation time point of the j-th frame image; is the queue number of the first image.

[0149] In this embodiment, the electronic device fuses the boundary of the first image and the boundaries of each aligned second image to obtain an intermediate boundary; and screens out more accurate second boundary points from each first boundary point in the intermediate boundary, so as to more accurately determine the fusion boundary. Further, the electronic device detects whether the first boundary point meets the target condition, which can further screen out more accurate second boundary points.

[0150] In one embodiment, obtaining the first image includes: obtaining a third image in the image stream; performing lens distortion correction on the third image to obtain an intermediate image; and performing moiré distortion correction on the intermediate image to obtain the first image.

[0151] It can be understood that both lens distortion correction and moiré distortion correction are non-linear distortions. Non-linear distortion refers to the distortion that is not uniformly distributed in the image, and the degree of distortion is related to the spatial position of the image and does not have global consistency. The distortion in the image stream is divided into two types. The first is the inherent distortion brought by the camera module (lens), which is fixed and determined by the lens design and production process; the second is the moiré distortion (generally called the jelly effect) caused by the movement of the camera module and the CMOS (Complementary Metal Oxide Semiconductor) rolling shutter effect. This distortion is caused by the motion state and is a dynamic distortion.

[0152] Optionally, the electronic device generates a third image through an image sensor and performs lens distortion correction on the third image to obtain an intermediate image. Among them, the intermediate image refers to the image after removing the lens distortion.

[0153] Optionally, the third image is an image in a two-dimensional space; performing lens distortion correction on the third image to obtain an intermediate image includes: obtaining the internal lens parameters and the lens distortion parameters; and based on the internal lens parameters and the lens distortion parameters, back-projecting the third image belonging to the two-dimensional space into a three-dimensional space to obtain an intermediate image.

[0154] Among them, the internal lens parameters are the internal parameters of the lens itself, which are determined by factors such as the material and hardware of the lens. The lens distortion parameters are calibrated when the lens leaves the factory. Both the internal lens parameters and the lens distortion parameters are obtained by offline calibration of the lens.

[0155] Optionally, the electronic device uses the following formula (9) to remove the lens distortion in the third image:

[0156] (9)

[0157] Among them, The projection point in the three-dimensional space (3D space), that is, the pixel coordinates of the intermediate image in the three-dimensional space, is the pixel coordinates in the image coordinate system (two-dimensional space); is the back-projection function, which transforms the pixel coordinates in the two-dimensional space to the pixel coordinates in the three-dimensional space, and are projection parameters, is the internal camera parameter, is the lens distortion parameter.

[0158] In this embodiment, the electronic device acquires the third image in the image stream, and respectively performs lens distortion correction and moiré distortion correction on the third image, so as to obtain a more accurate first image after distortion correction. Further, based on the internal camera parameter and the lens distortion parameter, the electronic device back-projects the third image belonging to the two-dimensional space into the three-dimensional space, and an intermediate image without lens distortion can be obtained.

[0159] In one embodiment, as Figure 6 shown, performing moiré distortion correction on the intermediate image to obtain the first image includes:

[0160] Step S602, determining a reference pixel row from the intermediate image.

[0161] Among them, the reference pixel row is a reference for registering other pixel rows. The intermediate image is an image in the three-dimensional space.

[0162] Optionally, the electronic device determines the first pixel row in the intermediate image as the reference pixel row.

[0163] Optionally, the electronic device randomly determines a pixel row in the intermediate image as the reference pixel row.

[0164] Step S604, for each first pixel row other than the reference pixel row, aligning the first pixel row to the reference pixel row to obtain an aligned second pixel row.

[0165] Among them, the first pixel row is the pixel row in the intermediate image other than the reference pixel row. The second pixel row is the pixel row after the first pixel row is aligned to the reference pixel row.

[0166] Optionally, for each first pixel row other than the reference pixel row, aligning the first pixel row to the reference pixel row to obtain an aligned second pixel row, including: obtaining the pose corresponding to the reference pixel row and the pose corresponding to each first pixel row other than the reference pixel row from the intermediate image in the three-dimensional space; for each first pixel row, determining the pose difference between the pose corresponding to the first pixel row and the pose corresponding to the reference pixel row; and based on the pose difference corresponding to the first pixel row, aligning the first pixel row to the reference pixel row to obtain an aligned second pixel row.

[0167] Wherein, the pose corresponding to the reference pixel row and the pose corresponding to each first pixel row other than the reference pixel row are both represented by quaternions.

[0168] It can be understood that the image is exposed row by row, so the pose corresponding to each row can be obtained through the inertial measurement unit. In the three-dimensional coordinate system (three-dimensional space), the pose corresponding to the reference pixel row of the intermediate image and the pose corresponding to each first pixel row other than the reference pixel row are obtained through the inertial measurement unit.

[0169] Optionally, the electronic device aligns the pose corresponding to the first pixel row to the pose corresponding to the reference pixel row using the following formula (10):

[0170] (10)

[0171] Wherein, is the projection point in the three-dimensional space (3D space), that is, the pixel point coordinates of the intermediate image in the three-dimensional space, is the aligned projection point, is the pose difference, is the conjugate of.

[0172] Step S606, generating a first image based on the reference pixel row and each aligned second pixel row.

[0173] Optionally, the electronic device combines the reference pixel row and each aligned second pixel row to generate a first image. The first image removes lens distortion and moiré distortion.

[0174] In this embodiment, the electronic device aligns the first pixel row in the intermediate image to the reference pixel row to obtain an aligned second pixel row, thereby generating a first image after moiré distortion correction. Further, the electronic device determines the pose difference between the pose corresponding to the first pixel row and the pose corresponding to the reference pixel row, and based on this pose difference, aligns the first pixel row to the reference pixel row, which can accurately remove the moiré distortion caused by movement and obtain an aligned second pixel row.

[0175] In one embodiment, performing moire distortion correction on an intermediate image to obtain a first image includes: performing moire distortion correction on the intermediate image to obtain a distortion-corrected image in three-dimensional space; projecting the distortion-corrected image in three-dimensional space onto a two-dimensional space to obtain a third grid; and obtaining the first image based on a third image and the third grid.

[0176] Wherein, the distortion-corrected image is an image that removes lens distortion and moire distortion and belongs to three-dimensional space.

[0177] Optionally, the electronic device projects pixel points in the distortion-corrected image in three-dimensional space through a projection function onto a two-dimensional space to obtain a third grid; determines an area where the third grid is located from the third image, and renders the area where the third grid is located to obtain the first image.

[0178] Optionally, the electronic device projects the distortion-corrected image in three-dimensional space onto a two-dimensional space by using the following formula (11):

[0179] (11)

[0180] Wherein, is a projection point of the distortion-corrected image in three-dimensional space, is a pixel point of the third grid in two-dimensional space, and are projection parameters, is the internal lens parameter, is the lens distortion parameter, and , is the projection function.

[0181] During the projection process, target scale transformation is performed, and a third grid with the target scale is output. The electronic device scales the focallength in the internal lens parameter, which can achieve target scale transformation during the projection process.

[0182] In this embodiment, the electronic device performs moire distortion correction on the intermediate image to obtain a distortion-corrected image in three-dimensional space, projects the distortion-corrected image in three-dimensional space back onto a two-dimensional space to obtain a third grid. Then, based on the third image and the third grid, a first image in two-dimensional space and without distortion can be accurately obtained.

[0183] In one embodiment, as Figure 7 shown, through the back-projection function Back-project the third image belonging to the two-dimensional space into the three-dimensional space to obtain an intermediate image in the three-dimensional space with lens distortion correction; from the intermediate image in the three-dimensional space, through the exposure time of each pixel row of the image sensor, index the attitude corresponding to the reference pixel row and the attitude corresponding to each first pixel row except the reference pixel row from the inertial measurement unit; for each first pixel row, determine the attitude difference between the attitude corresponding to the first pixel row and the attitude corresponding to the reference pixel row, and based on this attitude difference, align the first pixel row to the reference pixel row to obtain the aligned second pixel row; further, based on the reference pixel row and each aligned second pixel row, generate a distortion-corrected image in the three-dimensional space; through the projection function Project the distortion-corrected image in the three-dimensional space into the two-dimensional space to obtain a distortion-free third grid.

[0184] In one embodiment, as Figure 8 shown is a system block diagram of image processing. The image processing system includes a non-linear distortion correction module, an attitude filtering and optimization module, an image fusion module, and a rendering module; among them, the non-linear distortion correction module is responsible for correcting lens distortion and the water ripple distortion caused by movement, obtaining a third grid without lens distortion and water ripple distortion, and rendering the Nth frame image through the third grid to obtain the Nth frame image without distortion, and storing the Nth frame image without distortion in the image queue; the image fusion module is used to obtain the dequeued first image and at least one second image adjacent to the first image from the image queue, perform boundary extraction according to the first image and at least one second image adjacent to the first image to obtain a fusion boundary, and fuse the first image and at least one second image adjacent to the first image to obtain a fused image; the attitude filtering module and the optimization module filter the attitude of the camera module based on inertial measurement data to generate an initial attitude, and then optimize the initial attitude based on the fusion boundary to obtain a stable, smooth and target attitude that meets the cropping range; the rendering module mainly renders based on the calculated target attitude and the fused image to generate a final target image.

[0185] In one embodiment, there is also provided an image processing method applied to an electronic device. The image processing method includes the following steps,

[0186] Step A1, obtain the third image in the image stream.

[0187] Step A2, obtain the internal parameters of the lens and the lens distortion parameters; based on the internal parameters of the lens and the lens distortion parameters, back-project the third image belonging to the two-dimensional space into the three-dimensional space to obtain an intermediate image.

[0188] Step A3: Determine a reference pixel row from the intermediate image in the three-dimensional space, obtain the pose corresponding to the reference pixel row and the pose corresponding to each first pixel row except the reference pixel row; for each first pixel row, determine the pose difference between the pose corresponding to the first pixel row and the pose corresponding to the reference pixel row; based on the pose difference corresponding to the first pixel row, align the first pixel row to the reference pixel row to obtain an aligned second pixel row; based on the reference pixel row and each aligned second pixel row, obtain the distortion-corrected image of the three-dimensional space.

[0189] Step A4: Project the distortion-corrected image of the three-dimensional space onto the two-dimensional space to obtain a third grid; based on the third image and the third grid, obtain the first image.

[0190] Step A5: Align at least one second image temporally adjacent to the first image with the first image one by one.

[0191] Step A6: Obtain the boundary of the first image and the boundary of each aligned second image; fuse the boundary of the first image and the boundaries of each aligned second image to obtain an intermediate boundary; the intermediate boundary includes each first boundary point.

[0192] Step A7: For each first boundary point, if the first boundary point meets the target condition, then screen the first boundary point as a second boundary point; the target condition includes that the confidence of the registration data corresponding to the image to which the first boundary point belongs is greater than the confidence threshold, there is a boundary intersection between the image to which the first boundary point belongs and the adjacent image, and there is at least one of the boundary intersections between the image to which the first boundary point belongs and the first image, and the registration data corresponding to the image to which the first boundary point belongs is obtained by registering the image to which the first boundary point belongs with the first image; wherein, the second image adjacent to the first image includes at least one of the second image generated at the first time point and the second image generated at the second time point, the first time point is earlier than the generation time point of the first image, and the second time point is later than the generation time point of the first image; there is a boundary intersection between the image to which the first boundary point belongs and the adjacent image, including: if the image to which the first boundary point belongs is the second image generated at the first time point, then there is a boundary intersection between the image to which the first boundary point belongs and the adjacent image, and the generation time point of the adjacent image is later than the first time point; if the image to which the first boundary point belongs is the second image generated at the second time point, then there is a boundary intersection between the image to which the first boundary point belongs and the adjacent image, and the generation time point of the adjacent image is earlier than the second time point.

[0193] Step A8: Determine the fusion boundary based on each second boundary point.

[0194] Step A9: Determine the initial attitude of the camera module based on the inertial measurement data corresponding to the first image; adjust the initial attitude based on the fusion boundary to determine the target attitude of the camera module. The target attitude is such that the boundary pixels in the adjusted image are all located in the area included in the fusion boundary, and the similarity between the target attitude and the initial attitude is greater than the target similarity threshold. The adjusted image is the image obtained by adjusting the first image with the target attitude.

[0195] Step A10: Determine the circumscribed rectangle of the fusion boundary; perform grid division on the circumscribed rectangle to obtain the first grid. The fusion boundary is located in the area included in the first grid; adjust the first grid based on the target attitude to obtain the second grid; determine the area where the first image is located as the target area in the coordinate system of the second grid.

[0196] Step A11: Fuse the first image and at least one second image adjacent to the first image to obtain a fused image. The boundaries of the first image and at least one second image are both located in the fusion boundary of the fused image.

[0197] Step A12: Obtain the image information of the target area from the fused image to obtain the target image.

[0198] It should be understood that although the steps in the flowcharts involved in the above-described embodiments are shown in sequence according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless there is a clear indication in this article, the execution of these steps has no strict order limit, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-described embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be executed alternately or in turn with at least a part of other steps or steps or stages in other steps.

[0199] Based on the same inventive concept, the embodiments of the present application also provide an image processing apparatus for implementing the above-mentioned image processing method. The implementation solutions provided by this apparatus to solve problems are similar to the implementation solutions described in the above method. Therefore, the specific limitations in one or more of the following embodiments of the image processing apparatus can refer to the limitations on the image processing method in the above text, and will not be repeated here.

[0200] In one embodiment, as Figure 9 shown, an image processing apparatus is provided, including: an image fusion module 902, a target area determination module 904, and a target image determination module 906, where:

[0201] An image fusion module 902, configured to fuse a first image and at least one second image that is adjacent to the first image in time sequence to obtain a fused image; the boundaries of the first image and the at least one second image are both located in the fusion boundary of the fused image.

[0202] A target area determination module 904, configured to determine a target area within the fusion boundary based on the inertial measurement data corresponding to the first image and the fusion boundary.

[0203] A target image determination module 906, configured to obtain a target image according to the target area.

[0204] For the above image processing device, the electronic device fuses a first image and at least one second image that is adjacent to the first image in time sequence to obtain a fused image. The boundaries of the first image and the at least one second image are both located in the fusion boundary of the fused image, that is, through image fusion in the time domain, the field of view angle and image area in the spatial domain are broadened, and a fused image with a larger cropping range is obtained. Then, based on the inertial measurement data corresponding to the first image and the fusion boundary with a larger field of view angle, the target area required for anti-shake can be more accurately determined within the fusion boundary, and then a more accurate anti-shake target image can be obtained according to the target area, improving the accuracy of anti-shake.

[0205] In one embodiment, the above target area determination module 904 is further configured to determine the target pose of the camera module based on the inertial measurement data corresponding to the first image and the fusion boundary; and determine the target area within the fusion boundary according to the target pose and the fusion boundary.

[0206] In one embodiment, the above target area determination module 904 is further configured to determine the initial pose of the camera module based on the inertial measurement data corresponding to the first image; adjust the initial pose based on the fusion boundary to determine the target pose of the camera module; the target pose enables the boundary pixel points in the adjusted image to be all located in the area included in the fusion boundary, and the similarity degree between the target pose and the initial pose is greater than the target similarity threshold, and the adjusted image is the image obtained by adjusting the first image with the target pose.

[0207] In one embodiment, the above target area determination module 904 is further configured to determine a first grid according to the fusion boundary; the fusion boundary is located in the area included in the first grid; adjust the first grid based on the target pose to obtain a second grid; and determine the area where the first image is located within the fusion boundary as the target area in the coordinate system of the second grid.

[0208] In one embodiment, the above target area determination module 904 is further configured to determine the circumscribed rectangle of the fusion boundary; and perform grid division on the circumscribed rectangle to obtain a first grid.

[0209] In one embodiment, the above-mentioned device further includes a fusion boundary determination module; the fusion boundary determination module is used to determine a fusion boundary according to the boundary of the first image and the boundaries of at least one second image that is temporally adjacent to the first image.

[0210] In one embodiment, the above-mentioned fusion boundary determination module is further used to align at least one second image that is temporally adjacent to the first image with the first image one by one; obtain the boundary of the first image and the boundaries of each aligned second image; fuse the boundary of the first image and the boundaries of each aligned second image to determine the fusion boundary.

[0211] In one embodiment, the above-mentioned fusion boundary determination module is further used to fuse the boundary of the first image and the boundaries of each aligned second image to obtain an intermediate boundary; the intermediate boundary includes each first boundary point; screen out second boundary points from each first boundary point; determine the fusion boundary based on each second boundary point.

[0212] In one embodiment, the above-mentioned fusion boundary determination module is further used to, for each first boundary point, if the first boundary point meets the target condition, screen the first boundary point as a second boundary point; the target condition includes that the confidence of the registration data corresponding to the image to which the first boundary point belongs is greater than the confidence threshold, there is a boundary intersection between the image to which the first boundary point belongs and the adjacent image, and there is at least one of the boundary intersections between the image to which the first boundary point belongs and the first image; the registration data corresponding to the image to which the first boundary point belongs is obtained by registering the image to which the first boundary point belongs with the first image.

[0213] In one embodiment, the second image adjacent to the first image includes at least one of the second image generated at the first time point and the second image generated at the second time point, the first time point is earlier than the generation time point of the first image, and the second time point is later than the generation time point of the first image; there is a boundary intersection between the image to which the first boundary point belongs and the adjacent image, including: if the image to which the first boundary point belongs is the second image generated at the first time point, there is a boundary intersection between the image to which the first boundary point belongs and the adjacent image, and the generation time point of the adjacent image is later than the first time point; if the image to which the first boundary point belongs is the second image generated at the second time point, there is a boundary intersection between the image to which the first boundary point belongs and the adjacent image, and the generation time point of the adjacent image is earlier than the second time point.

[0214] In one embodiment, the above-mentioned device further includes a distortion correction module; the distortion correction module is used to obtain a third image in the image stream; perform lens distortion correction on the third image to obtain an intermediate image; perform water ripple distortion correction on the intermediate image to obtain the first image.

[0215] In one embodiment, the third image is an image in a two-dimensional space; the above-mentioned distortion correction module is further configured to obtain the internal camera parameters and the lens distortion parameters; based on the internal camera parameters and the lens distortion parameters, back-project the third image belonging to the two-dimensional space into a three-dimensional space to obtain an intermediate image.

[0216] In one embodiment, the above-mentioned distortion correction module is further configured to determine a reference pixel row from the intermediate image; for each first pixel row except the reference pixel row, align the first pixel row to the reference pixel row to obtain an aligned second pixel row; based on the reference pixel row and each aligned second pixel row, generate a first image.

[0217] In one embodiment, the above-mentioned distortion correction module is further configured to obtain the pose corresponding to the reference pixel row and the pose corresponding to each first pixel row except the reference pixel row from the intermediate image in the three-dimensional space; for each first pixel row, determine the pose difference between the pose corresponding to the first pixel row and the pose corresponding to the reference pixel row; based on the pose difference corresponding to the first pixel row, align the first pixel row to the reference pixel row to obtain an aligned second pixel row.

[0218] In one embodiment, the above-mentioned distortion correction module is further configured to perform moiré distortion correction on the intermediate image to obtain a distortion-corrected image in the three-dimensional space; project the distortion-corrected image in the three-dimensional space onto the two-dimensional space to obtain a third grid; based on the third image and the third grid, obtain a first image.

[0219] Each module in the above-mentioned image processing device can be implemented in whole or in part by software, hardware, and their combination. The above-mentioned modules can be embedded in the processor in the electronic device in hardware form or be independent of the processor, or can be stored in the memory in the electronic device in software form, so that the processor can call and execute the operations corresponding to the above-mentioned modules.

[0220] In one embodiment, an electronic device is provided. The electronic device can be a terminal, and its internal structure diagram can be as Figure 10As shown in the figure. The electronic device includes a processor, a memory, an input / output interface, a communication interface, a display unit, and an input device. Among them, the processor, the memory, and the input / output interface are connected through a system bus, and the communication interface, the display unit, and the input device are connected to the system bus through the input / output interface. Among them, the processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the electronic device is used to exchange information between the processor and external devices. The communication interface of the electronic device is used to communicate with external terminals in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, NFC (Near Field Communication), or other technologies. The computer program, when executed by the processor, implements an image processing method. The display unit of the electronic device is used to form a visually visible picture, which can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen. The input device of the electronic device can be a touch layer covering the display screen, or a button, a trackball, or a touchpad provided on the housing of the electronic device, or an external keyboard, touchpad, or mouse, etc.

[0221] Those skilled in the art can understand that Figure 10 the structure shown in the figure is only a block diagram of some structures related to the solution of this application, and does not constitute a limitation on the electronic device to which the solution of this application is applied. The specific electronic device may include more or fewer components than those shown in the figure, or combine some components, or have a different component layout.

[0222] The embodiment of this application also provides a computer-readable storage medium. One or more non-volatile computer-readable storage media containing computer-executable instructions, when the computer-executable instructions are executed by one or more processors, cause the processors to execute the steps of the image processing method.

[0223] The embodiment of this application also provides a computer program product containing instructions, which, when running on a computer, causes the computer to execute the image processing method.

[0224] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use, and processing of relevant data need to comply with relevant regulations.

[0225] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to a memory, database, or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memories. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc. The databases involved in the embodiments provided in the present application can include at least one of relational databases and non-relational databases. Non-relational databases can include distributed databases based on blockchain, etc., and are not limited thereto. The processors involved in the embodiments provided in the present application can be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, data processing logics based on quantum computing, etc., and are not limited thereto.

[0226] The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity of description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as the scope described in this specification.

[0227] The above-described embodiments merely represent several implementation manners of the present application. The description thereof is relatively specific and detailed, but it should not be construed as a limitation on the patent scope of the present application. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present application, several modifications and improvements can still be made, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the appended claims.

Claims

1. An image processing method, characterized in that, comprising: fusing a first image and at least one second image that is temporally adjacent to the first image to obtain a fused image; the boundaries of the first image and the at least one second image are both located in the fusion boundary of the fused image; based on the inertial measurement data corresponding to the first image and the fusion boundary, determining a target area within the fusion boundary, and obtaining a target image according to the target area.

2. The method according to claim 1, characterized in that, the determining a target area within the fusion boundary based on the inertial measurement data corresponding to the first image and the fusion boundary includes: determining a target pose of the camera module based on the inertial measurement data corresponding to the first image and the fusion boundary; determining a target area within the fusion boundary according to the target pose and the fusion boundary.

3. The method according to claim 2, characterized in that, the determining a target pose of the camera module based on the inertial measurement data corresponding to the first image and the fusion boundary includes: determining an initial pose of the camera module based on the inertial measurement data corresponding to the first image; adjusting the initial pose based on the fusion boundary to determine the target pose of the camera module; the target pose enables the boundary pixel points in the adjusted image to be all located in the area included in the fusion boundary, and the similarity degree between the target pose and the initial pose is greater than a target similarity threshold, and the adjusted image is an image obtained by adjusting the first image with the target pose.

4. The method according to claim 2, characterized in that, the determining a target area within the fusion boundary according to the target pose and the fusion boundary includes: determining a first grid according to the fusion boundary; the fusion boundary is located in the area included in the first grid; adjusting the first grid based on the target pose to obtain a second grid; in the coordinate system of the second grid, determining the area where the first image is located within the fusion boundary as the target area.

5. The method according to claim 4, characterized in that, the determining a first grid according to the fusion boundary includes: determining the circumscribed rectangle of the fusion boundary; performing grid division on the circumscribed rectangle to obtain a first grid.

6. The method according to claim 1, characterized in that, before the determining a target area within the fusion boundary based on the inertial measurement data corresponding to the first image and the fusion boundary, further comprising: determining a fusion boundary according to the boundary of the first image and the boundaries of at least one second image that is temporally adjacent to the first image.

7. The method according to claim 6, characterized in that, the determining a fusion boundary according to the boundary of the first image and the boundaries of at least one second image that is temporally adjacent to the first image includes: aligning at least one second image that is temporally adjacent to the first image with the first image one by one; obtaining the boundary of the first image and the boundaries of each aligned second image; Fuse the boundaries of the first image and the boundaries of each aligned second image to determine the fused boundary.

8. The method according to claim 7, wherein, the fusing the boundaries of the first image and the boundaries of each aligned second image to determine the fused boundary includes: Fuse the boundaries of the first image and the boundaries of each aligned second image to obtain an intermediate boundary; the intermediate boundary includes each first boundary point; Select second boundary points from the respective first boundary points; Determine the fused boundary based on each of the second boundary points.

9. The method according to claim 8, wherein, the selecting second boundary points from the respective first boundary points includes: For each first boundary point, if the first boundary point meets the target condition, then select the first boundary point as a second boundary point; the target condition includes at least one of the confidence of the registration data corresponding to the image to which the first boundary point belongs being greater than the confidence threshold, there being a boundary intersection between the image to which the first boundary point belongs and an adjacent image, and there being a boundary intersection between the image to which the first boundary point belongs and the first image; the registration data corresponding to the image to which the first boundary point belongs is obtained by registering the image to which the first boundary point belongs with the first image.

10. The method according to claim 9, wherein, The second image adjacent to the first image includes at least one of the second image generated at a first time point and the second image generated at a second time point, the first time point being earlier than the generation time point of the first image, and the second time point being later than the generation time point of the first image; There being a boundary intersection between the image to which the first boundary point belongs and an adjacent image includes: if the image to which the first boundary point belongs is the second image generated at the first time point, then there is a boundary intersection between the image to which the first boundary point belongs and an adjacent image, and the generation time point of the adjacent image is later than the first time point; if the image to which the first boundary point belongs is the second image generated at the second time point, then there is a boundary intersection between the image to which the first boundary point belongs and an adjacent image, and the generation time point of the adjacent image is earlier than the second time point.

11. The method according to claim 1, wherein, The method further includes: Obtain a third image in the image stream; Perform lens distortion correction on the third image to obtain an intermediate image; Perform water ripple distortion correction on the intermediate image to obtain the first image.

12. The method according to claim 11, wherein, The third image is an image in a two-dimensional space; the performing lens distortion correction on the third image to obtain an intermediate image includes: Obtain the internal lens parameters and the lens distortion parameters; Based on the internal lens parameters and the lens distortion parameters, back-project the third image belonging to the two-dimensional space into a three-dimensional space to obtain an intermediate image.

13. The method according to claim 11, wherein, The performing water ripple distortion correction on the intermediate image to obtain the first image includes: Determine a reference pixel row from the intermediate image; For each first pixel row other than the reference pixel row, align the first pixel row to the reference pixel row to obtain an aligned second pixel row; Generate a first image based on the reference pixel row and each aligned second pixel row.

14. The method according to claim 13, wherein, the intermediate image is an image in a three-dimensional space; for each first pixel row other than the reference pixel row, aligning the first pixel row to the reference pixel row to obtain an aligned second pixel row includes: Obtain the pose corresponding to the reference pixel row and the pose corresponding to each first pixel row other than the reference pixel row from the intermediate image in the three-dimensional space; For each first pixel row, determine the pose difference between the pose corresponding to the first pixel row and the pose corresponding to the reference pixel row; Based on the pose difference corresponding to the first pixel row, align the first pixel row to the reference pixel row to obtain an aligned second pixel row.

15. The method according to claim 11, wherein, performing moiré distortion correction on the intermediate image to obtain a first image includes: Perform moiré distortion correction on the intermediate image to obtain a distortion-corrected image in a three-dimensional space; Project the distortion-corrected image in the three-dimensional space onto a two-dimensional space to obtain a third grid; Obtain a first image based on the third image and the third grid.

16. An image processing apparatus, wherein, comprising: An image fusion module for fusing a first image and at least one second image temporally adjacent to the first image to obtain a fused image; the boundaries of the first image and the at least one second image are both located in the fusion boundary of the fused image; A target area determination module for determining a target area within the fusion boundary based on the inertial measurement data corresponding to the first image and the fusion boundary, A target image determination module for obtaining a target image according to the target area.

17. An electronic device comprising a memory and a processor, and a computer program is stored in the memory, wherein, when the computer program is executed by the processor, the processor is caused to execute the steps of the image processing method according to any one of claims 1 to 15.

18. A computer-readable storage medium, on which a computer program is stored, wherein, when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 15 are implemented.

Images