Image correction method and device, electronic equipment, medium and program product

Abstract

The invention relates to an image correction method and device, electronic equipment, a medium and a program product, and relates to the technical field of image processing.The method comprises the steps that the camera posture rotation amount of an original image is determined according to gyroscope information and camera exposure parameters, therefore, image distortion or distortion caused by camera motion in the shooting process can be corrected through the camera attitude rotation amount, and the object movement correction value of the original image is determined according to the camera attitude difference between the original image and the previous frame of image and the original position of the target point in the original image in the previous frame of image. Therefore, image distortion or distortion caused by movement of the shooting object in the shooting process is corrected through the object movement correction value, the original image is corrected through a method of combining the camera posture rotation amount and the object movement correction value, the correction effect can be improved, and the shooting quality of the image is improved.

Description

Technical Field

[0001] This disclosure relates to the field of image processing technology, and in particular to an image correction method, apparatus, electronic device, medium, and program product. Background Technology

[0002] Camera shutters can be divided into global shutters and rolling shutters. A rolling shutter, like a rolling door, scans pixels line by line from top to bottom and processes electrical signals until all pixels are exposed. If the subject is moving at high speed or vibrating rapidly relative to the camera, the line-by-line scanning speed is insufficient when using a rolling shutter, potentially resulting in "tilted," "wobbly," or "partially exposed" images. This phenomenon caused by rolling shutters is called the rolling shutter effect. The rolling shutter effect is determined by the imaging characteristics of CMOS (Complementary Metal-Oxide-Semiconductor) sensors themselves; it arises from the line-by-line exposure method of the image sensor. Currently, all mobile phone cameras use CMOS sensors and mostly employ rolling shutters for exposure. Therefore, addressing the rolling shutter effect in CMOS cameras is a crucial factor in ensuring good mobile phone image quality. Summary of the Invention

[0003] To overcome the problems existing in related technologies, this disclosure provides an image correction method, apparatus, electronic device, medium, and program product. Based on gyroscope information and camera exposure parameters, the camera attitude rotation amount of the original image is determined, thereby correcting image distortion or warping caused by camera movement during shooting. Furthermore, based on the camera attitude difference between the original image and the previous frame, and the original position of the target point in the original image in the previous frame, an object movement correction value is determined for the original image. This object movement correction value corrects image distortion or warping caused by the movement of the subject during shooting. By combining the camera attitude rotation amount and the object movement correction value to correct the original image, the correction effect can be improved, thereby improving the image shooting quality.

[0004] According to a first aspect of the present disclosure, an image correction method is provided, comprising: Based on the gyroscope information and camera exposure parameters, determine the camera attitude rotation amount of the original image; Based on the camera pose difference between the original image and the previous frame image, and the original position of the target point in the previous frame image, the object motion correction value of the original image is determined. The original image is corrected based on the camera attitude rotation amount and the object movement correction value to obtain the target corrected image.

[0005] Optionally, determining the object motion correction value of the original image based on the camera pose difference between the original image and the previous frame image, and the original position of the target point in the previous frame image, includes: Based on the camera pose difference between the original image and the previous frame image, determine the mapping position of the target point in the original image; The amount of active movement of the subject is determined based on the mapped position and the original position of the target point in the previous frame image; Based on the active movement amount, the object motion correction value of the original image is determined.

[0006] Optionally, the active movement amount includes lateral movement amount, and the object movement correction value includes lateral correction value; Determining the object motion correction value of the original image based on the active motion amount includes: The lateral movement speed of the photographed object is determined based on the lateral movement amount and the exposure time of the original image; Based on the lateral movement speed and the relative correction time corresponding to each sampled pixel row in the original image, the object movement correction value of the original image is determined. The relative correction time corresponding to any sampled pixel row is the time difference between the exposure time corresponding to that sampled pixel row and the exposure time of the image center of the original image.

[0007] Optionally, determining the amount of active movement of the subject based on the mapped position and the original position of the target point in the previous frame image includes: The lateral movement of the photographed object is determined based on the difference in horizontal coordinates between the mapped position and the original position.

[0008] Optionally, the object movement correction value includes a lateral correction value for each pixel row; The step of determining the object motion correction value of the original image based on the lateral movement speed and the relative correction time corresponding to each sampled pixel row in the original image includes: Based on the lateral movement speed and the relative correction time corresponding to each row of sampled pixels in the original image, the lateral correction value corresponding to each row of sampled pixels in the original image is determined. Based on the lateral correction value corresponding to each sampled pixel row in the original image, linear interpolation is performed on each original pixel row in the original image to obtain the lateral correction value of each pixel row in the original image.

[0009] Optionally, determining the camera attitude rotation amount of the original image based on gyroscope information and camera exposure parameters includes: Based on the camera exposure parameters, determine the exposure time corresponding to each row of sampled pixels and the exposure time at the center of the image; Based on the gyroscope information, determine the camera pose at the exposure time corresponding to each row of sampled pixels, as well as the camera pose at the exposure time of the image center. The camera pose rotation amount of the original image is determined based on the camera pose at the exposure time corresponding to each row of sampled pixels and the camera pose at the exposure time of the image center.

[0010] Optionally, the step of correcting the original image based on the camera pose rotation amount and the object movement correction value to obtain the target corrected image includes: Based on the camera pose rotation amount, the position of each pixel in the original image is corrected to obtain a preliminary corrected image; Based on the object movement correction value, the position of each pixel in the preliminary corrected image is corrected to obtain the target corrected image.

[0011] According to a second aspect of the present disclosure, an image correction apparatus is provided, comprising: The first determining module is configured to determine the camera attitude rotation amount of the original image based on gyroscope information and camera exposure parameters. The second determining module is configured to determine the object motion correction value of the original image based on the camera pose difference between the original image and the previous frame image, and the original position of the target point in the previous frame image. The third determining module is configured to correct the original image based on the camera attitude rotation amount and the object movement correction value to obtain the target corrected image.

[0012] According to a third aspect of the present disclosure, an electronic device is provided, comprising: processor; Memory used to store processor-executable instructions; The processor is configured to execute the steps of the image correction method provided in the first aspect of this disclosure.

[0013] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided that stores computer program instructions thereon, which, when executed by a processor, implement the steps of the image correction method provided in the first aspect of the present disclosure.

[0014] According to a fifth aspect of the present disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps of the image correction method provided in the first aspect of the present disclosure.

[0015] The technical solutions provided by the embodiments of this disclosure may include the following beneficial effects: Based on gyroscope information and camera exposure parameters, the camera attitude rotation of the original image is determined. This allows for the correction of image distortion or warping caused by camera movement during shooting. Furthermore, based on the difference in camera attitude between the original image and the previous frame, and the original position of the target point in the original image in the previous frame, an object movement correction value is determined for the original image. This object movement correction value is then used to correct image distortion or warping caused by the movement of the subject during shooting. By combining the camera attitude rotation and object movement correction values ​​to correct the original image, the correction effect can be improved, thereby enhancing the image shooting quality.

[0016] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0017] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0018] Figure 1 This is a schematic diagram illustrating an application scenario of an image correction method according to an exemplary embodiment.

[0019] Figure 2 This is a flowchart illustrating an image correction method according to an exemplary embodiment.

[0020] Figure 3 This is a schematic diagram illustrating a sampling pixel row division according to an exemplary embodiment.

[0021] Figure 4 This is a schematic diagram illustrating a target point according to an exemplary embodiment.

[0022] Figure 5 This is a block diagram illustrating an image correction device according to an exemplary embodiment.

[0023] Figure 6 This is a block diagram illustrating an electronic device according to an exemplary embodiment. Detailed Implementation

[0024] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. In the following description relating to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements.

[0025] The embodiments described in the following examples of this disclosure are not representative of all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0026] It should be noted that all actions involving the acquisition of signals, information, or data in this disclosure are carried out in compliance with the relevant data protection laws and policies of the country where the location is situated, and with authorization from the owner of the relevant device.

[0027] Camera shutters can be divided into global shutters and rolling shutters. A rolling shutter, like a rolling curtain, scans pixels line by line from top to bottom and processes electrical signals until all pixels are exposed. If the subject is moving at high speed or vibrating rapidly relative to the camera, the line-by-line scanning speed is insufficient when using a rolling shutter, potentially resulting in "tilted," "shaky," or "partially exposed" images. This phenomenon caused by rolling shutters is called the rolling shutter effect. The rolling shutter effect is determined by the imaging characteristics of the CMOS sensor itself; it arises from the line-by-line exposure method of the image sensor. Currently, all mobile phone cameras use CMOS sensors and mostly employ rolling shutters for exposure. Therefore, addressing the rolling shutter effect in CMOS cameras is a crucial factor in ensuring good mobile phone image quality.

[0028] When correcting the rolling shutter effect, it is necessary to quantify the amount of movement of the object as it is imaged on the CMOS to calculate the calibration value. The movement of the object actually includes two parts: the camera's own motion (rotation and movement) and the object's own motion. In related technologies, it is believed that the object's movement is only caused by the rotation of the phone, so the current correction of the rolling shutter effect in cameras mainly relies on gyroscope information.

[0029] In related technologies, the correction of the rolling shutter effect mainly relies on gyroscope data. This involves measuring the angular velocity of the camera-equipped terminal using a gyroscope, calculating the terminal's rotation angle through time integration of the angular velocity, and finally combining the terminal's intrinsic and extrinsic parameter matrices to calculate the translation amount of each row of pixels in the image, thus completing the correction. However, gyroscope-based correction methods are based on the assumption that the terminal's movement only involves rotation, not translation, and that the object being photographed has no active displacement. This is clearly an idealized approximation of actual shooting conditions. For example, in a front-facing selfie, the movement of our hand and the up-and-down movement of our face can all contribute to subtle rolling shutter effects. In such cases, gyroscope-based correction methods are not very effective at correcting the rolling shutter effect.

[0030] To address the aforementioned technical problems, this disclosure provides an image correction method, apparatus, electronic device, medium, and program product. Based on gyroscope information and camera exposure parameters, the camera attitude rotation amount of the original image is determined, thereby correcting image distortion or warping caused by camera movement during shooting. Furthermore, based on the camera attitude difference between the original image and the previous frame, and the original position of the target point in the original image in the previous frame, an object movement correction value is determined for the original image. This object movement correction value corrects image distortion or warping caused by the movement of the subject during shooting. By combining the camera attitude rotation amount and the object movement correction value to correct the original image, the correction effect can be improved, thereby enhancing the image shooting quality.

[0031] Figure 1 This is a schematic diagram illustrating an application scenario of an image correction method according to an exemplary embodiment, such as... Figure 1 As shown, this method can be applied to terminals with cameras, such as mobile phones and other devices with camera functions. Due to the movement of the camera and the movement of the subject being photographed, image distortion or warping occurs in the original captured image.

[0032] Figure 2 This is a flowchart illustrating an image correction method according to an exemplary embodiment. Figure 3 This is a schematic diagram illustrating a sampling pixel row division according to an exemplary embodiment, such as... Figure 2 and Figure 3 As shown, the following steps may be included.

[0033] In step S201, the camera attitude rotation amount of the original image is determined based on the gyroscope information and camera exposure parameters.

[0034] In this embodiment, the gyroscope information can be the camera pose corresponding to any exposure moment, and the camera exposure parameters can include the end time of the current frame's exposure, the exposure duration of the current frame, and the readout time of the CMOS sensor. The camera pose rotation amount of the original image can include the camera pose rotation amount corresponding to multiple sampling pixel rows, such as... Figure 3 As shown, the original image can be evenly divided into multiple image rows by sampling pixel rows. Each image row includes at least one row of pixels, and a sampling pixel row is one row of pixels in the original image. Based on gyroscope information and camera exposure parameters, the camera pose rotation amount corresponding to each sampling pixel row in the original image can be determined. Then, the pixels in the corresponding image row are corrected using the camera pose rotation amount corresponding to each sampling pixel row in the original image. Here, the camera pose rotation amount is the adjustment amount corresponding to the camera's motion.

[0035] In step S202, the object motion correction value of the original image is determined based on the camera pose difference between the original image and the previous frame image, and the original position of the target point in the previous frame image.

[0036] In this embodiment, the adjustment amount corresponding to the movement of the subject can also be determined based on two adjacent frames, namely the original image and the previous frame corresponding to the original image. Specifically, the active movement amount of the subject in the original image can be determined based on the camera pose difference between the original image and the previous frame, and the original position of the target point in the previous frame, thereby determining the object movement correction value of the original image. The multiple sampled pixel rows can be referenced to obtain the object movement correction value corresponding to each sampled pixel row in the original image based on the active movement amount of the subject.

[0037] In step S203, the original image is corrected based on the camera attitude rotation amount and the object movement correction value to obtain the target corrected image.

[0038] In this embodiment, the image distortion or warping caused by camera movement during shooting can be corrected by the camera posture rotation amount, and the image distortion or warping caused by the movement of the shooting object during shooting can be corrected by the object movement correction value, so as to obtain the target corrected image, which can improve the correction effect and thus improve the image shooting quality.

[0039] In this embodiment, the camera attitude rotation amount of the original image is determined based on gyroscope information and camera exposure parameters. This allows for the correction of image distortion or warping caused by camera movement during shooting. Furthermore, based on the difference in camera attitude between the original image and the previous frame, and the original position of the target point in the original image in the previous frame, an object movement correction value is determined for the original image. This object movement correction value is then used to correct image distortion or warping caused by the movement of the subject during shooting. Combining camera attitude rotation and object movement correction values ​​to correct the original image improves the correction effect and thus enhances the image quality.

[0040] In one possible implementation, determining the object motion correction value of the original image based on the camera pose difference between the original image and the previous frame image, and the original position of the target point in the previous frame image, includes: Based on the camera pose difference between the original image and the previous frame, determine the mapped position of the target point in the original image; based on the mapped position and the original position of the target point in the previous frame, determine the active movement amount of the subject; based on the active movement amount, determine the object movement correction value in the original image.

[0041] Figure 4This is a schematic diagram illustrating a target point according to an exemplary embodiment, such as... Figure 4 As shown, in this embodiment, the rotation matrix between the original image and the previous frame image can be determined based on the camera pose difference between the original image and the previous frame image. Based on the rotation matrix, the original position of the target point in the previous frame image is transformed to obtain the mapped position of the target point in the original image. The target point can be a feature point of the subject in the original image or the previous frame image; for example, if the subject is a face, the target point could be the point corresponding to the tip of the nose. Figure 4 As shown, the actual position of the target point in the original image, the mapped position of the target point in the original image, and the original position of the target point in the previous frame image can be marked on the original image, such as... Figure 4 The target point's movement vector can be decomposed into the camera's movement vector. and the active movement vector of the subject The vector sum of the vectors can be used to obtain the active movement vector of the photographed object. This allows us to obtain the amount of active movement of the subject being photographed, and then determine the object movement correction value of the original image based on the amount of active movement.

[0042] In one possible implementation, the camera pose difference between the original image and the previous frame image can be obtained using the following method: The exposure time of the image center of the original image can be determined based on the gyroscope parameters and camera exposure parameters. And, the exposure time of the center of the previous frame image. Thus, the attitude quaternion is determined. and Then, the camera pose difference between the original image and the previous frame image can be calculated using the following formula:

[0043] After obtaining the camera pose difference between the original image and the previous frame, the mapped position of the target point in the original image can be obtained by combining the original position of the target point in the previous frame. Specifically, the mapped position of the target point in the original image can be obtained by... Transform into a rotation matrix Based on the coordinates of the target point and the rotation matrix By combining the transformation formula, the mapped position of the target point in the original image can be obtained. The transformation formula can be:

[0044] in, These are the coordinates of the target point in the original image, i.e., the mapped position of the target point in the original image. This represents the original position of the target point in the previous frame image.

[0045] In one possible implementation, the active movement amount includes a lateral movement amount, and the object movement correction value includes a lateral correction value.

[0046] Based on the amount of active movement, determine the object motion correction value for the original image, including: The lateral movement speed of the subject is determined based on the lateral movement amount and the exposure time of the original image. The object movement correction value of the original image is determined based on the lateral movement speed and the relative correction time corresponding to each sampled pixel row in the original image. The relative correction time corresponding to any sampled pixel row is the time difference between the exposure time corresponding to that sampled pixel row and the exposure time of the image center of the original image.

[0047] In this embodiment, under normal circumstances, no correction is performed along the exposure direction. Therefore, the active movement amount may include the lateral movement amount, and the object movement correction value may include the lateral correction value, so as to correct the horizontal coordinate of the pixel in the original image.

[0048] Optionally, the method for determining the active movement of the subject based on the mapped position and the original position of the target point in the previous frame can be as follows: determine the lateral movement of the subject based on the difference in the horizontal coordinates between the mapped position and the original position. Here, both the mapped position and the original position are coordinate points composed of horizontal and vertical coordinates. The difference in the horizontal coordinates of the two points can be calculated based on the horizontal coordinates of the mapped position and the original position to obtain the lateral movement of the subject.

[0049] Then, based on the lateral movement amount and the exposure time of the original image, determine the lateral movement speed of the subject. The specific calculation formula is as follows:

[0050] in, The lateral movement speed of the subject being photographed. This represents the lateral movement. This refers to the exposure duration of the original image, i.e., the duration of one frame of video. For example, for a 30fps video... It takes 33.3ms.

[0051] Then, using the exposure time at the center of the original image as a benchmark, the relative correction time for each sampled pixel row is determined based on the exposure time corresponding to each sampled pixel row and the exposure time at the center of the original image. Therefore, based on the lateral movement speed and the relative correction time for each sampled pixel row in the original image, the object movement correction value of the original image, i.e., the lateral correction value for each sampled pixel row in the original image, can be obtained. The specific calculation formula is as follows:

[0052] in, For the first i The lateral correction value corresponding to each row of sampled pixels. For the first i The relative correction time corresponding to each row of sampled pixels, where... For the first i The exposure time corresponding to each row of sampled pixels. The exposure time is the center of the original image.

[0053] In one possible implementation, the active movement amount also includes a longitudinal movement amount, and the object movement correction value also includes a longitudinal correction value.

[0054] Determining the object motion correction value for the original image based on the amount of active motion also includes: Based on the longitudinal movement amount and the exposure time of the original image, determine the longitudinal movement speed of the subject; based on the longitudinal movement speed and the relative correction time corresponding to each row of sampled pixels in the original image, determine the object movement correction value of the original image.

[0055] In this embodiment, correction can be performed along the exposure direction. Therefore, the active movement amount may also include the longitudinal movement amount, and the object movement correction value may also include the longitudinal correction value, so as to correct the vertical coordinate of the pixel in the original image.

[0056] Optionally, the method for determining the active movement of the subject based on the mapped position and the original position of the target point in the previous frame image can also be as follows: determine the longitudinal movement of the subject based on the difference in the ordinates of the mapped position and the original position. Here, both the mapped position and the original position are coordinate points composed of abscissa and ordinate. The longitudinal movement of the subject can be obtained by calculating the difference in the ordinates of the mapped position and the original position.

[0057] Then, based on the vertical movement amount and the exposure time of the original image, determine the vertical movement speed of the subject. The specific calculation formula is as follows:

[0058] in, The vertical movement speed of the subject being photographed. This is the longitudinal movement amount. This represents the exposure time of the original image.

[0059] Therefore, based on the longitudinal movement speed and the relative correction time corresponding to each sampled pixel row in the original image, the object movement correction value of the original image can be obtained, that is, the longitudinal correction value corresponding to each sampled pixel row in the original image. The specific calculation formula is as follows:

[0060] in, For the first i The vertical correction value corresponding to each row of sampled pixels. For the first i The relative correction time corresponding to each row of sampled pixels, where... For the first i The exposure time corresponding to each row of sampled pixels. The exposure time is the center of the original image.

[0061] In one possible implementation, the object movement correction value includes a lateral correction value for each pixel row.

[0062] Based on the lateral movement speed and the relative correction time corresponding to each sampled pixel row in the original image, the object movement correction value of the original image is determined, including: Based on the lateral movement speed and the relative correction time corresponding to each sampled pixel row in the original image, the lateral correction value corresponding to each sampled pixel row in the original image is determined; based on the lateral correction value corresponding to each sampled pixel row in the original image, linear interpolation is performed on each original pixel row in the original image to obtain the lateral correction value of each pixel row in the original image.

[0063] In this embodiment, to make the correction more accurate, the lateral correction value of each pixel row in the original image can be determined. Specifically, the lateral correction value of each sampled pixel row in the original image can be determined based on the lateral movement speed and the relative correction time corresponding to each sampled pixel row in the original image. After obtaining the lateral correction value of each sampled pixel row, linear interpolation is performed on the pixel rows between any two sampled pixel rows based on the lateral correction values ​​of any two sampled pixel rows, thereby obtaining the lateral correction value of each pixel row in the original image. This allows the horizontal coordinate of the pixels in the original image to be corrected using the lateral correction value corresponding to each pixel row.

[0064] In one possible implementation, determining the camera pose rotation amount of the original image based on gyroscope information and camera exposure parameters includes: Based on the camera exposure parameters, determine the exposure time corresponding to each row of sampled pixels and the exposure time at the center of the image; based on the gyroscope information, determine the camera pose at the exposure time corresponding to each row of sampled pixels and the camera pose at the exposure time at the center of the image; based on the camera pose at the exposure time corresponding to each row of sampled pixels and the camera pose at the exposure time at the center of the image, determine the camera pose rotation amount of the original image.

[0065] Continue to refer to Figure 3In this embodiment, the exposure time corresponding to each sampled pixel row and the exposure time at the image center can be determined based on the camera exposure parameters. Furthermore, the camera pose at the exposure time corresponding to each sampled pixel row and the camera pose at the exposure time at the image center can be determined based on the gyroscope information. Then, based on the camera pose at the exposure time corresponding to each sampled pixel row and the camera pose at the exposure time at the image center, the quaternion corresponding to each sampled pixel row and the image center can be determined, which can be calculated as follows: ... in, for The quaternion corresponding to the sampling pixel row at time step. for The gyroscope reading at any given time represents the rate of change of the object's angle of rotation around the x, y, and z axes. for The quaternion corresponding to the sampling pixel row at time step. for The gyroscope reading at any given time. for The quaternion corresponding to the sampling pixel row at time step. for The gyroscope reading at any given time. For time intervals. Figure 3 middle to The meaning of is deduced by analogy.

[0066] Then, the rotation matrix corresponding to each sampled pixel row and the image center is determined by the quaternion corresponding to each sampled pixel row and the image center. , , Then, based on the rotation matrix corresponding to each sampled pixel row and the image center, the camera pose rotation amount corresponding to each sampled pixel row in the original image is determined. The camera pose rotation amount can be: , where K is the camera intrinsic parameter matrix (3×3). This is the inverse operation of the camera intrinsic parameter matrix. Let be the rotation matrix corresponding to the quaternion of the exposure time at the center of the image. for The inverse operation of the rotation matrix corresponding to the quaternion of the sampled pixel row at time step.

[0067] In one possible implementation, the original image is corrected based on the camera pose rotation and object movement correction values ​​to obtain a target corrected image, including: Based on the camera pose rotation, the position of each pixel in the original image is corrected to obtain a preliminary corrected image; based on the object movement correction value, the position of each pixel in the preliminary corrected image is corrected to obtain the target corrected image.

[0068] In this embodiment, given the camera pose rotation amount corresponding to each row of sampled pixels, the position of the pixels corresponding to each row of sampled pixels can be corrected based on this rotation amount, resulting in a preliminary corrected image. The correction formula can be:

[0069] in, For the corrected The position of the pixel corresponding to the sampling pixel row at any given time. This corresponds to the camera attitude rotation amount. Before correction The position of the pixel corresponding to the sampling pixel row at any given time.

[0070] Given the object motion correction value corresponding to each sampled pixel row, the position of the pixel point corresponding to each sampled pixel row in the preliminary corrected image can be corrected according to the object motion correction value corresponding to each sampled pixel row, so as to obtain the target corrected image.

[0071] Grid[y][x] represents the value in the y-th row and x-th column. For the first row of pixels, the correction formula can be:

[0072] in, The position of the corrected pixel. The position of the pixel before correction. This is the horizontal correction value corresponding to the first row of pixels. The correction formulas for the second and third rows of pixels, and so on, follow the same pattern.

[0073] Figure 5 This is a block diagram illustrating an image correction apparatus according to an exemplary embodiment. (Refer to...) Figure 5 The image correction device 500 includes a first determining module 501, a second determining module 502, and a third determining module 503.

[0074] The first determining module 501 is configured to determine the camera attitude rotation amount of the original image based on gyroscope information and camera exposure parameters. The second determining module 502 is configured to determine the object motion correction value of the original image based on the camera pose difference between the original image and the previous frame image, and the original position of the target point in the previous frame image. The third determining module 503 is configured to correct the original image based on the camera attitude rotation amount and the object movement correction value to obtain a target corrected image.

[0075] Optionally, the second determining module 502 includes: The first determining submodule is configured to determine the mapping position of the target point in the original image based on the camera pose difference between the original image and the previous frame image; The second determining submodule is configured to determine the amount of active movement of the subject based on the mapped position and the original position of the target point in the previous frame image; The third determining submodule is configured to determine the object motion correction value of the original image based on the active motion amount.

[0076] Optionally, the active movement amount includes lateral movement amount, and the object movement correction value includes lateral correction value; The third determining submodule includes: The first determining unit is configured to determine the lateral movement speed of the subject based on the lateral movement amount and the exposure time of the original image; The second determining unit is configured to determine the object movement correction value of the original image based on the lateral movement speed and the relative correction time corresponding to each sampled pixel row in the original image, wherein the relative correction time corresponding to any sampled pixel row is the time difference between the exposure time corresponding to that sampled pixel row and the exposure time of the image center of the original image.

[0077] Optionally, the second determining submodule includes: The third determining unit is configured to determine the lateral movement of the photographed object based on the difference in the horizontal coordinates between the mapped position and the original position.

[0078] Optionally, the object movement correction value includes a lateral correction value for each pixel row; The second determining unit includes: The determining subunit is configured to determine the lateral correction value corresponding to each sampled pixel row in the original image based on the lateral movement speed and the relative correction time corresponding to each sampled pixel row in the original image; The subunit is configured to perform linear interpolation on each original pixel row in the original image based on the lateral correction value corresponding to each sampled pixel row in the original image, so as to obtain the lateral correction value of each pixel row in the original image.

[0079] Optionally, the first determining module 501 includes: The fourth determining submodule is configured to determine the exposure time corresponding to each row of sampled pixels and the exposure time of the image center based on the camera exposure parameters. The fifth determining submodule is configured to determine the camera pose at the exposure time corresponding to each row of sampled pixels, and the camera pose at the exposure time of the image center, based on the gyroscope information. The sixth determining submodule is configured to determine the camera pose rotation amount of the original image based on the camera pose at the exposure time corresponding to each sampled pixel row and the camera pose at the exposure time of the image center.

[0080] Optionally, the correction module includes: The first correction submodule is configured to correct the position of each pixel in the original image according to the camera pose rotation amount to obtain a preliminary corrected image; The second correction submodule is configured to correct the position of each pixel in the preliminary correction image based on the object movement correction value to obtain the target correction image.

[0081] Regarding the image correction device 500 in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated here.

[0082] This disclosure also provides a computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, implement the steps of the image correction method provided in this disclosure.

[0083] Figure 6 This is a block diagram illustrating an electronic device according to an exemplary embodiment. For example, the electronic device 600 may be a mobile phone, computer, messaging device, tablet device, medical device, personal digital assistant, etc.

[0084] Reference Figure 6 The electronic device 600 may include one or more of the following components: a first processing component 602, a first memory 604, a first power supply component 606, a multimedia component 608, an audio component 610, a first input / output interface 612, a sensor component 614, and a communication component 616.

[0085] The first processing component 602 typically controls the overall operation of the electronic device 600, such as operations associated with display, telephone calls, data communication, camera operation, and recording. The first processing component 602 may include one or more processors 620 to execute instructions to complete all or part of the steps of the image correction method described above. Furthermore, the first processing component 602 may include one or more modules to facilitate interaction between the first processing component 602 and other components. For example, the first processing component 602 may include a multimedia module to facilitate interaction between the multimedia component 608 and the first processing component 602.

[0086] The first memory 604 is configured to store various types of data to support the operation of the electronic device 600. Examples of this data include instructions for any application or method operating on the electronic device 600, contact data, phone book data, messages, pictures, videos, etc. The first memory 604 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0087] The first power supply component 606 provides power to various components of the electronic device 600. The first power supply component 606 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device 600.

[0088] Multimedia component 608 includes a screen that provides an output interface between the electronic device 600 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of the touch or swipe action but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 608 includes a front-facing camera and / or a rear-facing camera. When the electronic device 600 is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or the rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0089] Audio component 610 is configured to output and / or input audio signals. For example, audio component 610 includes a microphone (MIC) configured to receive external audio signals when electronic device 600 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in first memory 604 or transmitted via communication component 616. In some embodiments, audio component 610 also includes a speaker for outputting audio signals.

[0090] The first input / output interface 612 provides an interface between the first processing component 602 and the peripheral interface module, which may be a keyboard, click wheel, buttons, etc. These buttons may include, but are not limited to, a home button, volume buttons, a start button, and a lock button.

[0091] Sensor assembly 614 includes one or more sensors for providing state assessments of various aspects of electronic device 600. For example, sensor assembly 614 can detect the on / off state of electronic device 600, the relative positioning of components such as the display and keypad of electronic device 600, changes in position of electronic device 600 or a component of electronic device 600, the presence or absence of user contact with electronic device 600, orientation or acceleration / deceleration of electronic device 600, and temperature changes of electronic device 600. Sensor assembly 614 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 614 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 614 may also include an accelerometer, gyroscope, magnetometer, pressure sensor, or temperature sensor.

[0092] Communication component 616 is configured to facilitate wired or wireless communication between electronic device 600 and other devices. Electronic device 600 can access wireless networks based on communication standards, such as WiFi, 2G, or 3G, or combinations thereof. In one exemplary embodiment, communication component 616 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 616 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0093] In an exemplary embodiment, the electronic device 600 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the image correction method described above.

[0094] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a first memory 604 including instructions, which can be executed by a processor 620 of an electronic device 600 to complete the image correction method described above. For example, the non-transitory computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0095] In another exemplary embodiment, a computer program product is also provided, which includes a computer program executable by a programmable device, the computer program having a code portion for performing the image correction method described above when executed by the programmable device.

[0096] Those skilled in the art will also understand that the various illustrative logical blocks and steps listed in the embodiments of this application can be implemented by electronic hardware, computer software, or a combination of both. Whether such functionality is implemented through hardware or software depends on the specific application and the overall system design requirements. Those skilled in the art can implement the described functionality using various methods for each specific application, but such implementation should not be construed as exceeding the scope of protection of the embodiments of this application.

[0097] In the above detailed description, terms such as "center," "upper," "lower," "left," and "right" indicate direction or positional relationship. Since components of the described device can be positioned in multiple different orientations, these directional terms are for illustrative purposes and not restrictive. It should be understood that other aspects can be utilized and structural or logical changes can be made without departing from the concept of this disclosure. Therefore, the following detailed description should not be considered limiting.

[0098] It should be understood that, unless otherwise specifically indicated, features of various embodiments of this disclosure described herein can be combined with each other.

[0099] Although terms such as “first,” “second,” and “third” may be used herein to describe various components, parts, regions, layers, or sections, these components, parts, regions, layers, or sections are not limited to these terms. Rather, these terms are used only to distinguish one component, part, region, layer, or section from another. Therefore, without departing from the teachings of the examples described herein, the first component, part, region, layer, or section mentioned in the examples may also be referred to as the second component, part, region, layer, or section. Furthermore, the terms “first” and “second” are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as “first” or “second” may explicitly or implicitly include at least one of that feature. In the description herein, “a plurality” means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0100] Furthermore, the term “exemplary” is used herein to mean serving as an example, instance, or illustration. Any aspect or design described herein as “exemplary” is not necessarily to be construed as advantageous compared to other aspects or designs. Rather, the use of the term “exemplary” is intended to present the concept in a concrete manner. As used herein, the term “or” is intended to mean an inclusive “or” rather than an exclusive “or.” That is, unless otherwise specified or clear from the context, “X applies A or B” is intended to mean any of the natural inclusive arrangements. That is, “X applies A or B” satisfies any of the foregoing instances if X applies A; X applies B; or both X applies A and B. Additionally, unless otherwise specified or clear from the context to refer to the singular form, the articles “a” and “an” as used in this application and the appended claims are generally understood to mean “one or more.”

[0101] Similarly, although this disclosure has been shown and described with respect to one or more implementations, equivalent variations and modifications will occur to those skilled in the art upon reading and understanding this specification and the accompanying drawings. This disclosure includes all such modifications and variations and is limited only by the scope of the claims. In particular, with respect to the various functions performed by the components described above (e.g., elements, resources, etc.), unless otherwise indicated, the terminology used to describe such components is intended to correspond to any component (functionally equivalent) that performs the specific function of the described component, even if structurally not equivalent to the disclosed structure. Furthermore, although specific features of this disclosure may have been disclosed with respect to only one of several implementations, such features may be combined with one or more other features of other implementations, as may be desired and advantageous to any given or particular application. Moreover, with regard to the terms “comprising,” “owning,” “having,” “having,” or variations thereof as used in the detailed description or claims, such terms are intended to be inclusive in a manner similar to the term “including.”

[0102] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the appended claims.

[0103] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. An image correction method, characterized in that, include: Based on the gyroscope information and camera exposure parameters, determine the camera attitude rotation amount of the original image; Based on the camera pose difference between the original image and the previous frame image, and the original position of the target point in the previous frame image, the object motion correction value of the original image is determined. The original image is corrected based on the camera attitude rotation amount and the object movement correction value to obtain the target corrected image.

2. The image correction method according to claim 1, characterized in that, The step of determining the object motion correction value of the original image based on the camera pose difference between the original image and the previous frame image, and the original position of the target point in the previous frame image, includes: Based on the camera pose difference between the original image and the previous frame image, determine the mapping position of the target point in the original image; The amount of active movement of the subject is determined based on the mapped position and the original position of the target point in the previous frame image; Based on the active movement amount, the object motion correction value of the original image is determined.

3. The image correction method according to claim 2, characterized in that, The active movement amount includes lateral movement amount, and the object movement correction value includes lateral correction value; Determining the object motion correction value of the original image based on the active motion amount includes: The lateral movement speed of the photographed object is determined based on the lateral movement amount and the exposure time of the original image; Based on the lateral movement speed and the relative correction time corresponding to each sampled pixel row in the original image, the object movement correction value of the original image is determined. The relative correction time corresponding to any sampled pixel row is the time difference between the exposure time corresponding to that sampled pixel row and the exposure time of the image center of the original image.

4. The image correction method according to claim 3, characterized in that, Determining the amount of active movement of the subject based on the mapped position and the original position of the target point in the previous frame image includes: The lateral movement of the photographed object is determined based on the difference in horizontal coordinates between the mapped position and the original position.

5. The image correction method according to claim 3, characterized in that, The object movement correction value includes the lateral correction value for each pixel row; The step of determining the object motion correction value of the original image based on the lateral movement speed and the relative correction time corresponding to each sampled pixel row in the original image includes: Based on the lateral movement speed and the relative correction time corresponding to each row of sampled pixels in the original image, the lateral correction value corresponding to each row of sampled pixels in the original image is determined. Based on the lateral correction value corresponding to each sampled pixel row in the original image, linear interpolation is performed on each original pixel row in the original image to obtain the lateral correction value of each pixel row in the original image.

6. The image correction method according to any one of claims 1 to 5, characterized in that, The step of determining the camera attitude rotation amount of the original image based on gyroscope information and camera exposure parameters includes: Based on the camera exposure parameters, determine the exposure time corresponding to each row of sampled pixels and the exposure time at the center of the image; Based on the gyroscope information, determine the camera pose at the exposure time corresponding to each row of sampled pixels, as well as the camera pose at the exposure time of the image center. The camera pose rotation amount of the original image is determined based on the camera pose at the exposure time corresponding to each row of sampled pixels and the camera pose at the exposure time of the image center.

7. The image correction method according to any one of claims 1 to 5, characterized in that, The step of correcting the original image based on the camera pose rotation amount and the object movement correction value to obtain the target corrected image includes: Based on the camera pose rotation amount, the position of each pixel in the original image is corrected to obtain a preliminary corrected image; Based on the object movement correction value, the position of each pixel in the preliminary corrected image is corrected to obtain the target corrected image.

8. An image correction device, characterized in that, include: The first determining module is configured to determine the camera attitude rotation amount of the original image based on gyroscope information and camera exposure parameters. The second determining module is configured to determine the object motion correction value of the original image based on the camera pose difference between the original image and the previous frame image, and the original position of the target point in the previous frame image. The third determining module is configured to correct the original image based on the camera attitude rotation amount and the object movement correction value to obtain the target corrected image.

9. An electronic device, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to execute the steps of the image correction method according to any one of claims 1 to 7.

10. A computer-readable storage medium having computer program instructions stored thereon, characterized in that, When the computer program instructions are executed by the processor, they implement the steps of the image correction method according to any one of claims 1 to 7.

11. A computer program product, characterized in that, It includes a computer program that, when executed by a processor, implements the steps of the image correction method according to any one of claims 1 to 7.

Images