Image motion estimation method and electronic device

Abstract

The invention discloses an image motion estimation method and an electronic device. The image motion estimation method comprises the steps of obtaining a first RGB image shot at a current time point and a second RGB image shot at a previous time point through an RGB image sensor of the electronic device; obtaining a first angular velocity of the electronic device at the current time point; obtaining a first acceleration of the electronic device at the current time point; and determining a first movement speed of pixels in the first RGB image and a first translation speed of the electronic device at the current time point based on the first RGB image, the second RGB image, a second translation speed of the electronic device at the previous time point, the first angular speed and the first acceleration.

Description

Technical Field

[0001] This disclosure relates to the field of image processing, and more specifically, to image motion estimation methods and electronic devices. Background Technology

[0002] In image processing, the primary goal of motion estimation is to analyze (e.g., the displacement (or velocity) of corresponding pixels between consecutive image frames, caused by at least one of the motion of an electronic device including an image sensor and the motion of an object being photographed by the electronic device). However, when the motion speed of the electronic device and / or the object is fast, the image frames are blurred, and the motion speed of the pixels obtained through motion estimation has a large error. Summary of the Invention

[0003] The present invention is provided in a brief form to introduce the choice of concepts further described in the following detailed description. This summary is not intended to identify key or / or essential features of the claimed subject matter, nor is it intended to help determine the scope of the claimed subject matter.

[0004] This application provides an image motion estimation method and electronic device, which can at least improve the accuracy of estimating the motion speed of pixels.

[0005] According to one or more example embodiments, an image motion estimation method includes: obtaining a first RGB image captured at a current time point and a second RGB image captured at a previous time point by an RGB image sensor of an electronic device; obtaining a first angular velocity of the electronic device at the current time point; obtaining a first acceleration of the electronic device at the current time point; and determining a first motion velocity of a plurality of first pixels in the first RGB image and a first translational velocity of the electronic device at the current time point based on the first RGB image, the second RGB image, a second translational velocity of the electronic device at the previous time point, the first angular velocity of the electronic device at the current time point, and the first acceleration of the electronic device at the current time point, to estimate image motion.

[0006] The image motion estimation method may further include: deblurring a third RGB image captured by an RGB image sensor at a later time point based on the first motion velocity of the plurality of first pixels in the first RGB image.

[0007] The steps for determining the first motion velocity and the first translational velocity may include: based on the first RGB image, the second RGB image, the second translational velocity, the first angular velocity, and the first acceleration, simultaneously establishing a first function, a second function, a third function, and a fourth function to determine the first motion velocity, the first translational velocity, and the derivative of the first rotation matrix at the current time point. The first function reflects the relationship between the derivative of the camera coordinates corresponding to the first pixel representing a stationary object among the plurality of first pixels, and the camera coordinates, the derivative of the first rotation matrix, and the first translational velocity. The stationary object is an object that remains stationary during the time period between the previous time point and the current time point among corresponding objects in the first RGB image and the second RGB image. The second function reflects the relationship between the coordinates of the plurality of first pixels in the first RGB image and the coordinates of the plurality of second pixels corresponding to the plurality of first pixels in the second RGB image, and the first motion velocity of the plurality of first pixels in the first RGB image. The third function reflects the relationship between the first angular velocity and the derivative of the first rotation matrix. The fourth function reflects the relationship between the second translational velocity and the first acceleration and the first translational velocity.

[0008] In this application, when performing motion estimation, the translational velocity of the electronic device, which serves as the global motion vector, is determined by using the acceleration of the electronic device, thereby improving the accuracy and reliability of estimating the motion velocity of pixels.

[0009] In this application, by fusing the outputs of various types of sensors (e.g., RGB image sensors, sensors that directly or indirectly measure angular velocity and acceleration) and using these outputs as constraints for joint optimization, an improved motion estimate is obtained, thereby further improving the accuracy of the estimated pixel motion velocity.

[0010] In the first function, the derivative of the first rotation matrix and the product of the first translation velocity and the camera coordinates are equal to the derivative of the camera coordinates. The camera coordinates are obtained by substituting the coordinates of the first pixel representing the stationary object in the first RGB image and the Z coordinate of the stationary object at the current time point into the intrinsic parameter matrix. The Z coordinate of the stationary object at the current time point is obtained based on the second translation velocity, the Z coordinate of the stationary object at the previous time point, and the time period between the previous time point and the current time point. The derivative of the camera coordinates includes the first motion velocity representing the first pixel of the stationary object.

[0011] The image motion estimation method may further include: obtaining a first depth image at the current time point, wherein, in the first function, the product of the derivative of the first rotation matrix and the first translation velocity with the camera coordinates is equal to the derivative of the camera coordinates, wherein the camera coordinates are obtained by substituting the coordinates of the first pixel representing the stationary object in the first RGB image and the Z coordinate of the stationary object at the current time point into the intrinsic parameter matrix, wherein the Z coordinate of the stationary object at the current time point is obtained based on the first depth image, and wherein the derivative of the camera coordinates includes the first motion velocity representing the first pixel of the stationary object.

[0012] In this application, the accuracy of determining the Z coordinate can be improved by using a depth sensor, and more constraints can be provided, thereby further improving the accuracy of estimating the motion speed of pixels.

[0013] The steps of obtaining a first depth image at the current time point may include: performing a transformation on a depth image captured by a depth sensor of an electronic device, such that the transformed depth image is time-synchronized with a first RGB image and a second RGB image, wherein the transformed depth image corresponds to an image coordinate system, and wherein the resolution of the transformed depth image is the same as the resolution of each of the first RGB image and the second RGB image; obtaining a first depth image at the current time point from the transformed depth image, wherein the depth sensor and the RGB image sensor captured the same scene.

[0014] The image motion estimation method may further include: obtaining a first event image at the current time point; and further combining a fifth function, wherein the fifth function embodies the relationship between the spatial gradient of the first RGB image in the x and y directions, the first motion velocity of the pixels in the first RGB image, and the first event image.

[0015] In this application, by introducing a high temporal resolution event sensor, changes in image frames can be perceived more sensitively, and more constraints can be provided, thereby further improving the accuracy of estimating pixel motion speed.

[0016] The step of obtaining a first event image at the current time point may include: performing a transformation on an event image captured by an event sensor of an electronic device, such that the transformed event image is time-synchronized with a first RGB image and a second RGB image, wherein the transformed event image corresponds to an image coordinate system, and wherein the resolution of the transformed event image is the same as the resolution of each of the first RGB image and the second RGB image; obtaining a first event image at the current time point from the transformed event image, wherein the event sensor and the RGB image sensor captured the same scene.

[0017] According to one or more example embodiments, an electronic device includes: an RGB image sensor configured to: capture a first RGB image at a current time point and capture a second RGB image at a previous time point; a processor configured to: obtain the first RGB image and the second RGB image; obtain a first angular velocity of the electronic device at the current time point; obtain a first acceleration of the electronic device at the current time point; and, based on the first RGB image, the second RGB image, a second translational velocity of the electronic device at the previous time point, the first angular velocity of the electronic device at the current time point, and the first acceleration of the electronic device at the current time point, determine a first motion velocity of a plurality of first pixels in the first RGB image and a first translational velocity of the electronic device at the current time point to estimate image motion.

[0018] The processor can also be configured to deblur a third RGB image captured by an RGB image sensor at a later time point based on a first motion velocity of the plurality of first pixels in the first RGB image.

[0019] The processor can be configured to: determine a first motion velocity, a first translational velocity, and the derivative of a first rotation matrix at the current time point by combining a first function, a second function, a third function, and a fourth function based on a first RGB image, a second RGB image, a second translational velocity, a first angular velocity, and a first acceleration. The first function reflects the relationship between the derivative of the camera coordinates corresponding to the first pixel representing a stationary object among the plurality of first pixels and the camera coordinates, the derivative of the first rotation matrix, and the first translational velocity. The stationary object is an object that is stationary during the time period between the previous time point and the current time point among corresponding objects in the first RGB image and the second RGB image. The second function reflects the relationship between the coordinates of the plurality of first pixels in the first RGB image and the coordinates of the plurality of second pixels corresponding to the plurality of first pixels in the second RGB image and the first motion velocity of the plurality of first pixels in the first RGB image. The third function reflects the relationship between the first angular velocity and the derivative of the first rotation matrix. The fourth function reflects the relationship between the second translational velocity and the first acceleration and the first translational velocity.

[0020] In the first function, the derivative of the first rotation matrix and the product of the first translation velocity and the camera coordinates are equal to the derivative of the camera coordinates. The camera coordinates are obtained by substituting the coordinates of the first pixel representing the stationary object in the first RGB image and the Z coordinate of the stationary object at the current time point into the intrinsic parameter matrix. The Z coordinate of the stationary object at the current time point is obtained based on the second translation velocity, the Z coordinate of the stationary object at the previous time point, and the time period between the previous time point and the current time point. The derivative of the camera coordinates includes the first motion velocity representing the first pixel of the stationary object.

[0021] The processor can also be configured to: obtain a first depth image at the current time point, wherein, in a first function, the product of the derivative of the first rotation matrix and the first translation velocity with the camera coordinates is equal to the derivative of the camera coordinates, wherein the camera coordinates are obtained by substituting the coordinates of a first pixel representing the stationary object in the first RGB image and the Z coordinate of the stationary object at the current time point into an intrinsic parameter matrix, wherein the Z coordinate of the stationary object at the current time point is obtained based on the first depth image, and wherein the derivative of the camera coordinates includes a first motion velocity representing the first pixel of the stationary object.

[0022] The processor can be configured to: perform a transformation on a depth image captured by a depth sensor of an electronic device, such that the transformed depth image is time-synchronized with a first RGB image and a second RGB image, wherein the transformed depth image corresponds to an image coordinate system, and wherein the resolution of the transformed depth image is the same as the resolution of each of the first RGB image and the second RGB image; and obtain a first depth image at the current time point from the transformed depth image, wherein the depth sensor and the RGB image sensor captured the same scene.

[0023] The processor can also be configured to: obtain a first event image at the current time point; and further combine a fifth function, wherein the fifth function embodies the relationship between the spatial gradient of the first RGB image in the x and y directions, the first motion velocity of the pixels in the first RGB image, and the first event image.

[0024] The processor can be configured to: perform a transformation on an event image captured by an event sensor of an electronic device, such that the transformed event image is time-synchronized with a first RGB image and a second RGB image, wherein the transformed event image corresponds to an image coordinate system, and wherein the resolution of the transformed event image is the same as the resolution of each of the first RGB image and the second RGB image; and obtain a first event image at the current time point from the transformed event image, wherein the event sensor and the RGB image sensor capture the same scene.

[0025] A non-transitory computer-readable storage medium is provided that stores instructions which, when executed by a processor, cause the processor to perform the methods described above.

[0026] Further aspects and / or advantages of the inventive concept will be set forth in part in the description which follows, and in part will be obvious from the description and / or may be learned by practice of various exemplary embodiments. Attached Figure Description

[0027] The above and other objects, features and advantages of this disclosure will become clearer from the following detailed description taken in conjunction with the accompanying drawings; Figure 1 This is a flowchart illustrating an image motion estimation method according to one or more embodiments.

[0028] Figure 2 This is a schematic diagram illustrating an image motion estimation method according to one or more embodiments.

[0029] Figure 3 This is a flowchart illustrating in more detail an image motion estimation method according to one or more embodiments.

[0030] Figure 4 A block diagram of an electronic device according to one or more embodiments is shown.

[0031] Figure 5 A block diagram of an electronic device according to one or more embodiments is shown. Detailed Implementation

[0032] The following detailed description is provided to assist the reader in gaining a comprehensive understanding of the methods, apparatus, and / or systems described herein. However, upon understanding the disclosure of this application, various changes, modifications, and equivalents of the methods, apparatus, and / or systems described herein will become apparent. For example, the order of operations described herein is merely illustrative and is not limited to those orders set forth herein, except for those that must occur in a specific order, but may be changed as will become clear upon understanding the disclosure of this application. Furthermore, for clarity and brevity, descriptions of features known in the art may be omitted.

[0033] The features described herein may be implemented in different forms and should not be construed as limited to the examples described herein. Rather, the examples described herein have been provided merely to illustrate some of the many feasible ways of implementing the methods, apparatus, and / or systems described herein that will be clear upon understanding the disclosure of this application.

[0034] The structural or functional descriptions of the examples disclosed herein are intended for illustrative purposes only, and the examples may be implemented in various forms. The examples are not intended to be limiting, but rather to include various modifications, equivalents, and substitutions within the scope of the claims.

[0035] Although the terms “first” or “second” are used to describe various components, the components are not limited to the terms. These terms should only be used to distinguish one component from another. For example, within the scope of the claims based on the concept of this disclosure, a “first” component may be referred to as a “second” component, or similarly, a “second” component may be referred to as a “first” component.

[0036] It will be understood that when a component is referred to as being "connected to" another component, the component may be directly connected to or combined with the other component, or there may be an intermediate component.

[0037] As used herein, unless the context clearly indicates otherwise, the singular form is intended to include the plural form as well. It should also be understood that when the terms “comprising” and / or “including” are used in this specification, they indicate the presence of the stated features, integrals, steps, operations, elements, components, or combinations thereof, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or groups thereof.

[0038] Unless otherwise defined, all terms used herein (including technical or scientific terms) shall have the same meaning as commonly understood by one of ordinary skill in the art to which the examples pertain. It will also be understood that, unless expressly defined herein, terms (such as those defined in a general dictionary) shall be interpreted as having a meaning consistent with their meaning in the context of the relevant field and shall not be interpreted in an idealized or overly formalized sense.

[0039] The examples will be described in detail below with reference to the accompanying drawings. Regarding the reference numerals assigned to the elements in the drawings, it should be noted that the same elements will be represented by the same reference numerals, and redundant descriptions will be omitted.

[0040] Figure 1 This is a flowchart illustrating an image motion estimation method according to one or more embodiments. Figure 2 This is a schematic diagram illustrating an image motion estimation method according to one or more embodiments.

[0041] Figure 1 A method for motion estimation is shown for a first RGB image (e.g., the i-th frame image) captured at the current time point, and Figure 1 The method shown can be applied to RGB images captured at any time other than the start time.

[0042] like Figure 1 As shown, in step S110, a first RGB image captured at the current time point by the RGB image sensor of the electronic device and a second RGB image captured at the previous time point (e.g., the i-1th frame image) can be obtained.

[0043] For example, to perform motion estimation on a first RGB image captured at the current time point, the first RGB image and a second RGB image captured at a previous time point can be obtained first. Furthermore, a second translational velocity of the electronic device at the previous time point, generated by motion estimation on the second RGB image captured at the previous time point, can also be obtained. .

[0044] For example, the first RGB image and the second RGB image can be adjacent image frames.

[0045] In step S120, the first angular velocity of the electronic device at the current time point can be obtained.

[0046] For example, when an electronic device includes a gyroscope sensor, the first angular velocity of the electronic device can be measured via the gyroscope sensor. However, the example is not limited to this, and the first angular velocity of the electronic device can be obtained in other ways.

[0047] In step S130, the first acceleration of the electronic device at the current time point can be obtained.

[0048] For example, when an electronic device includes an acceleration sensor, the first acceleration of the electronic device can be measured by the acceleration sensor. However, the example is not limited to this, and the first acceleration of the electronic device can be obtained in other ways.

[0049] In step S140, the first motion velocity of pixels (e.g., multiple first pixels) in the first RGB image can be determined based on the first RGB image, the second RGB image, the second translational velocity, the first angular velocity, and the first acceleration of the electronic device at the previous time point. , ) and the first translational speed of the electronic device at the current time point. .

[0050] For example, the aforementioned pixels (e.g., multiple first pixels) in the first RGB image can be pixels among corresponding pixels in the first RGB image and the second RGB image (e.g., for the aforementioned pixel in the first RGB image, there exists a corresponding pixel in the second RGB image). Here, corresponding pixels can be pixels representing corresponding objects (or the same object) in the first RGB image and the second RGB image. For example, when the same tree exists in the first RGB image and the second RGB image, corresponding pixels in the first RGB image and the second RGB image can include pixels representing the same tree.

[0051] In one example, the first motion velocity, the first translational velocity, and the derivative of the first rotation matrix at the current time point can be determined by combining the first to fourth functions based on the first RGB image, the second RGB image, the second translational velocity, the first angular velocity, and the first acceleration.

[0052] In this application, during the motion estimation process, the translational velocity of the electronic device, which serves as the global motion vector, is determined by using the acceleration of the electronic device, thereby improving the accuracy and reliability of estimating the motion velocity of pixels.

[0053] According to the example embodiment, the first function reflects the relationship between the derivative of the camera coordinates corresponding to the first pixel representing a stationary object in the first RGB image and the derivative of the camera coordinates, the first rotation matrix, and the first translation velocity. For example, a stationary object may represent an object in the first RGB image and the second RGB image that is stationary during the time period between the previous time point and the current time point. For example, a stationary object may be a street lamp in the first RGB image and the second RGB image, which is stationary during the time period between the previous time point and the current time point.

[0054] In one example, in the first function, the derivative of the first rotation matrix and the product of the first translation velocity and the camera coordinates are equal to the derivative of the camera coordinates. For example, the first function can be expressed as Equation 1 below.

[0055] (Formula 1) In Formula 1, , and It can represent the X, Y, and Z coordinates in the camera coordinate system. , and It can represent the rate of change of the pixel's X, Y, and Z coordinates in the camera coordinate system. This can represent the first rotation matrix. It can represent the derivative of the first rotation matrix. This can represent the first translational velocity.

[0056] In one example, the Z-coordinate of a stationary object at the current time point can be determined. The camera coordinates can be obtained based on the determined Z-coordinates and the coordinates of the first pixel. Then, the derivative of the camera coordinates can be obtained. Next, the camera coordinates and the derivative of the camera coordinates can be substituted into Formula 1 above.

[0057] For example, the Z-coordinate of the stationary object at the current time point can be obtained based on the second translational velocity, the Z-coordinate of the stationary object at the previous time point, and the aforementioned time period. For example, the Z-coordinate of the stationary object at the current time point can be expressed as the following formula 2.

[0058] (Formula 2) In Formula 2, It can represent the Z-coordinate of a stationary object at the current time point. It can represent the Z-coordinate of a stationary object at the previous time point. This can represent the second translational velocity. The Z component, It can represent the length of the time interval between the previous time point and the current time point.

[0059] In one example, camera coordinates can be determined by substituting the coordinates of the first pixel representing the stationary object in the first RGB image and the Z-coordinate of the stationary object at the current time point into the intrinsic parameter matrix. For example, camera coordinates can be determined using the following formula 3, and the derivative of the camera coordinates can be expressed using the following formula 4.

[0060] (Formula 3) In formula 3, and It can represent the X and Y coordinates of the first pixel representing a stationary object in the first RGB image.

[0061] = (Formula 4) In formula 4, and It can represent the rate of change of the X and Y coordinates of the first pixel representing a stationary object in the first RGB image with respect to time, that is, the motion speed of the first pixel representing the stationary object (e.g., the derivative of the camera coordinates includes the first motion speed of the first pixel representing the stationary object). It can represent the rate of change of the Z-coordinate of a stationary object with respect to time at the current time point, and for example, it can be... .

[0062] For example, formulas 2, 3, and 4 can be substituted into formula 1 above to obtain the final first function. For example, the final first function can be expressed as formula 5 below.

[0063] (Formula 5) In another example, when the electronic device includes a depth sensor, a first depth image at the current time point can be obtained, the Z coordinate of the stationary object at the current time point can be obtained based on the first depth image, the camera coordinates can be obtained based on the determined Z coordinates and the coordinates of the first pixel representing the stationary object, and then the derivative of the camera coordinates can be obtained. Next, the camera coordinates and the derivative of the camera coordinates can be substituted into the above formula 1.

[0064] For example, the Z-coordinate of a stationary object at the current time point can be obtained based on the output of a depth sensor. .

[0065] For example, it can be determined based on the first depth image. .For example, It can be expressed as the following formula 6.

[0066] (Formula 6) In Formula 6, , and Can represent The values ​​obtained by taking partial derivatives with respect to x, y, and t.

[0067] For example, the final first function can be obtained by substituting formulas 3, 4, and 6 into formula 1 above. For example, the final first function can be expressed as formula 7 below.

[0068] (Formula 7) In one example, a transformation can be performed on a depth image captured by a depth sensor of an electronic device, such that the transformed depth image is time-synchronized with a first RGB image and a second RGB image, the transformed depth image corresponds to an image coordinate system, and the resolution of the transformed depth image is the same as that of the first RGB image and the second RGB image; a first depth image at the current time point is obtained from the transformed depth image. For example, the depth sensor and the RGB image sensor can capture the same scene.

[0069] In this application, by fusing the outputs of various types of sensors (e.g., RGB image sensors, sensors that directly or indirectly measure angular velocity and acceleration) and using these outputs as constraints for joint optimization, an improved motion estimate is obtained, thereby further improving the accuracy of the estimated pixel motion velocity.

[0070] In this application, the accuracy of determining the Z coordinate can be improved by using a depth sensor, and more constraints can be provided, thereby further improving the accuracy of estimating the motion speed of pixels.

[0071] According to the example embodiment, the second function reflects the relationship between the coordinates of corresponding pixels in the first RGB image and the second RGB image and the first motion velocity of the corresponding pixels. For example, the second function can be expressed as the following formula 8.

[0072] (Formula 8) In Formula 8, and It can represent the X and Y coordinates of the pixel in the second RGB image that corresponds to the first pixel in the first RGB image representing a stationary object.

[0073] According to the example embodiment, the third function reflects the relationship between the first angular velocity and the derivative of the first rotation matrix. For example, the third function can be expressed as Equation 9 below.

[0074] (Formula 9) In Formula 9, , and It can represent the X, Y, and Z components of the first angular velocity.

[0075] According to the example embodiment, the fourth function reflects the relationship between the second translational velocity and the first acceleration and the first translational velocity. For example, the fourth function can be expressed as the following formula 10.

[0076] (Formula 10) In Formula 10, It can represent the first acceleration.

[0077] According to an example embodiment, when the electronic device includes an event sensor, it can also obtain a first event image at the current time point, and further combine a fifth function to determine the derivatives of the first motion speed, the first translation speed, and the first rotation matrix.

[0078] According to the example embodiment, the fifth function embodies the relationship between the spatial gradient of the first RGB image in the x and y directions, the first motion velocity of the pixels in the first RGB image, and the first event image. The fifth function can be expressed as the following formula 11.

[0079] (Formula 11) In Formula 11, E can represent the spatial gradient of the first RGB image in the x and y directions (e.g., values ​​obtained by taking the partial derivatives of the brightness determined based on the first RGB image with respect to x and y), and E can represent the pixel value of the first event image.

[0080] In one example, an event image captured by an event sensor of an electronic device can be transformed such that the transformed event image is time-synchronized with a first RGB image and a second RGB image, the transformed event image corresponds to an image coordinate system, and the resolution of the transformed event image is the same as that of the first RGB image and the second RGB image; a first event image at the current time point is obtained from the transformed event image. For example, the event sensor and the RGB image sensor can capture the same scene.

[0081] In this application, by introducing a high temporal resolution event sensor, changes in image frames can be perceived more sensitively, and more constraints can be provided, thereby further improving the accuracy of estimating pixel motion speed.

[0082] According to the example embodiment, the pixel coordinates of the first pixel representing the stationary object in the first RGB image (e.g., the coordinates of the first pixel representing the stationary object), the Z coordinate of the stationary object at the previous time point, the second translational velocity, and the aforementioned time length can be substituted into the first function (or the pixel coordinates of the first pixel representing the stationary object in the first RGB image, the Z coordinate of the first depth image, and the corresponding partial derivative values ​​can be substituted into the first function), the pixel coordinates of the corresponding pixels in the first RGB image and the second RGB image can be substituted into the second function, the first angular velocity can be substituted into the third function, the second translational velocity, the first acceleration, and the aforementioned time length can be substituted into the fourth function, and the first function to the fourth function can be combined (Formula 5 or 7, Formula 8, Formula 9, and Formula 10) to determine the first motion velocity, the first translational velocity, and the derivative of the first rotation matrix.

[0083] According to the example embodiment, the pixel coordinates of the first pixel representing the stationary object in the first RGB image, the Z coordinate of the stationary object at the previous time point, the second translational velocity, and the aforementioned time length can be substituted into the first function (or the pixel coordinates of the first pixel representing the stationary object in the first RGB image, the Z coordinate of the first depth image, and the corresponding partial derivative values ​​can be substituted into the first function), the pixel coordinates of the corresponding pixels in the first RGB image and the second RGB image can be substituted into the second function, the first angular velocity can be substituted into the third function, the second translational velocity, the first acceleration, and the aforementioned time length can be substituted into the fourth function, the spatial gradient of the first RGB image in the x and y directions and the pixel values ​​of the first event image can be substituted into the fifth function, and the first to fifth functions above (Equation 5 or 7, Equation 8, Equation 9, Equation 10 and Equation 11) can be combined to determine the first motion velocity, the first translational velocity, and the derivative of the first rotation matrix.

[0084] According to the example embodiment, in the process of solving the above first to fourth functions simultaneously, the first to fourth functions can be converted into first to fourth cost functions respectively, and adjustable first to fourth weights can be applied to the first to fourth cost functions to determine the total cost function. Finally, the derivatives of the first motion velocity, the first translation velocity, and the first rotation matrix are determined by minimizing the total cost function.

[0085] According to the example embodiment, in the process of solving the above first to fifth functions simultaneously, the first to fifth functions can be converted into first to fifth cost functions respectively, and adjustable first to fifth weights can be applied to the first to fifth cost functions to determine the total cost function. Finally, the derivatives of the first motion velocity, the first translation velocity, and the first rotation matrix are determined by minimizing the total cost function.

[0086] In one example, for each of the first through fifth functions, the unknowns involved in the function (e.g., , By moving at least one term in the function and the term in the function that does not involve unknowns to both sides of the equal sign, the distance between the term in the function that involves unknowns and the term in the function that does not involve unknowns can be determined as the cost function of the function, and the reciprocal of the variance of the term that does not involve unknowns can be determined as the weight to be applied to the cost function.

[0087] For example, for the fifth function, the terms involving unknowns in the fifth function can be... ) and the terms in the fifth function that do not involve unknowns ( The distance between them is determined as the fifth cost function. And the fifth weight can be determined as The reciprocal of the variance. For example, it can be determined by pre-calibrating the event sensor. The variance.

[0088] In one example, the first motion velocity of pixels in the first RGB image, as determined above, can be used to deblur a third RGB image (e.g., the (i+1)th frame image) captured by an RGB image sensor at a later time point. For example, It can represent the velocity of motion in the X-axis direction. It can represent the velocity of motion in the Y-axis direction.

[0089] In one example, the second motion speed of the pixels in the second RGB image can be used to deblur the first RGB image, and step S140 can be performed using the deblurred first RGB image.

[0090] In one example, for an initial RGB image captured at the start time point, the translational velocity of the electronic device at the start time point can be determined based on the acceleration measured at the start time point (or using other known methods), and the motion velocity of the pixels in the initial RGB image can be uncertain.

[0091] Figure 3 This is a flowchart illustrating in more detail an image motion estimation method according to one or more embodiments.

[0092] Figure 3 This shows the case of combining the first to fifth functions (Equations 7, 8, 9, 10, and 11).

[0093] like Figure 3 As shown, in step S210, an RGB image can be captured by the RGB image sensor of the electronic device, and a first RGB image captured at the current time point can be obtained.

[0094] In step S220, the first angular velocity at the current time point can be measured by the gyroscope sensor of the electronic device.

[0095] In step S230, the first acceleration at the current time point can be measured by the acceleration sensor of the electronic device.

[0096] In step S240, a depth image can be captured by the depth sensor of the electronic device.

[0097] In step S250, an event image can be captured by the event sensor of the electronic device.

[0098] In step S211, the spatial gradients of the first RGB image in the x and y directions can be obtained.

[0099] In step S212, the pixel coordinates of corresponding pixels in the first RGB image and the second RGB image can be obtained.

[0100] In step S213, the pixel coordinates of the first pixel representing the stationary object in the first RGB image can be obtained.

[0101] In step S231, the second translational velocity of the electronic device at the previous time point can be obtained.

[0102] In step S241, a first depth image can be obtained.

[0103] In one example, a transformation can be performed on a depth image captured by the depth sensor of an electronic device, such that the transformed depth image is synchronized in time with a first RGB image and a second RGB image, the transformed depth image corresponds to an image coordinate system, and the resolution of the transformed depth image is the same as that of the first RGB image and the second RGB image; a first depth image at the current time point is obtained from the transformed depth image.

[0104] In step S251, the first event image can be obtained.

[0105] In one example, an event image captured by an event sensor of an electronic device can be transformed such that the transformed event image is synchronized in time with a first RGB image and a second RGB image, the transformed event image corresponds to an image coordinate system, and the resolution of the transformed event image is the same as that of the first RGB image and the second RGB image; a first event image at the current time point is obtained from the transformed event image.

[0106] In step S310, the pixel coordinates of the first pixel representing the stationary object in the first RGB image and the Z coordinates of the first depth image can be substituted into the first function.

[0107] In step S320, the pixel coordinates of corresponding pixels in the first RGB image and the second RGB image can be substituted into the second function.

[0108] In step S330, the first angular velocity can be substituted into the third function.

[0109] In step S340, the second translational velocity and the first acceleration can be substituted into the fourth function.

[0110] In step S350, the spatial gradients of the first RGB image in the x and y directions and the pixel values ​​of the first event image can be substituted into the fifth function.

[0111] In step S410, the first to fifth functions can be combined to determine the first motion velocity and the first translation velocity.

[0112] Figure 4 A block diagram of an electronic device according to some example embodiments is shown.

[0113] Electronic devices according to various exemplary embodiments of this disclosure may be, or may include, at least one of a camera, smartphone, tablet PC, personal digital assistant (PDA), portable multimedia player (PMP), augmented reality device, virtual reality device, and various wearable devices (e.g., smartwatch, smart glasses, smart bracelet, etc.). However, the exemplary embodiments are not limited thereto, and electronic devices conceived in this invention may be any electronic device with image processing capabilities.

[0114] like Figure 4 As shown, the electronic device 100 according to some example embodiments includes at least a sensor module 110 and a processor 120. Furthermore, the electronic device 100 may also include a memory unit (not shown).

[0115] Sensor module 110 can capture images of an object to obtain multiple sensor images. Sensor module 110 may include an RGB image sensor. Sensor module 110 may also include at least one of a gyroscope sensor, an accelerometer sensor, a depth sensor, and an event sensor.

[0116] The processor 120 can receive captured sensor images from the sensor module 110 and perform image processing on the sensor images. The processor 120 can also perform reference... Figures 1 to 3 Describes an image motion estimation method.

[0117] The memory unit can acquire images from external sources or receive images captured by the sensor module. The memory unit can store images required for performing image processing tasks.

[0118] The memory unit may also store data and / or software or instructions for implementing image motion estimation methods according to some example embodiments. When the processor 120 executes the software or instructions, the image motion estimation methods according to some example embodiments can be implemented. The memory unit may be implemented as part of the processor 120, or may be implemented within the electronic device 100 separately from the processor 120.

[0119] The processor 120 can be implemented as hardware such as a general-purpose processor, an application processor (AP), an integrated circuit dedicated to image processing, a field-programmable gate array, or a combination of hardware and software.

[0120] Figure 5 A block diagram of an electronic device according to another example embodiment is shown.

[0121] like Figure 5As shown, in one or more embodiments, the electronic device 200 includes a sensor unit 210, at least one processor 220, a communication unit 230, an input unit 240, a storage unit 250, and a display unit 260.

[0122] Sensor unit 210 is connected to processor 220. Processor 220 can execute the above-mentioned parameters. Figures 1 to 3 Describes an image motion estimation method.

[0123] The communication unit 230 can perform communication operations of the electronic device. The communication unit 230 can establish a communication channel to a communication network and / or perform communications associated with, for example, image processing.

[0124] The input unit 240 is configured to receive various input information and various control signals, and to send the input information and control signals to the processor 220. The input unit 240 can be implemented by various input devices such as a touch screen; however, one or more embodiments are not limited thereto.

[0125] Storage unit 250 may include volatile memory and / or non-volatile memory. Storage unit 250 may store various data generated and used by the electronic device. For example, storage unit 250 may store operating systems and applications (e.g., applications associated with the methods of this disclosure) used to control the operation of the electronic device. Processor 220 may control the overall operation of the electronic device and may control some or all of the internal components of the electronic device. Processor 220 may be implemented as a general-purpose processor, application processor (AP), application-specific integrated circuit, field-programmable gate array, etc., but one or more embodiments are not limited thereto.

[0126] The devices, units, modules, and other components described herein are implemented by hardware components. Examples of hardware components that can be used to perform the operations described herein include, where appropriate, controllers, cameras, generators, drivers, memories, comparators, arithmetic logic units, adders, subtractors, multipliers, dividers, integrators, and any other electronic components configured to perform the operations described herein. In other examples, one or more hardware components performing the operations described herein are implemented by computing hardware (e.g., by one or more processors or computers). The processor or computer may be implemented by one or more processing elements, such as logic gate arrays, controllers and arithmetic logic units, digital signal processors, microcomputers, programmable logic controllers, field-programmable gate arrays, programmable logic arrays, microprocessors, or any other means or combination of means configured to respond to and execute instructions in a defined manner to achieve a desired result. In one example, the processor or computer includes or is connected to one or more memories storing instructions or software executed by the processor or computer. The hardware components implemented by the processor or computer can execute instructions or software (such as an operating system (OS) and one or more software applications running on the OS) for performing the operations described herein. Hardware components may also access, manipulate, process, create, and store data pages in response to the execution of instructions or software. For simplicity, the singular terms "processor" or "computer" may be used in the description of the examples described in this application; however, in other examples, multiple processors or computers may be used, or a processor or computer may include multiple processing elements, or multiple types of processing elements, or both. For example, a single hardware component, or two or more hardware components, may be implemented by a single processor, or two or more processors, or a processor and a controller. One or more hardware components may be implemented by one or more processors, or a processor and a controller, and one or more other hardware components may be implemented by one or more other processors, or another processor and another controller. One or more processors, or a processor and a controller, may implement a single hardware component, or two or more hardware components. Hardware components may have any one or more different processing configurations, examples of which include: a single processor, a standalone processor, a parallel processor, single-instruction single-data-page (SISD) multiprocessing, single-instruction multiple-data-page (SIMD) multiprocessing, multiple-instruction single-data-page (MISD) multiprocessing, and multiple-instruction multiple-data-page (MIMD) multiprocessing.

[0127] The methods for performing the operations described in this application are executed by computing hardware (e.g., by one or more processors or a computer), which is implemented to execute instructions or software as described above to perform the operations performed by the methods described in this application. For example, a single operation, or two or more operations, may be executed by a single processor, or two or more processors, or a processor and a controller. One or more operations may be executed by one or more processors, or a processor and a controller, and one or more other operations may be executed by one or more other processors, or another processor and another controller. One or more processors, or a processor and a controller, may execute a single operation, or two or more operations.

[0128] Instructions or software for controlling a processor or computer to implement hardware components and perform the methods described above can be written as computer programs, code segments, instructions, or any combination thereof to individually or collectively instruct or configure the processor or computer to operate as a machine or special-purpose computer to perform operations performed by the hardware components and methods described above. In one example, the instructions or software include machine code (such as machine code generated by a compiler) that is directly executed by the processor or computer. In another example, the instructions or software include high-level code that is executed by the processor or computer using an interpreter. Those skilled in the art can readily write instructions or software based on the block diagrams and flowcharts shown in the accompanying drawings and the corresponding descriptions in the specification, which disclose algorithms for performing operations performed by the hardware components and methods described above.

[0129] Instructions or software used to control a processor or computer to implement hardware components and perform the methods described above, along with any associated data pages, data page files, and data page structures, are recorded, stored, or fixed in, or on, one or more non-transitory computer-readable storage media. Examples of non-transitory computer-readable storage media include: read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disc storage devices, hard disk drives (HDDs), solid-state drives (SSDs), flash memory, card storage (such as multimedia cards or microcards (e.g., Secure Digital (SD) or Extreme Digital (XD))), magnetic tape, floppy disks, magneto-optical data page storage devices, optical data page storage devices, hard disks, solid-state drives, and any other device configured to store instructions or software and any associated data pages, data page files, and data page structures in a non-transitory manner and to provide instructions or software and any associated data pages, data page files, and data page structures to a processor or computer, enabling the processor or computer to execute the instructions.

[0130] Although this disclosure has been specifically shown and described with reference to exemplary embodiments thereof, those skilled in the art should understand that various changes in form and detail may be made therein without departing from the spirit and scope of this disclosure as defined by the claims.

Claims

1. An image motion estimation method, comprising: Obtain a first RGB image captured at the current time point and a second RGB image captured at the previous time point using an RGB image sensor of an electronic device; Obtain the first angular velocity of the electronic device at the current time point; Obtain the first acceleration of the electronic device at the current point in time; Based on the first RGB image, the second RGB image, the second translational velocity of the electronic device at the previous time point, the first angular velocity of the electronic device at the current time point, and the first acceleration of the electronic device at the current time point, the first motion velocity of a plurality of first pixels in the first RGB image and the first translational velocity of the electronic device at the current time point are determined to estimate the image motion.

2. The image motion estimation method according to claim 1 further includes: Based on the first motion velocity of the plurality of first pixels in the first RGB image, the third RGB image captured by the RGB image sensor at a later time point is deblurred.

3. The image motion estimation method according to claim 1, wherein, The steps for determining the first motion velocity and the first translational velocity include: Based on the first RGB image, the second RGB image, the second translational velocity, the first angular velocity, and the first acceleration, a first function, a second function, a third function, and a fourth function are combined to determine the first motion velocity, the first translational velocity, and the derivative of the first rotation matrix at the current time point. The first function embodies the relationship between the derivative of the camera coordinates corresponding to the first pixel representing the stationary object among the plurality of first pixels, and the derivative of the camera coordinates, the first rotation matrix, and the first translation velocity. The stationary object is the object that remains stationary during the time interval between the previous time point and the current time point, among the corresponding objects in the first RGB image and the second RGB image. The second function embodies the relationship between the coordinates of the plurality of first pixels in the first RGB image and the coordinates of the plurality of second pixels in the second RGB image corresponding to the plurality of first pixels in the first RGB image, and the first motion velocity of the plurality of first pixels in the first RGB image. The third function reflects the relationship between the first angular velocity and the derivative of the first rotation matrix. The fourth function reflects the relationship between the second translational velocity and the first acceleration and the first translational velocity.

4. The image motion estimation method according to claim 3, in, In the first function, the product of the derivative of the first rotation matrix and the first translation velocity with the camera coordinates is equal to the derivative of the camera coordinates. The camera coordinates are obtained by substituting the coordinates of the first pixel representing the stationary object in the first RGB image and the Z-coordinate of the stationary object at the current time point into the intrinsic parameter matrix. The Z-coordinate of the stationary object at the current time point is obtained based on the second translational velocity, the Z-coordinate of the stationary object at the previous time point, and the time interval between the previous time point and the current time point. The derivative of the camera coordinates includes the first motion velocity of the first pixel representing the stationary object.

5. The image motion estimation method according to claim 3 further includes: Obtain the first depth image at the current time point. In the first function, the product of the derivative of the first rotation matrix and the first translation velocity with the camera coordinates is equal to the derivative of the camera coordinates. The camera coordinates are obtained by substituting the coordinates of the first pixel representing the stationary object in the first RGB image and the Z-coordinate of the stationary object at the current time point into the intrinsic parameter matrix. The Z-coordinate of the stationary object at the current time point is obtained based on the first depth image. The derivative of the camera coordinates includes the first motion velocity of the first pixel representing the stationary object.

6. The image motion estimation method according to claim 5, wherein, The steps to obtain the first depth image at the current time point include: A transformation is performed on the depth image captured by the depth sensor of the electronic device, such that the transformed depth image is synchronized in time with the first RGB image and the second RGB image, wherein the transformed depth image corresponds to the image coordinate system, and wherein the resolution of the transformed depth image is the same as the resolution of each of the first RGB image and the second RGB image; Obtain the first depth image at the current time point from the transformed depth image. The depth sensor and the RGB image sensor capture the same scene.

7. The image motion estimation method according to claim 3 further includes: Obtain the image of the first event at the current time point; as well as Further, combine the fifth function, The fifth function embodies the relationship between the spatial gradient of the first RGB image in the x and y directions, the first motion velocity of the pixels in the first RGB image, and the first event image.

8. The image motion estimation method according to claim 7, wherein, The steps to obtain the first event image at the current time point include: A transformation is performed on an event image captured by an event sensor of an electronic device, such that the transformed event image is synchronized in time with a first RGB image and a second RGB image, wherein the transformed event image corresponds to an image coordinate system, and wherein the resolution of the transformed event image is the same as the resolution of each of the first RGB image and the second RGB image; Obtain the first event image at the current time point from the transformed event image. The event sensor and the RGB image sensor capture the same scene.

9. An electronic device, comprising An RGB image sensor is configured to capture a first RGB image at the current time point and a second RGB image at the previous time point; The processor is configured to: acquire a first RGB image and a second RGB image; acquire the first angular velocity of the electronic device at the current time point; Obtain the first acceleration of the electronic device at the current time point; based on the first RGB image, the second RGB image, the second translational velocity of the electronic device at the previous time point, the first angular velocity of the electronic device at the current time point, and the first acceleration of the electronic device at the current time point, determine the first motion velocity of a plurality of first pixels in the first RGB image and the first translational velocity of the electronic device at the current time point to estimate image motion.

10. A non-transitory computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to perform the method according to claim 1.

Images