Gyroscope alignment method and device, electronic device, and storage medium

Abstract

The present disclosure relates to a gyro alignment method and device, electronic equipment and storage medium. The method comprises: acquiring a plurality of image frames collected by a mobile terminal during a back-and-forth movement along a single axis, and a plurality of gyro parameters detected during the process; the plurality of image frames are collected by a camera module of the mobile terminal; determining a time delay of the gyro compared with the camera module based on the plurality of image frames and the plurality of gyro parameters, and performing time alignment of the gyro and the camera module based on the time delay.

Description

Technical Field

[0001] This disclosure relates to the field of image processing, and more particularly to a gyroscope alignment method and apparatus, electronic device, and storage medium. Background Technology

[0002] In image processing, image stabilization is required based on gyroscope parameters to avoid problems such as image ghosting in the acquired images. Therefore, before image acquisition, it is necessary to prioritize time alignment between the gyroscope and the device's camera module to ensure that image stabilization is performed on the camera module's exposure operation at the corresponding moment based on the gyroscope parameters at the same instant.

[0003] However, when aligning the time of a gyroscope and an image sensor using a rolling shutter in related technologies, a large number of complex calculations are required, resulting in poor alignment, low alignment efficiency, and high processing resource consumption. Summary of the Invention

[0004] This disclosure provides a gyroscope alignment method, apparatus, electronic device, and storage medium, which can achieve rapid alignment of gyroscopes and camera modules.

[0005] According to a first aspect of this disclosure, a gyroscope alignment method is provided, comprising:

[0006] The system acquires multiple image frames captured during the reciprocating motion of a mobile terminal along a single axis, as well as multiple gyroscope parameters detected during the process; the multiple image frames are acquired by the camera module of the mobile terminal.

[0007] The delay of the gyroscope relative to the camera module is determined based on the plurality of image frames and the plurality of gyroscope parameters, so as to perform time alignment of the gyroscope and the camera module based on the delay.

[0008] According to a second aspect of this disclosure, a gyroscope alignment device is provided, comprising:

[0009] The acquisition unit acquires multiple image frames collected during the reciprocating movement of the mobile terminal along a single axis, as well as multiple gyroscope parameters detected during the process; the multiple image frames are acquired by the camera module of the mobile terminal.

[0010] An alignment unit determines the delay of the gyroscope relative to the camera module based on the plurality of image frames and the plurality of gyroscope parameters, and performs time alignment of the gyroscope and the camera module based on the delay.

[0011] According to a third aspect of this disclosure, an electronic device is provided, comprising:

[0012] processor;

[0013] Memory used to store processor-executable instructions;

[0014] The processor implements the method as described in the first aspect by running the executable instructions.

[0015] According to a fourth aspect of this disclosure, a computer-readable storage medium is provided that stores computer instructions thereon, which, when executed by a processor, implement the steps of the method as described in the first aspect.

[0016] In the technical solution disclosed herein, a mobile terminal can be controlled to move back and forth along a single axis. During this process, the mobile terminal can acquire multiple image frames through a camera module and obtain multiple gyroscope parameters from a gyroscope. Based on this, the delay of the gyroscope relative to the camera module can be determined according to the acquired multiple image frames and the obtained multiple gyroscope parameters, and then the time correspondence between the gyroscope and the camera module can be performed based on this delay.

[0017] It should be understood that, since this disclosure enables the mobile terminal to move back and forth along a single axis, it can greatly simplify the calculation logic of subsequent time alignment, avoiding the problems of low alignment efficiency and high processing resource consumption caused by simulating actual suspension jitter for time alignment in related technologies. Attached Figure Description

[0018] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0019] Figure 1 This is a flowchart illustrating an exemplary embodiment of the present disclosure of a gyroscope alignment method;

[0020] Figure 2 This is a flowchart illustrating another gyroscope alignment method according to an exemplary embodiment of this disclosure;

[0021] Figure 3 This is a schematic diagram illustrating a smartphone reciprocating along a single axis, as shown in an exemplary embodiment of this disclosure;

[0022] Figure 4 This is a schematic diagram illustrating the exposure process of a rolling shutter according to an exemplary embodiment of the present disclosure;

[0023] Figure 5 This is a block diagram illustrating a gyroscope alignment device according to an exemplary embodiment of the present disclosure;

[0024] Figure 6 This is a schematic diagram of the structure of an electronic device according to an exemplary embodiment of the present disclosure. Detailed Implementation

[0025] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0026] The terminology used in this disclosure is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. The singular forms “a,” “the,” and “the” as used in this disclosure and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any and all possible combinations of one or more of the associated listed items.

[0027] It should be understood that although the terms first, second, third, etc., may be used in this disclosure to describe various information, such information should not be limited to these terms. These terms are used only to distinguish information of the same type from one another. For example, without departing from the scope of this disclosure, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0028] In the field of image processing, image stabilization is performed during image acquisition to avoid problems such as image ghosting. This stabilization operation usually relies on the gyroscope installed in the mobile terminal. Therefore, before the mobile terminal leaves the factory, it is usually necessary to align the gyroscope and the camera module of the mobile terminal in time to ensure that during image acquisition, image stabilization is performed on the image acquired at the corresponding moment based on the gyroscope parameters at the same moment, thereby avoiding problems such as poor image stabilization effect.

[0029] In related technologies, when aligning the time of the gyroscope and camera module, the process of a mobile terminal capturing images is usually simulated, and the time alignment is performed based on the images and gyroscope parameters collected during this process.

[0030] For example, when the mobile terminal is a smartphone, one can hold the smartphone to take pictures and intentionally shake it to obtain multiple image frames and gyroscope parameters during the shaking process. Based on this, the delay of the gyroscope relative to the camera module can be calculated based on the image parameters of multiple image frames and the acquired gyroscope parameters, in order to perform time alignment.

[0031] When using related technologies for time alignment, the movement of mobile terminals is completely irregular. Therefore, the obtained image parameters, such as the offset between image frames, and gyroscope parameters, such as angular velocity, are quite complex. Consequently, the algorithms used for time alignment also need to incorporate complex logic such as direction determination. It is evident that the gyroscope alignment methods in related technologies suffer from problems such as low alignment efficiency and high consumption of processing resources.

[0032] To address this, this disclosure proposes a gyroscope alignment method. When using this method for time alignment, the mobile terminal is controlled to move back and forth along a single axis. During this process, multiple image frames can be acquired through the mobile terminal's camera module, and multiple gyroscope parameters can be obtained from the gyroscope. Based on these, the delay of the gyroscope relative to the camera module can be determined using the acquired image frames and gyroscope parameters, and time alignment of the gyroscope and camera module can be performed based on this delay.

[0033] It should be understood that time alignment between the gyroscope and camera module is actually based on the offset of the acquired image and the displacement of the gyroscope. The timestamps of the camera module and gyroscope are aligned, essentially aligning spatial parameters with temporal parameters. Based on this, it's easy to understand that movement along all axes in the spatial dimension has a consistent impact on the temporal dimension, whether for the image or the device. Therefore, even limiting displacement to a single axis will not affect the time alignment operation (in other words, completing time alignment on a single axis is equivalent to completing time alignment on all axes). However, limiting the axis significantly simplifies the subsequent alignment calculations because it eliminates the need to consider multi-directional issues, improving alignment efficiency and reducing the resource consumption of the alignment algorithm.

[0034] As can be seen, the gyroscope alignment method disclosed herein can avoid the problems of low alignment efficiency and excessive processing resources caused by the need to use complex alignment algorithms to perform time alignment of the gyroscope and camera module in related technologies.

[0035] The following describes each step included in the technical solution disclosed herein.

[0036] Figure 1 This disclosure illustrates a gyroscope alignment method as an exemplary embodiment. Figure 1 As shown, the method may include the following steps:

[0037] Step 102: Acquire multiple image frames collected during the reciprocating movement of the mobile terminal along a single axis, as well as multiple gyroscope parameters detected during the process; the multiple image frames are acquired by the camera module of the mobile terminal.

[0038] In this disclosure, the mobile terminal can be controlled to move back and forth along any axis. For example, the mobile terminal can be controlled to move back and forth along the horizontal axis to acquire multiple image frames and multiple gyroscope parameters during the back and forth movement along the horizontal axis. As another example, the mobile terminal can be controlled to move back and forth along the axis of gravity to acquire multiple image frames and multiple gyroscope parameters during the back and forth movement along that axis.

[0039] In this disclosure, the movement of a mobile terminal can be maintained along a single axis in a variety of ways. For example, one side of the mobile terminal can be moved back and forth against a plane so that it cannot move along an axis perpendicular to that plane; another example is that a device with a groove containing only a single axis can be designed, so that the mobile terminal can be moved along a single axis simply by inserting it into the groove.

[0040] Of course, the above examples are merely illustrative. The specific axis on which the mobile terminal moves back and forth, and how to ensure that the movement of the mobile terminal is on a single axis, can be determined by those skilled in the art based on the actual situation. This disclosure does not impose any restrictions on this.

[0041] Step 104: Determine the delay of the gyroscope relative to the camera module based on the plurality of image frames and the plurality of gyroscope parameters, so as to perform time alignment of the gyroscope and the camera module based on the delay.

[0042] In this disclosure, after acquiring multiple image frames through the camera module and obtaining multiple gyroscope parameters, the delay of the gyroscope relative to the camera module can be determined.

[0043] In one embodiment, feature point matching can be performed on multiple image frames first to determine the offset of the same feature point between different image frames based on the matching results. On this basis, a target gyroscope parameter that matches the offset can be determined from multiple gyroscope parameters. Then, based on the acquisition time of the target gyroscope parameter and the acquisition time of the different image frames, the delay of the gyroscope relative to the camera module can be determined.

[0044] It should be understood that since both image frames and gyroscope parameters are acquired during the back-and-forth movement of the mobile terminal, if the acquisition time of the image frame and the acquisition time of the gyroscope parameters are consistent on the objective timeline, the offset of the same feature point in different image frames must match the displacement calculated from the gyroscope parameters acquired at the acquisition time of those different image frames (on the objective timeline). Therefore, this embodiment, through reverse deduction, shows that the target gyroscope parameter that matches the offset determined from multiple gyroscope parameters usually represents the gyroscope parameter acquired at the acquisition time of the image frame (on the objective timeline) corresponding to that offset. Based on this, and based on the acquisition time of the determined target gyroscope parameter and the acquisition time of the image frame, the delay of the gyroscope relative to the camera module can obviously be determined. For example, the timestamps of the camera module acquiring the aforementioned different image frames and the timestamps of the gyroscope acquiring the determined target gyroscope parameters can be obtained, and the difference between the two can be calculated as the delay of the gyroscope relative to the camera module.

[0045] In this embodiment, the acquired multiple image frames may include at least one set of adjacent frames. Therefore, when performing feature point matching, adjacent frames can be matched. In other words, feature point matching can be performed on at least one set of adjacent frames in the multiple image frames to determine the offset of the same feature point between the at least one set of adjacent frames based on the matching results. Based on this, at least one set of target gyroscope parameters matching the offset of the at least one set of adjacent frames can be determined from multiple gyroscope parameters. The delay of the gyroscope relative to the camera module can then be determined based on the acquisition time of the at least one set of target gyroscope parameters and the acquisition time of the at least one set of adjacent frames. It is worth noting that the "offset of the same feature point between at least one set of adjacent frames" refers to the offset of the same feature point between two image frames contained in a set of adjacent frames. This can be achieved by determining the offset of only one set of adjacent frames or by determining the offset of each set of adjacent frames in multiple sets of adjacent frames.

[0046] Of course, the determination of the offset between adjacent frames here is only illustrative. For example, after acquiring multiple image frames, the offset between the first frame and the last frame can be calculated to determine the delay of the gyroscope relative to the camera module. The specific determination of which image frames among the multiple image frames are offset can be determined by those skilled in the art according to actual needs, and this disclosure does not limit this.

[0047] In this embodiment, since the displacement of the physical object and the offset of the pixel on the image sensor are related to the focal length of the camera to which the image sensor belongs, when determining the target gyroscope parameter that matches the offset from multiple gyroscope parameters, the offset can be converted into the displacement of the mobile terminal based on the focal length of the camera to which the image sensor in the camera module belongs. On this basis, the target gyroscope parameter used to calculate the displacement can be determined from multiple gyroscope parameters.

[0048] In this embodiment, when determining the target gyroscope parameter matching the offset from multiple gyroscope parameters, the determination can also be based on the inter-frame motion velocity. For example, after determining the displacement of the same feature point between different image frames, the inter-frame motion velocity between multiple image frames can be determined based on the displacement and the exposure time of each image frame. On this basis, the target gyroscope parameter matching the inter-frame motion velocity can be determined from multiple gyroscope parameters. For example, since the gyroscope parameters acquired by the gyroscope include angular velocity, the velocity at each acquisition time can be obtained by integration, and the acquired velocity can be compared with the inter-frame motion velocity. Then, the gyroscope parameter corresponding to the inter-frame motion velocity is used as the determined target gyroscope parameter.

[0049] In another embodiment, after acquiring multiple image frames, the feature values ​​of each image frame can be determined first, and feature value matching can be performed between different image frames to obtain the offset of the same feature value between different image frames. Based on this, the exposure time of different image frames corresponding to the same feature value and the acquisition time of multiple gyroscope parameters can be obtained. Based on the exposure time, acquisition time, offset, and displacement represented by multiple gyroscope parameters, the delay of the gyroscope relative to the camera module can be determined.

[0050] For example, this embodiment can determine the exposure time of at least one group of adjacent frames in multiple image frames based on the same feature value, so as to determine the undetermined exposure time of the camera module based on the exposure time; and calculate the undetermined focal length and undetermined delay of the camera module based on the above offset and multiple gyroscope parameters.

[0051] For example, when calculating the exposure time, the pixel corresponding to the same feature value in an adjacent image frame can be determined, and the exposure time of that pixel can be used as the exposure time of the entire image frame. The exposure duration of the camera module can then be calculated based on the exposure times of adjacent frames. When calculating the undetermined focal length, it can be assumed that the gyroscope and the camera module have completed time alignment. Based on this, the initial focal length of the camera module can be calculated based on the offset of adjacent image frames obtained at a certain timestamp and the gyroscope parameters collected at the same timestamp, which can then be used as the undetermined focal length. The undetermined delay can be determined based on the method of calculating the inter-frame motion speed and obtaining the angular velocity from the gyroscope parameters.

[0052] After obtaining the undetermined exposure time, undetermined focal length, and undetermined delay, a loss function can be constructed based on the undetermined exposure time, undetermined focal length, and undetermined delay to characterize the difference in feature values ​​between adjacent image frames. The loss function is then iteratively optimized by different time steps until it converges. The time step that makes the loss function converge is then determined as the delay of the gyroscope relative to the camera module.

[0053] Of course, the above method of constructing the loss function is only illustrative. How to construct the loss function and how to iteratively optimize and determine the delay can be determined by those skilled in the art according to actual needs. This disclosure does not impose any restrictions on this.

[0054] In this disclosure, the camera module can use different exposure methods for image acquisition. For example, it can use a global shutter or a rolling shutter for image acquisition.

[0055] In the case of using a rolling shutter for image exposure, since the rolling shutter exposes the image line by line, feature point matching can be performed on the same line of different image frames to determine the feature points corresponding to each line between two image frames. Similarly, feature value matching can also be performed on the same line of different image frames to determine the feature values ​​corresponding to each line between two image frames.

[0056] It is worth noting that this disclosure can select a preset range in different image frames and perform feature point or feature value matching only within that preset range, thereby reducing the computational load of feature value matching. For example, the region within a preset width in the middle of the image frame can be used as the aforementioned preset range for feature value matching.

[0057] It should be stated that the implementing entity of the technical solution disclosed herein can be any type of mobile terminal. For example, it can be a traditional mobile terminal such as a smartphone or tablet computer, or a new type of mobile terminal such as a smart car or robot. It should be understood that any mobile terminal equipped with a camera module and a gyroscope can be used as the mobile terminal in this disclosure. The specific type of mobile terminal used as the implementing entity of the technical solution disclosed herein can be determined by those skilled in the art based on actual needs, and this disclosure does not impose any restrictions on this.

[0058] As can be seen from the above technical solution, when aligning the gyroscope, this disclosure can limit the mobile terminal to move back and forth on a single axis, so as to perform time alignment of the gyroscope and camera module based on the multiple image frames collected by the mobile terminal during the movement and the obtained gyroscope parameters.

[0059] It should be understood that, since this disclosure limits the mobile terminal to only move back and forth on a single axis during gyroscope alignment, the gyroscope parameters obtained by this disclosure are relatively simple. Based on this, this disclosure does not need to use a complex alignment algorithm to perform time alignment based on gyroscope parameters and image frames, thus avoiding the problem of low alignment efficiency caused by the need to use a complex alignment algorithm for gyroscope alignment in related technologies.

[0060] The following section will use "gyroscope alignment for smartphones" as an example to introduce the technical solution disclosed herein.

[0061] Figure 2 A flowchart illustrating another gyroscope alignment method as an exemplary embodiment of this disclosure. Figure 2 As shown, the method includes the following steps:

[0062] Step 201: Place the bottom edge of the smartphone against the table and move it back and forth along a single axis.

[0063] In this embodiment, before the smartphone leaves the factory, an assembly worker can press the bottom edge of the smartphone firmly against a tabletop to prevent it from wobbling in the direction perpendicular to the tabletop and control its reciprocating movement along a single axis of the tabletop. For example, it can be done as follows: Figure 3 The movement is shown in the diagram, and it moves back and forth.

[0064] Step 202: Activate the smartphone's video recording function to continuously capture multiple image frames.

[0065] In this embodiment, when the smartphone is in a back-and-forth motion, its video recording function can be activated to capture multiple image frames. For example, the assembly worker can pre-activate the video recording function before starting to move, so as to take continuous pictures through the camera module; or, for another example, the smartphone's camera application can include a timed video recording function, in which case the assembly worker can activate the timed video recording function before moving the smartphone, and then the smartphone will automatically start recording after the movement of the smartphone begins.

[0066] Step 203: The smartphone determines the delay of the gyroscope relative to the camera module based on multiple captured image frames and gyroscope parameters obtained during movement.

[0067] In this embodiment, the smartphone acquires multiple image frames while moving, and simultaneously continuously acquires gyroscope parameters output by the gyroscope. Based on this, the gyroscope's delay relative to the camera module, i.e., Gyro Delay, can be determined according to a preset correction strategy.

[0068] For example, if a smartphone uses a rolling shutter for image exposure, this correction strategy can include two stages:

[0069] Phase 1: Determine three initial values ​​to be corrected: gyroscope delay, rolling shutter time, and focal length of the camera included in the camera module.

[0070] It is worth noting that the exposure time mentioned here specifically refers to the scanning time of the rolling shutter. The exposure process of the rolling shutter can be found in [reference needed]. Figure 4 This includes the exposure time frameSExp and the scan time RollingShutter time for each row.

[0071] This stage can include three steps:

[0072] Step ①: Extract the center region bounding box of adjacent frames and obtain the average grayscale value of each row of pixels within that bounding box, using it as the feature value Frame1D for that row. Based on this, the offset between adjacent frames can be measured using the feature difference. This can be done by calculating the offset for each row of adjacent frames and then integrating the results to obtain the offset for the entire image.

[0073] Step 2: By using different pixel step sizes, match the motion trajectory of the same feature value between adjacent frames to obtain the pixel difference patchMVPixel of the inter-frame motion. Combined with the timestamp frameDT of the adjacent frame at the time of acquisition, the linear velocity of the inter-frame motion can be obtained: pixelSpeed ​​= patchMVPixel / frameDT.

[0074] Step 3: Based on the relationship between angular velocity, linear velocity, and radius, and given the timestamp of each image frame acquired by the camera module and its corresponding gyroscope parameter FrameTimeGyro, the initial value of the focal length can be calculated: FocalLength = pixelSpeed ​​ / FrameTimeGyro.

[0075] Based on the obtained focal length and linear velocity, the gyroscope parameters corresponding to the inter-frame motion of the video can be further predicted: VideoGyro = pixelSpeed ​​ / Focal Length.

[0076] Step 4: Since the linear velocity measured based on adjacent image frames and the displacement velocity measured based on gyroscope parameters should match at the same time, the initial value of GyroDelay can be predicted by iterating through different time steps.

[0077] Step 5: The camera module can then calculate the initial value of the Rolling Shutter time (ROS) of a single frame image by acquiring two exposure times at adjacent frames. In actual acquisition of exposure times, the timestamp of the image exposure can be directly used as the exposure time of the corresponding image, or the intermediate time of the image exposure can be used as the exposure time of that image; this embodiment does not impose any restrictions on this.

[0078] The second stage involves constructing a loss function "to characterize the differences in feature values ​​between adjacent frames" based on the three initial values ​​to be corrected. This stage may include the following steps:

[0079] Step ①: Obtain the angular velocity and timestamp of the gyroscope when acquiring a single frame image, and calculate the displacement gyroPos based on the angular velocity and timestamp.

[0080] And, find the intermediate time of this image frame:

[0081] frameSTimeMid = frameSTime (exposure end time) - frameSExp (exposure duration per line) / 2 - Rolling Shutter / 2 + Gyro Delay;

[0082] Step 2: Calculate the Rolling Shutter time rowTimesBase for each row of pixels in the central region frame. Obtain the actual time for each row of this frame: rowTimes = rowTimesBase + frameSTimeMid[frmID];

[0083] Step 3: Construct the interpolation function gyroIntp for the midpoint time of each frame and the inter-frame rotation angle using frameSTimeMid and gyroPos, and calculate the rotation angle camAngles = gyroIntp(rowTimes) of the image frame at the midpoint time of each row.

[0084] Step 4: When the rotation angle is very small, the rotation angle corresponding to each row of pixels can be approximately calculated: rowAnglesBase = arctan(pixel[i] / Focal Length).

[0085] Step 5: Construct the interpolation function frame1DIntp using rowAnglesBase+camAngles and frame1D to calculate the feature values ​​corresponding to the pixel changes in each row: newFrame = frame1DIntp(rowAnglesBase)

[0086] Step 6: Define the feature value of the previous frame as oldFrame and the feature value of the next frame as newFrame. Then the feature value of the next frame mapped to the previous frame is:

[0087] NormFrame=(oldFrame-avg(oldFrame))*std(newFrame) / std(oldFrame)+avg(newFrame);

[0088] Step 7: Construct the loss function: loss = abs(NormFrame - newFrame);

[0089] The third stage involves iterating through the loss function with different step sizes to obtain the final values ​​of Gyro Delay, RollingShutter time, and Focal Length. For example, a global optimization algorithm (Basin Hopping) can be used for iteration.

[0090] As can be seen from the above technical solution, this disclosure enables the smartphone to move back and forth along a single axis during gyroscope alignment, so as to perform time alignment of the gyroscope and camera module based on multiple image frames collected by the mobile terminal during the movement and the obtained gyroscope parameters.

[0091] One approach is to first obtain the initial value of the gyroscope's delay, and then iteratively optimize it using a loss function with different timings until the loss function converges. At this point, the final, corrected delay value can be used as the gyroscope's delay relative to the camera module. In addition, iterative optimization can be used to obtain parameters such as the shutter's exposure time and the camera's focal length, avoiding the need to separately calculate these parameters.

[0092] Figure 5 This is a block diagram illustrating a gyroscope alignment device according to an exemplary embodiment of this disclosure. (Refer to...) Figure 5 The device includes an acquisition unit 501 and an alignment unit 502.

[0093] The acquisition unit acquires multiple image frames collected during the reciprocating movement of the mobile terminal along a single axis, as well as multiple gyroscope parameters detected during the process; the multiple image frames are acquired by the camera module of the mobile terminal.

[0094] An alignment unit determines the delay of the gyroscope relative to the camera module based on the plurality of image frames and the plurality of gyroscope parameters, and performs time alignment of the gyroscope and the camera module based on the delay.

[0095] Optionally, the alignment unit is further used for:

[0096] Feature point matching is performed on the multiple image frames, and the offset of the same feature point between different image frames is determined based on the matching results;

[0097] From the plurality of gyroscope parameters, a target gyroscope parameter matching the offset is determined, and the delay of the gyroscope relative to the camera module is determined based on the acquisition time of the target gyroscope parameter and the acquisition time of the different image frames.

[0098] Optionally, the alignment unit is further used for:

[0099] Feature point matching is performed on at least one group of adjacent frames in the plurality of image frames to determine the offset of the same feature point between the at least one group of adjacent frames based on the matching results;

[0100] From the plurality of gyroscope parameters, at least one set of target gyroscope parameters that match the offset of the at least one set of adjacent frames are determined, so as to determine the delay of the gyroscope relative to the camera module based on the acquisition time of the at least one set of target gyroscope parameters and the acquisition time of the at least one set of adjacent frames.

[0101] Optionally, the alignment unit is further used for:

[0102] Based on the focal length of the camera to which the image sensor in the camera module belongs, the offset is converted into the displacement of the mobile terminal;

[0103] From the plurality of gyroscope parameters, the target gyroscope parameter used to calculate the displacement is determined.

[0104] Optionally, the alignment unit is further used for:

[0105] Based on the displacement of the same feature point between different image frames and the exposure time of each image frame, the inter-frame motion speed between the multiple image frames is determined.

[0106] From the plurality of gyroscope parameters, a target gyroscope parameter that matches the inter-frame motion speed is determined.

[0107] Optionally, the alignment unit is further used for:

[0108] The feature values ​​of each image frame in the plurality of image frames are obtained, and feature value matching is performed between different image frames to obtain the offset of the same feature value between different image frames.

[0109] The exposure time of different image frames corresponding to the same feature value and the acquisition time of the multiple gyroscope parameters are obtained, so as to determine the delay of the gyroscope relative to the camera module based on the exposure time, the acquisition time, the offset, and the displacement represented by the multiple gyroscope parameters.

[0110] Optionally, the alignment unit is further used for:

[0111] Based on the same feature value, the exposure time of at least one group of adjacent frames in the plurality of image frames is determined, so as to determine the undetermined exposure duration of the camera module based on the exposure time; and, based on the offset and the plurality of gyroscope parameters, the undetermined focal length and the undetermined delay of the camera module are calculated.

[0112] Based on the undetermined exposure duration, the undetermined focal length, and the undetermined delay, a loss function is constructed to characterize the difference in feature values ​​between adjacent image frames. The loss function is then iteratively optimized through different time steps until it converges. The time step that causes the loss function to converge is then determined as the delay of the gyroscope relative to the camera module.

[0113] Optionally, the camera module uses a rolling shutter for image exposure.

[0114] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this disclosure according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0115] Accordingly, this disclosure also provides a gyroscope alignment device, comprising: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to implement a gyroscope alignment method as described in any of the above embodiments, for example, the method may include: acquiring multiple image frames collected during a mobile terminal's reciprocating movement along a single axis, and multiple gyroscope parameters detected during the process; the multiple image frames are acquired by a camera module of the mobile terminal; determining the delay of the gyroscope relative to the camera module based on the multiple image frames and the multiple gyroscope parameters, so as to perform time alignment of the gyroscope and the camera module based on the delay.

[0116] Accordingly, this disclosure also provides an electronic device, the electronic device including a memory and one or more programs, wherein one or more programs are stored in the memory and configured to be executed by one or more processors. The one or more programs include instructions for implementing the gyroscope alignment method as described in any of the above embodiments. For example, the method may include: acquiring multiple image frames collected during a mobile terminal's reciprocating movement along a single axis, and multiple gyroscope parameters detected during the process; the multiple image frames are acquired by the camera module of the mobile terminal; determining the delay of the gyroscope relative to the camera module based on the multiple image frames and the multiple gyroscope parameters, so as to perform time alignment of the gyroscope and the camera module based on the delay.

[0117] Figure 6 This is a block diagram illustrating an apparatus 600 for implementing a gyroscope alignment method according to an exemplary embodiment. For example, apparatus 600 may be a mobile phone, computer, digital broadcasting terminal, messaging device, game console, tablet device, medical device, fitness equipment, personal digital assistant, etc.

[0118] Reference Figure 6The device 600 may include one or more of the following components: a processing component 602, a memory 604, a power supply component 606, a multimedia component 608, an audio component 610, an input / output (I / O) interface 612, a sensor component 614, and a communication component 616.

[0119] Processing component 602 typically controls the overall operation of device 600, such as operations associated with display, telephone calls, data communication, camera operation, and recording. Processing component 602 may include one or more processors 620 to execute instructions to perform all or part of the steps of the methods described above. Furthermore, processing component 602 may include one or more modules to facilitate interaction between processing component 602 and other components. For example, processing component 602 may include a multimedia module to facilitate interaction between multimedia component 608 and processing component 602.

[0120] Memory 604 is configured to store various types of data to support the operation of device 600. Examples of such data include instructions for any application or method operating on device 600, contact data, phonebook data, messages, pictures, videos, etc. Memory 604 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0121] Power supply component 606 provides power to the various components of device 600. Power supply component 606 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to device 600.

[0122] Multimedia component 608 includes a screen that provides an output interface between the device 600 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of the touch or swipe action but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 608 includes a front-facing camera and / or a rear-facing camera. When the device 600 is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or the rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0123] Audio component 610 is configured to output and / or input audio signals. For example, audio component 610 includes a microphone (MIC) configured to receive external audio signals when device 600 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 604 or transmitted via communication component 616. In some embodiments, audio component 610 also includes a speaker for outputting audio signals.

[0124] I / O interface 612 provides an interface between processing component 602 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0125] Sensor assembly 614 includes one or more sensors for providing status assessments of various aspects of device 600. For example, sensor assembly 614 may detect the on / off state of device 600, the relative positioning of components such as the display and keypad of device 600, changes in the position of device 600 or a component of device 600, the presence or absence of user contact with device 600, the orientation or acceleration / deceleration of device 600, and temperature changes of device 600. Sensor assembly 614 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 614 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 614 may also include an accelerometer, a gyroscope, a magnetometer, a pressure sensor, or a temperature sensor.

[0126] Communication component 616 is configured to facilitate wired or wireless communication between device 600 and other devices. Device 600 can access wireless networks based on communication standards, such as WiFi, 2G or 3G, 4G LTE, 5G NR (New Radio), or combinations thereof. In one exemplary embodiment, communication component 616 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 616 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0127] In an exemplary embodiment, the apparatus 600 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the methods described above.

[0128] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 604 including instructions, which can be executed by a processor 620 of the device 600 to perform the above-described method. For example, the non-transitory computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0129] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the disclosure herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0130] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

[0131] The above description is merely a preferred embodiment of this disclosure and is not intended to limit this disclosure. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A method of aligning a gyroscope, characterized by, The method comprises: obtaining a plurality of image frames collected by a mobile terminal during a back-and-forth movement of the mobile terminal along a single axis, and a plurality of gyroscope parameters detected during the movement; the plurality of image frames are collected by a camera module of the mobile terminal; determining a time delay of the gyroscope relative to the camera module based on the plurality of image frames and the plurality of gyroscope parameters, comprising: performing feature point matching on the plurality of image frames, and determining a displacement of a same feature point between different image frames based on a matching result; determining a target gyroscope parameter matching the displacement from the plurality of gyroscope parameters, to determine the time delay of the gyroscope relative to the camera module based on a time when the target gyroscope parameter is obtained and a time when the different image frames are collected; or, obtaining feature values of each of the plurality of image frames, and performing feature value matching between different image frames to obtain a displacement of a same feature value between different image frames; obtaining exposure times of different image frames corresponding to the same feature value, and obtaining times when the plurality of gyroscope parameters are obtained, to determine the time delay of the gyroscope relative to the camera module based on the exposure times, the obtaining times, the displacement, and a displacement represented by the plurality of gyroscope parameters; time aligning the gyroscope and the camera module based on the time delay.

2. The method of claim 1, wherein: the feature point matching on the plurality of image frames, and determining a displacement of a same feature point between different image frames based on a matching result, comprises: performing feature point matching on at least one group of adjacent frames in the plurality of image frames, to determine a displacement of a same feature point between the at least one group of adjacent frames based on a matching result; the determining a target gyroscope parameter matching the displacement from the plurality of gyroscope parameters, to determine the time delay of the gyroscope relative to the camera module based on a time when the target gyroscope parameter is obtained and a time when the different image frames are collected, comprises: determining at least one group of target gyroscope parameters matching the displacement of the at least one group of adjacent frames from the plurality of gyroscope parameters, to determine the time delay of the gyroscope relative to the camera module based on a time when the at least one group of target gyroscope parameters is obtained and a time when the at least one group of adjacent frames is collected.

3. The method of claim 1, wherein, the determining a target gyroscope parameter matching the displacement from the plurality of gyroscope parameters comprises: converting the displacement into a displacement of the mobile terminal based on a focal length of a camera including an image sensor of the camera module; determining a target gyroscope parameter used to calculate the displacement from the plurality of gyroscope parameters.

4. The method of claim 1, wherein, the determining a target gyroscope parameter matching the displacement from the plurality of gyroscope parameters comprises: determining an inter-frame motion speed between the plurality of image frames based on a displacement of a same feature point between different image frames and exposure times of each of the image frames; determining a target gyroscope parameter matching the inter-frame motion speed from the plurality of gyroscope parameters.

5. The method of claim 1, wherein, determine the time delay of the gyroscope relative to the camera module based on the exposure time, the acquisition time, the displacement amount represented by the plurality of gyroscope parameters, and the offset amount. determine the exposure time of at least one set of adjacent frames in the plurality of image frames based on the same feature value, to determine a to-be-determined exposure duration of the camera module based on the exposure time; and calculate a to-be-determined focal length and a to-be-determined time delay of the camera module based on the offset amount and the plurality of gyroscope parameters; construct a loss function for representing the difference in feature values between adjacent image frames based on the to-be-determined exposure duration, the to-be-determined focal length, and the to-be-determined time delay, and iteratively optimize the loss function at different time steps until the loss function converges, to determine the time step at which the loss function converges as the time delay of the gyroscope relative to the camera module.

6. The method of claim 1, wherein, The camera module uses a rolling shutter for image exposure.

7. A gyroscope alignment device characterized by, comprise: an acquisition unit configured to acquire a plurality of image frames collected by a camera module of a mobile terminal during a process in which the mobile terminal moves back and forth along a single axis, and a plurality of gyroscope parameters detected during the process; the plurality of image frames are collected by the camera module of the mobile terminal; an alignment unit configured to determine a time delay of a gyroscope relative to the camera module based on the plurality of image frames and the plurality of gyroscope parameters; the alignment unit is further configured to perform feature point matching on the plurality of image frames, and determine an offset amount of a same feature point between different image frames based on the matching result; from the plurality of gyroscope parameters, determine a target gyroscope parameter that matches the offset amount, to determine the time delay of the gyroscope relative to the camera module based on an acquisition time of the target gyroscope parameter and an acquisition time of the different image frames; or, acquire feature values of each image frame in the plurality of image frames, and perform feature value matching between different image frames to acquire an offset amount of a same feature value between different image frames; acquire an exposure time of different image frames corresponding to the same feature value, an acquisition time of the plurality of gyroscope parameters, to determine the time delay of the gyroscope relative to the camera module based on the exposure time, the acquisition time, the displacement amount represented by the plurality of gyroscope parameters, and the offset amount; the alignment unit is further configured to perform time alignment on the gyroscope and the camera module based on the time delay.

8. An electronic device, comprising: comprise: a processor; a memory for storing processor-executable instructions; wherein the processor implements the method of any one of claims 1-6 by running the executable instructions.

9. A computer readable storage medium having stored thereon computer instructions, wherein, The instructions, when executed by the processor, implement the steps of the method of any one of claims 1-6.

Images