Gyroscope alignment method and device, electronic equipment and storage medium

Abstract

The invention relates to a gyroscope alignment method and device, electronic equipment and a storage medium. The method comprises the following steps: acquiring a plurality of image frames acquired in the process that the mobile terminal moves back and forth along a single axial direction, and a plurality of gyroscope parameters detected in the process; the plurality of image frames are acquired by a camera module of the mobile terminal; and determining the time delay of the gyroscope relative to the camera module based on the plurality of image frames and the plurality of gyroscope parameters, so as to perform time alignment on the gyroscope and the camera module based on the time delay.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing, and in particular to a gyroscope alignment method and device, electronic equipment, and a storage medium. Background Art

[0002] In the field of image processing, anti-shake is required based on gyroscope parameters to avoid problems such as ghosting in the captured images. Therefore, before image capture, it is necessary to prioritize the time alignment of the gyroscope and the camera module of the device to ensure that the exposure operation of the camera module at the corresponding moment is anti-shake based on the gyroscope parameters at the same moment.

[0003] However, in the related art, when performing time alignment on a gyroscope and an image sensor using a rolling shutter, a large number of complex operations need to be performed, which not only results in poor alignment effect, but also has low alignment efficiency and consumes a lot of processing resources. Summary of the invention

[0004] The present disclosure provides a gyroscope alignment method and device, electronic equipment, and storage medium, which can realize rapid alignment of a gyroscope and a camera module.

[0005] According to a first aspect of the present disclosure, a gyroscope alignment method is provided, comprising:

[0006] Acquire a plurality of image frames collected during the reciprocating movement of the mobile terminal along a single axis, and a plurality of gyroscope parameters detected during the process; the plurality of image frames are collected by a camera module of the mobile terminal;

[0007] A delay of the gyroscope compared to the camera module is determined based on the multiple image frames and the multiple gyroscope parameters to time-align the gyroscope and the camera module based on the delay.

[0008] According to a second aspect of the present disclosure, a gyroscope alignment device is provided, comprising:

[0009] An acquisition unit, which acquires a plurality of image frames collected during a reciprocating movement of a mobile terminal along a single axis, and a plurality of gyroscope parameters detected during the process; the plurality of image frames are collected by a camera module of the mobile terminal;

[0010] An alignment unit determines a delay of the gyroscope compared to the camera module based on the multiple image frames and the multiple gyroscope parameters, so as to perform time alignment on the gyroscope and the camera module based on the delay.

[0011] According to a third aspect of the present disclosure, there is provided an electronic device, including:

[0012] processor;

[0013] a memory for storing processor-executable instructions;

[0014] The processor implements the method as described in the first aspect by running the executable instructions.

[0015] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided, on which computer instructions are stored, and when the instructions are executed by a processor, the steps of the method described in the first aspect are implemented.

[0016] In the technical solution disclosed in the present invention, the mobile terminal can be controlled to move back and forth in a single axis. In this process, the mobile terminal can collect multiple image frames through the camera module on the one hand, and obtain multiple gyroscope parameters from the gyroscope on the other hand. On this basis, the delay of the gyroscope compared with the camera module can be determined according to the collected multiple image frames and the obtained multiple gyroscope parameters, and then the gyroscope and the camera module are time-correlated according to the delay.

[0017] It should be understood that since the present disclosure enables the mobile terminal to move back and forth on a single axis, it can greatly simplify the calculation logic of subsequent time alignment, avoiding the problem of simulating actual air jitter for time alignment in related technologies, which leads to low alignment efficiency and occupies a large amount of processing resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0019] Figure 1 is a flow chart of a gyroscope alignment method shown in an exemplary embodiment of the present disclosure;

[0020] Figure 2 is a flow chart of another gyroscope alignment method shown in an exemplary embodiment of the present disclosure;

[0021] Figure 3 is a schematic diagram showing a smartphone moving back and forth along a single axis according to an exemplary embodiment of the present disclosure;

[0022] Figure 4 is a schematic diagram of an exposure process of a rolling shutter shown in an exemplary embodiment of the present disclosure;

[0023] Figure 5 is a block diagram of a gyroscope alignment device shown in an exemplary embodiment of the present disclosure;

[0024] Figure 6 It is a schematic structural diagram of an electronic device in an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0025] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0026] The terms used in this disclosure are for the purpose of describing specific embodiments only and are not intended to limit the disclosure. The singular forms of "a", "said" and "the" used in this disclosure and the appended claims are also intended to include plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0027] It should be understood that although the terms first, second, third, etc. may be used in the present disclosure to describe various information, such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present disclosure, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0028] In the field of image processing, in order to avoid problems such as afterimages in the captured images, anti-shake is performed during the image acquisition process. This anti-shake operation usually relies on the gyroscope installed in the mobile terminal. Therefore, before the mobile terminal leaves the factory, it is usually necessary to time-align the gyroscope and the camera module of the mobile terminal to ensure that during the image capture process, the image captured at the corresponding moment is anti-shake based on the gyroscope parameters at the same moment, thereby avoiding the problem of poor anti-shake effect.

[0029] In the related art, when time alignment is performed on a gyroscope and a camera module, a process of capturing an image by a mobile terminal is usually simulated, and time alignment is performed based on the image and gyroscope parameters collected in the process.

[0030] For example, when the mobile terminal is a smart phone, you can hold the smart phone to take pictures and deliberately shake it to a certain extent to obtain multiple image frames and gyroscope parameters collected during the shaking process. On this basis, the delay of the gyroscope compared to the camera module can be calculated based on the image parameters of the multiple image frames and the collected gyroscope parameters to perform time alignment.

[0031] When using the relevant technology for time alignment, since the shaking of the mobile terminal is completely irregular, the acquired image parameters such as the offset between image frames and the gyroscope parameters such as angular velocity are relatively complex, so the algorithm used for time alignment also needs to add complex logic such as direction judgment. It can be seen that the gyroscope alignment method in the relevant technology has problems such as low alignment efficiency and occupying a large amount of processing resources.

[0032] To this end, the present disclosure proposes a gyroscope alignment method. When the method is used for time alignment, the mobile terminal is controlled to move back and forth along a single axis. During the process, on the one hand, multiple image frames can be collected by the camera module of the mobile terminal, and on the other hand, multiple gyroscope parameters can be obtained from the gyroscope. On this basis, the delay of the gyroscope compared to the camera module can be determined based on the collected multiple image frames and multiple gyroscope parameters, so as to perform time alignment on the gyroscope and the camera module based on the delay.

[0033] It should be understood that when the gyroscope and the camera module are time-aligned, the timestamps of the camera module and the gyroscope are actually aligned based on the offset of the captured image and the displacement of the gyroscope, which is essentially the alignment of the parameters of the time dimension based on the parameters of the spatial dimension. Based on this, it is not difficult to understand that, whether it is an image or a device, the impact of movement in each axis of the spatial dimension on the time dimension is consistent. Therefore, even if the displacement is limited to a single axis, it will not affect the alignment operation of the time dimension (in other words, only time alignment needs to be completed on a single axis, which is equivalent to completing time alignment for all axes). However, after the axis is limited, the subsequent alignment calculation operation can greatly simplify the calculation logic because there is no need to consider multi-directional issues, which not only improves the alignment efficiency, but also reduces the occupation of processing resources by the alignment algorithm.

[0034] It can be seen that the gyroscope alignment method disclosed in the present invention can avoid the problem of low alignment efficiency and the problem of occupying more processing resources caused by the need to use complex alignment algorithms to perform time alignment on the gyroscope and the camera module in the related art.

[0035] The following is an introduction to the various steps included in the technical solution of the present disclosure.

[0036] Figure 1 A gyroscope alignment method is shown as an exemplary embodiment of the present disclosure. Figure 1 As shown, the method may include the following steps:

[0037] Step 102, acquiring a plurality of image frames collected during the reciprocating movement of the mobile terminal along a single axis, and a plurality of gyroscope parameters detected during the process; the plurality of image frames are collected by a camera module of the mobile terminal.

[0038] In the present disclosure, the mobile terminal can be controlled to move back and forth in any axis. For example, the mobile terminal can be controlled to move back and forth in the horizontal axis to obtain multiple image frames and multiple gyroscope parameters collected when the mobile terminal moves back and forth in the horizontal axis. For another example, the mobile terminal can be controlled to move back and forth in the axis where the gravity direction is located to obtain multiple image frames and multiple gyroscope parameters collected when the mobile terminal moves back and forth in the axis.

[0039] In the present disclosure, the movement of the mobile terminal can be kept in a single axis in a variety of ways. For example, one side of the mobile terminal can be moved back and forth close to a plane so that it cannot move in an axis perpendicular to the plane; for another example, a special device containing only a single axis groove can be designed, so that the mobile terminal can be ensured to move along a single axis by simply inserting the mobile terminal into the groove.

[0040] Of course, the above examples are only illustrative. The specific axis along which the mobile terminal moves back and forth and how to ensure that the mobile terminal moves along a single axis can be determined by those skilled in the art according to actual conditions, and the present disclosure does not limit this.

[0041] Step 104: Determine a delay of the gyroscope compared to the camera module based on the multiple image frames and the multiple gyroscope parameters, so as to time-align the gyroscope and the camera module based on the delay.

[0042] In the present disclosure, after a plurality of image frames are captured by a camera module and a plurality of gyroscope parameters are obtained, the delay of the gyroscope compared to the camera module can be determined.

[0043] In one embodiment, feature point matching can be preferentially performed on multiple image frames to determine the offset of the same feature point between different image frames based on the matching results. On this basis, the target gyroscope parameters that match the offset can be determined from multiple gyroscope parameters, and then the delay of the gyroscope compared to the camera module can be determined based on the acquisition time of the target gyroscope parameters and the acquisition time of the above-mentioned different image frames.

[0044] It should be understood that since both the image frame and the gyroscope parameters are obtained during the round-trip movement of the mobile terminal, if the acquisition time of the image frame and the acquisition time of the gyroscope parameters are consistent on the objective time axis, the offset of the same feature point in different image frames must match the displacement calculated by the gyroscope parameters acquired at the acquisition time of the different image frames (on the objective time axis). Therefore, it can be known from the reverse deduction method in this embodiment that the target gyroscope parameters matched with the offset determined from multiple gyroscope parameters are usually characterized by the gyroscope parameters acquired at the acquisition time of the image frame (on the objective time axis) corresponding to the offset. On this basis, based on the acquisition time of the determined target gyroscope parameters and the acquisition time of the image frame, it is obvious that the delay of the gyroscope compared to the camera module can be determined. For example, the timestamps of the camera module acquiring the above-mentioned different image frames and the timestamps of the gyroscope acquiring the determined target gyroscope parameters can be obtained, and the difference between the two can be calculated as the delay of the gyroscope compared to the camera module.

[0045] In this embodiment, the collected multiple image frames may include at least one group of adjacent frames, so when the feature point matching is actually performed, the feature point matching may be performed on the adjacent frames. In other words, feature point matching may be performed on at least one group of adjacent frames in the multiple image frames to determine the offset of the same feature point between the at least one group of adjacent frames based on the matching result. On this basis, at least one group of target gyroscope parameters that match the offset of the at least one group of adjacent frames may be determined from the multiple gyroscope parameters to determine the delay of the gyroscope compared to the camera module based on the acquisition time of the at least one group of target gyroscope parameters and the acquisition time of the determined at least one group of adjacent frames. It is worth noting that the "offset of the same feature point between at least one group of adjacent frames" mentioned here refers to: the offset of the same feature point between two image frames contained in a group of adjacent frames, which may determine only the offset of one group of adjacent frames, or determine the offset corresponding to each group of adjacent frames in multiple groups of adjacent frames.

[0046] Of course, determining the offset of adjacent frames here is only schematic. For example, after acquiring multiple image frames, the offset between the first frame and the last frame can be calculated to determine the delay of the gyroscope compared to the camera module. The specific determination of the offset between the multiple image frames can be determined by those skilled in the art according to actual needs, and the present disclosure does not limit this.

[0047] In this embodiment, since the displacement of the physical object and the offset of the pixel on the image sensor are related to the focal length of the camera to which the image sensor belongs, when determining the target gyroscope parameters that match the offset from multiple gyroscope parameters, the offset can be converted into the displacement of the mobile terminal based on the focal length of the camera to which the image sensor included in the camera module belongs. On this basis, the target gyroscope parameters used to calculate the displacement can be determined from the multiple gyroscope parameters.

[0048] In this embodiment, when determining the target gyroscope parameter that matches the offset from multiple gyroscope parameters, the determination can also be made based on the inter-frame motion speed. For example, after determining the displacement of the same feature point between different image frames, the inter-frame motion speed between multiple image frames can be determined based on the displacement and the exposure time of each image frame. On this basis, the target gyroscope parameter that matches the inter-frame motion speed can be determined from multiple gyroscope parameters. For example, since the gyroscope parameters acquired by the gyroscope include angular velocity, the speed at each acquisition time can be acquired by integration, and the acquired speed can be compared with the inter-frame motion speed, and the gyroscope parameter corresponding to the inter-frame motion speed can be used as the determined target gyroscope parameter.

[0049] In another embodiment, after acquiring multiple image frames, the characteristic values ​​of each image frame can be determined first, and characteristic value matching can be performed between different image frames to obtain the offset of the same characteristic value between different image frames. On this basis, the exposure time of different image frames corresponding to the same characteristic value and the acquisition time of multiple gyroscope parameters can be obtained, so as to determine the delay of the gyroscope compared to the camera module based on the exposure time, acquisition time, offset, and displacement represented by multiple gyroscope parameters.

[0050] For example, this embodiment can determine the exposure time of at least one group of adjacent frames in multiple image frames based on the same characteristic value, so as to determine the pending exposure time of the camera module based on the exposure time; and, based on the above-mentioned offset and multiple gyroscope parameters, calculate the pending focal length and pending delay of the camera module.

[0051] For example, when calculating the exposure time, the pixel point corresponding to the same characteristic value in adjacent image frames can be determined, and the exposure time of the pixel point can be used as the exposure time of the entire image frame, so as to calculate the exposure time of the camera module based on the exposure time of the adjacent frames; when calculating the pending focal length, it can be assumed that the gyroscope and the camera module have completed time alignment. On this basis, the initial focal length of the camera module can be calculated based on the offset of adjacent image frames obtained at a certain timestamp and the gyroscope parameters collected at the same timestamp as the above-mentioned pending focal length; and the determination of the pending delay can be determined based on the above-mentioned method of calculating the inter-frame motion speed and obtaining the angular velocity in the gyroscope parameters.

[0052] After obtaining the pending exposure time, pending focal length and pending delay, a loss function for characterizing the difference in eigenvalues ​​in adjacent image frames can be constructed based on the pending exposure time, pending focal length and pending delay, and the loss function can be iteratively optimized through different time steps until the loss function converges, so as to determine the time step that makes the loss function converge as the delay of the gyroscope compared to the camera module.

[0053] Of course, the above-mentioned method of constructing the loss function is only illustrative. How to construct the loss function and how to iteratively optimize and determine the delay can be determined by those skilled in the art according to actual needs, and the present disclosure does not limit this.

[0054] In the present disclosure, the camera module can use different exposure modes to capture images. For example, a global shutter can be used to capture images, or a rolling shutter can be used to capture images.

[0055] Wherein, when a rolling shutter is used for image exposure, since the rolling shutter uses a row-by-row exposure method to expose the image, when performing the above-mentioned feature point matching, feature point matching can be performed on the same row of different image frames to determine feature points corresponding to each row between the two image frames. Similarly, when performing the above-mentioned feature value matching, feature value matching can also be performed on the same row of different image frames to determine feature values ​​corresponding to each row between the two image frames.

[0056] It is worth noting that the present disclosure can select a preset range in different image frames, and only perform feature point or feature value matching within the preset range to reduce the computational complexity of feature value matching. For example, the area within the middle preset width of the image frame can be used as the above preset range to perform feature value matching.

[0057] It should be noted that the execution subject of the technical solution of the present disclosure can be any type of mobile terminal, for example, it can be a traditional mobile terminal such as a smart phone, a tablet computer, or a new mobile terminal such as a smart car, a robot, etc. It should be understood that any mobile terminal equipped with a camera module and a gyroscope can be used as the mobile terminal in the present disclosure, and which type of mobile terminal is used as the execution subject of the technical solution of the present disclosure can be determined by those skilled in the art according to actual needs, and the present disclosure does not limit this.

[0058] It can be seen from the above technical solution that when aligning the gyroscope, the present invention can limit the mobile terminal to move back and forth in a single axis, so as to time-align the gyroscope and the camera module based on multiple image frames collected by the mobile terminal during the movement and the acquired gyroscope parameters.

[0059] It should be understood that since the present disclosure limits the mobile terminal to reciprocating movement only in a single axis during gyroscope alignment, the gyroscope parameters obtained by the present disclosure are relatively simple. On this basis, the present disclosure does not need to adopt complex alignment algorithms, and can perform time alignment based on gyroscope parameters and image frames, thereby avoiding the problem of low alignment efficiency caused by the need to use complex alignment algorithms for gyroscope alignment in related technologies.

[0060] Below, taking “aligning the gyroscope of a smartphone” as an example, the technical solution of the present disclosure is introduced.

[0061] Figure 2 FIG. 1 is a flow chart of another gyroscope alignment method shown in an exemplary embodiment of the present disclosure. Figure 2 As shown, the method comprises the following steps:

[0062] Step 201, make the lower edge of the smart phone close to the desktop and move back and forth along a single axis.

[0063] In this embodiment, before the smartphone leaves the factory, the assembler can place the lower edge of the smartphone close to the desktop, so that the smartphone cannot shake in the direction perpendicular to the desktop, and control the smartphone to move back and forth along a single axis of the desktop. Figure 3 Move back and forth as shown.

[0064] Step 202, start the video recording function of the smart phone, so that the smart phone continuously captures multiple image frames.

[0065] In this embodiment, when the smartphone is in the round-trip state, the recording function of the smartphone can be activated to capture multiple image frames. For example, the assembler can activate the recording function before starting to move, so as to continuously shoot through the camera module; for another example, the camera application of the smartphone can include a timed recording function, so the assembler can activate the timed recording function before moving the smartphone, and then the smartphone automatically starts recording after starting to move the smartphone.

[0066] Step 203 : The smartphone determines the delay of the gyroscope compared to the camera module based on the multiple captured image frames and the gyroscope parameters acquired during the movement.

[0067] In this embodiment, the smartphone acquires multiple image frames while moving, and on the other hand, it continues to acquire gyroscope parameters output by the gyroscope. On this basis, the delay of the gyroscope compared to the camera module, i.e., Gyro Delay, can be determined based on a preset calibration strategy.

[0068] For example, if a smartphone uses a rolling shutter for image exposure, the correction strategy can include two stages:

[0069] The first stage: determine the three initial values ​​to be corrected, namely the gyroscope delay Gyro Delay, the rolling shutter exposure time Rolling Shutter time, and the focal length of the camera included in the camera module.

[0070] It is worth noting that the exposure time here refers specifically to the scanning time of the rolling shutter. The exposure process of the rolling shutter can be referred to Figure 4 , including the exposure time frameSExp and scanning time RollingShutter time of each row.

[0071] This phase can include three steps:

[0072] Step ①: Take the central area frame of the adjacent frame images, and obtain the average grayscale value of each row of pixels in the central area frame as the feature value Frame1D of the row. On this basis, the offset of the adjacent frames can be measured based on the feature difference of the adjacent frames. The calculation can be performed on each row of the adjacent frames to integrate the offset of the entire image.

[0073] Step ②: Use different pixel step sizes to match the motion trajectory of the same feature value between adjacent frames, and obtain the pixel difference patchMVPixel of the inter-frame motion. Combined with the timestamp frameDT of the adjacent frames when they were acquired, the linear speed of the inter-frame motion can be obtained: pixelSpeed ​​= patchMVPixel / frameDT.

[0074] Step 3: Based on the relationship between angular velocity, linear velocity and radius, the timestamp of each image frame collected by the known camera module and its corresponding gyroscope parameter FrameTimeGyro, the initial value of the focal length can be calculated: FocalLength = pixelSpeed / FrameTimeGyro.

[0075] Based on the focal length and linear velocity, the gyroscope parameters corresponding to the motion between video frames can be further predicted: VideoGyro=pixelSpeed / FocalLength.

[0076] Step ④: Since the linear velocity measured based on adjacent image frames and the displacement velocity measured based on the gyroscope parameters should match at the same time, the initial value of GyroDelay can be predicted by iterating through different time steps.

[0077] Step ⑤: The camera module can calculate the initial value of the exposure time of the single-frame image, Rolling Shutter time, by using the two exposure moments when acquiring adjacent frames. When actually acquiring the exposure moment, the timestamp when the image is exposed can be directly used as the exposure moment of the corresponding image, or the middle moment when the image is exposed can be used as the exposure moment of the image. This embodiment does not limit this.

[0078] The second stage: construct a loss function "for characterizing the difference in feature values ​​of adjacent frames" based on the three initial values ​​to be corrected. This stage may include the following steps:

[0079] Step ①: Obtain the angular velocity and timestamp of the gyroscope when acquiring a single frame image, so as to calculate the displacement gyroPos measured by the gyroscope based on the angular velocity and timestamp.

[0080] And, find the middle moment of the image frame:

[0081] frameSTimeMid=frameSTime (exposure end time)-frameSExp (exposure time of each line) / 2-Rolling Shutter / 2+Gyro Delay;

[0082] Step ②: Calculate the Rolling Shutter time rowTimesBase corresponding to each row of pixels in the central area frame. Get the real time of each row of this frame: rowTimes=rowTimesBase+frameSTimeMid[frmID];

[0083] Step 3: Use frameSTimeMid and gyroPos to construct the interpolation function gyroIntp of the middle time of each frame and the rotation angle between frames, and calculate the rotation angle camAngles=gyroIntp(rowTimes) of the image frame at the middle time of each row.

[0084] Step ④: When the rotation angle is very small, the rotation angle corresponding to each row of pixels can be approximately calculated: rowAnglesBase = arctan(pixel[i] / Focal Length).

[0085] Step 5: Use rowAnglesBase+camAngles and frame1D to construct the interpolation function frame1DIntp, and find the eigenvalues ​​corresponding to the pixel changes in each row newFrame=frame1DIntp(rowAnglesBase)

[0086] Step ⑥: Define the feature value of the previous frame as oldFrame, and the feature value of the next frame as newFrame. Then the feature value of the next frame mapped to the previous frame is:

[0087] NormFrame=(oldFrame-avg(oldFrame))*std(newFrame) / std(oldFrame)+avg(newFrame);

[0088] Step 7: Construct loss function: loss = abs(NormFrame-newFrame);

[0089] The third stage: iterate the loss function with different step sizes to obtain the final values ​​of the above-mentioned Gyro Delay, RollingShutter time, and Focal Length. For example, a global optimization algorithm (Basin Hopping) can be used for iteration.

[0090] It can be seen from the above technical solution that the present invention can enable the smartphone to move back and forth in a single axis when the gyroscope is aligned, so as to time-align the gyroscope and the camera module based on multiple image frames collected by the mobile terminal during the movement and the acquired gyroscope parameters.

[0091] Among them, the initial value of the gyroscope delay can be obtained first, and by constructing a loss function, iterative optimization is performed with different step lengths until the loss function converges. At this time, the final value of the corrected delay can be used as the delay of the gyroscope compared to the camera module. In addition, the exposure time of the rolling shutter, the focal length of the camera, and other parameters can be obtained through iterative optimization to avoid the problem of needing to measure such parameters separately.

[0092] Figure 5 is a block diagram of a gyroscope alignment device shown in an exemplary embodiment of the present disclosure. Figure 5 , the device includes an acquisition unit 501 and an alignment unit 502.

[0093] An acquisition unit, which acquires a plurality of image frames collected during a reciprocating movement of a mobile terminal along a single axis, and a plurality of gyroscope parameters detected during the process; the plurality of image frames are collected by a camera module of the mobile terminal;

[0094] An alignment unit determines a delay of the gyroscope compared to the camera module based on the multiple image frames and the multiple gyroscope parameters, so as to perform time alignment on the gyroscope and the camera module based on the delay.

[0095] Optionally, the alignment unit is further used to:

[0096] Performing feature point matching on the multiple image frames, and determining the offset of the same feature point between different image frames based on the matching result;

[0097] A target gyroscope parameter matching the offset is determined from the multiple gyroscope parameters, so as to determine the delay of the gyroscope compared with the camera module based on the acquisition time of the target gyroscope parameter and the acquisition time of the different image frames.

[0098] Optionally, the alignment unit is further used to:

[0099] Performing feature point matching on at least one group of adjacent frames among the plurality of image frames to determine an offset of the same feature point between the at least one group of adjacent frames based on the matching result;

[0100] At least one set of target gyroscope parameters that matches the offset of the at least one set of adjacent frames is determined from the multiple gyroscope parameters, so as to determine the delay of the gyroscope compared to the camera module based on the acquisition time of the at least one set of target gyroscope parameters and the acquisition time of the at least one set of adjacent frames.

[0101] Optionally, the alignment unit is further used to:

[0102] Based on the focal length of a camera to which the image sensor included in the camera module belongs, converting the offset into a displacement of the mobile terminal;

[0103] A target gyroscope parameter for calculating the displacement is determined from the plurality of gyroscope parameters.

[0104] Optionally, the alignment unit is further used to:

[0105] Determining the inter-frame motion speed between the plurality of image frames based on the displacement of the same feature point between different image frames and the exposure time of each image frame;

[0106] A target gyroscope parameter matching the inter-frame motion speed is determined from the multiple gyroscope parameters.

[0107] Optionally, the alignment unit is further used to:

[0108] Acquire a feature value of each image frame in the plurality of image frames, and perform feature value matching between different image frames to obtain an offset of the same feature value between different image frames;

[0109] The exposure time of different image frames corresponding to the same characteristic value and the acquisition time of the multiple gyroscope parameters are obtained, so as to determine the delay of the gyroscope compared with the camera module based on the exposure time, the acquisition time, the offset, and the displacement represented by the multiple gyroscope parameters.

[0110] Optionally, the alignment unit is further used to:

[0111] Determine the exposure time of at least one group of adjacent frames in the multiple image frames based on the same characteristic value, so as to determine the pending exposure time of the camera module based on the exposure time; and calculate the pending focal length and the pending delay of the camera module based on the offset and the multiple gyroscope parameters;

[0112] Based on the to-be-determined exposure time, the to-be-determined focal length and the to-be-determined delay, a loss function is constructed to characterize the difference in eigenvalues ​​in adjacent image frames, and the loss function is iteratively optimized through different time steps until the loss function converges, so as to determine the time step that makes the loss function converge as the delay of the gyroscope compared to the camera module.

[0113] Optionally, the camera module uses a rolling shutter for image exposure.

[0114] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can refer to the partial description of the method embodiments. The device embodiments described above are only schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the disclosed solution. A person of ordinary skill in the art can understand and implement it without paying any creative work.

[0115] Accordingly, the present disclosure also provides a gyroscope alignment device, comprising: a processor; a memory for storing processor executable instructions; wherein the processor is configured to implement a gyroscope alignment method as described in any of the above embodiments, for example, the method may include: obtaining a plurality of image frames collected during the reciprocating movement of a mobile terminal along a single axis, and a plurality of gyroscope parameters detected during the process; the plurality of image frames are collected by a camera module of the mobile terminal; based on the plurality of image frames and the plurality of gyroscope parameters, determining the delay of the gyroscope compared to the camera module, so as to perform time alignment on the gyroscope and the camera module based on the delay.

[0116] Accordingly, the present disclosure also provides an electronic device, which includes a memory and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by one or more processors. The one or more programs include instructions for implementing a gyroscope alignment method as described in any of the above embodiments. For example, the method may include: acquiring multiple image frames captured during the reciprocating movement of a mobile terminal along a single axis, and multiple gyroscope parameters detected during the process; the multiple image frames are captured by a camera module of the mobile terminal; based on the multiple image frames and the multiple gyroscope parameters, determining the delay of the gyroscope compared to the camera module, so as to time-align the gyroscope and the camera module based on the delay.

[0117] Figure 6 6 is a block diagram of an apparatus 600 for implementing a gyroscope alignment method according to an exemplary embodiment. For example, the apparatus 600 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0118] Reference Figure 6, the device 600 may include one or more of the following components: a processing component 602 , a memory 604 , a power component 606 , a multimedia component 608 , an audio component 610 , an input / output (I / O) interface 612 , a sensor component 614 , and a communication component 616 .

[0119] The processing component 602 generally controls the overall operation of the device 600, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 602 may include one or more processors 620 to execute instructions to complete all or part of the steps of the above-mentioned method. In addition, the processing component 602 may include one or more modules to facilitate the interaction between the processing component 602 and other components. For example, the processing component 602 may include a multimedia module to facilitate the interaction between the multimedia component 608 and the processing component 602.

[0120] The memory 604 is configured to store various types of data to support operations on the device 600. Examples of such data include instructions for any application or method operating on the device 600, contact data, phone book data, messages, pictures, videos, etc. The memory 604 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0121] The power supply component 606 provides power to the various components of the device 600. The power supply component 606 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the device 600.

[0122] The multimedia component 608 includes a screen that provides an output interface between the device 600 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 608 includes a front camera and / or a rear camera. When the device 600 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0123] The audio component 610 is configured to output and / or input audio signals. For example, the audio component 610 includes a microphone (MIC), and when the device 600 is in an operating mode, such as a call mode, a recording mode, and a speech recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 604 or sent via the communication component 616. In some embodiments, the audio component 610 also includes a speaker for outputting audio signals.

[0124] I / O interface 612 provides an interface between processing component 602 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: a home button, a volume button, a start button, and a lock button.

[0125] The sensor assembly 614 includes one or more sensors for providing various aspects of status assessment for the device 600. For example, the sensor assembly 614 can detect the open / closed state of the device 600, the relative positioning of components, such as the display and keypad of the device 600, and the sensor assembly 614 can also detect the position change of the device 600 or a component of the device 600, the presence or absence of user contact with the device 600, the orientation or acceleration / deceleration of the device 600, and the temperature change of the device 600. The sensor assembly 614 may include a proximity sensor configured to detect the presence of a nearby object without any physical contact. The sensor assembly 614 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 614 may also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0126] The communication component 616 is configured to facilitate wired or wireless communication between the device 600 and other devices. The device 600 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, 4G LTE, 5G NR (New Radio) or a combination thereof. In an exemplary embodiment, the communication component 616 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 616 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0127] In an exemplary embodiment, the apparatus 600 may be implemented by one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic components to perform the above method.

[0128] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 604 including instructions, and the instructions can be executed by a processor 620 of the device 600 to perform the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0129] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the disclosure disclosed herein. The present disclosure is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0130] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

[0131] The above description is only a preferred embodiment of the present disclosure and is not intended to limit the present disclosure. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present disclosure shall be included in the scope of protection of the present disclosure.

Claims

1. A gyroscope alignment method, It is characterized in that include: Acquire a plurality of image frames collected during the reciprocating movement of the mobile terminal along a single axis, and a plurality of gyroscope parameters detected during the process; The multiple image frames are acquired by a camera module of the mobile terminal; A delay of the gyroscope compared to the camera module is determined based on the multiple image frames and the multiple gyroscope parameters to time-align the gyroscope and the camera module based on the delay.

2. The method according to claim 1, It is characterized in that The determining the delay of the gyroscope compared to the camera module based on the multiple image frames and the multiple gyroscope parameters includes: Performing feature point matching on the multiple image frames, and determining the offset of the same feature point between different image frames based on the matching result; A target gyroscope parameter matching the offset is determined from the multiple gyroscope parameters, so as to determine the delay of the gyroscope compared with the camera module based on the acquisition time of the target gyroscope parameter and the acquisition time of the different image frames.

3. The method according to claim 2, It is characterized in that The performing feature point matching on the multiple image frames and determining the offset of the same feature point between different image frames based on the matching results comprises: performing feature point matching on at least one group of adjacent frames in the multiple image frames, and determining the offset of the same feature point between the at least one group of adjacent frames based on the matching results; The step of determining target gyroscope parameters matching the offset from the multiple gyroscope parameters to determine the delay of the gyroscope compared to the camera module based on the acquisition time of the target gyroscope parameters and the acquisition time of the different image frames comprises: determining at least one group of target gyroscope parameters matching the offset of the at least one group of adjacent frames from the multiple gyroscope parameters to determine the delay of the gyroscope compared to the camera module based on the acquisition time of the at least one group of target gyroscope parameters and the acquisition time of the at least one group of adjacent frames.

4. The method according to claim 2, It is characterized in that The step of determining a target gyroscope parameter that matches the offset from the plurality of gyroscope parameters comprises: Based on the focal length of a camera to which the image sensor included in the camera module belongs, converting the offset into a displacement of the mobile terminal; A target gyroscope parameter for calculating the displacement is determined from the plurality of gyroscope parameters.

5. The method according to claim 2, It is characterized in that The step of determining a target gyroscope parameter that matches the offset from the plurality of gyroscope parameters comprises: Determining the inter-frame motion speed between the multiple image frames based on the displacement of the same feature point between different image frames and the exposure time of each image frame; A target gyroscope parameter matching the inter-frame motion speed is determined from the multiple gyroscope parameters.

6. The method according to claim 1, It is characterized in that The determining the delay of the gyroscope compared to the camera module based on the multiple image frames and the multiple gyroscope parameters includes: Acquire a feature value of each image frame in the plurality of image frames, and perform feature value matching between different image frames to obtain an offset of the same feature value between different image frames; The exposure time of different image frames corresponding to the same characteristic value and the acquisition time of the multiple gyroscope parameters are obtained, so as to determine the delay of the gyroscope compared with the camera module based on the exposure time, the acquisition time, the offset, and the displacement represented by the multiple gyroscope parameters.

7. The method according to claim 6, It is characterized in that The determining, based on the exposure time, the acquisition time, the offset, and the displacement represented by the multiple gyroscope parameters, the delay of the gyroscope compared to the camera module includes: Determine the exposure time of at least one group of adjacent frames in the multiple image frames based on the same characteristic value, so as to determine the pending exposure time of the camera module based on the exposure time; and calculate the pending focal length and the pending delay of the camera module based on the offset and the multiple gyroscope parameters; Based on the to-be-determined exposure time, the to-be-determined focal length and the to-be-determined delay, a loss function is constructed to characterize the difference in eigenvalues ​​in adjacent image frames, and the loss function is iteratively optimized through different time steps until the loss function converges, so as to determine the time step that makes the loss function converge as the delay of the gyroscope compared to the camera module.

8. The method according to claim 1, It is characterized in that The camera module uses a rolling shutter to expose images.

9. A gyroscope alignment device, It is characterized in that include: An acquisition unit, which acquires a plurality of image frames collected during the reciprocating movement of the mobile terminal along a single axis, and a plurality of gyroscope parameters detected during the process; The multiple image frames are acquired by a camera module of the mobile terminal; An alignment unit determines a delay of the gyroscope compared to the camera module based on the multiple image frames and the multiple gyroscope parameters, so as to perform time alignment on the gyroscope and the camera module based on the delay.

10. An electronic device, It is characterized in that include: processor; a memory for storing processor-executable instructions; The processor implements the method according to any one of claims 1 to 8 by running the executable instructions.

11. A computer-readable storage medium having computer instructions stored thereon, It is characterized in that When the instruction is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images