Visual inertial calibration method and device

Abstract

The present application discloses a visual inertial calibration method and device thereof, belonging to the field of visual technology. The method comprises: in the process of driving an electronic device to rotate according to target motion parameter data through a displacement stage, obtaining image data collected by an image sensor of the electronic device and posture data collected by an inertial measurement unit of the electronic device; based on set constraints reflecting the relationship between the image data, the posture data and the target motion parameter, determining the parameter value of the calibration parameter according to the image data, the posture data and the target motion parameter; wherein the calibration parameter is a parameter for anti-shake compensation of the image data collected by the image sensor through the posture data collected by the inertial measurement unit; and saving the determined parameter value in the electronic device for the anti-shake compensation.

Description

Technical Field

[0001] The present application belongs to the field of visual technology, and specifically relates to a visual inertial calibration method and a device thereof. Background Art

[0002] With the widespread use of mobile Internet and smart devices, shooting videos has become a new way for people to record and share their lives.

[0003] In the process of shooting videos, mobile devices will inevitably shake, which affects the shooting effect. In this regard, an inertial measurement unit (IMU) is usually used for motion estimation, and anti-shake processing is performed based on the estimation results to solve the anti-shake problem of mobile devices.

[0004] In order to obtain a better anti-shake effect, the mobile device needs to be calibrated for visual inertia before it leaves the factory. In the related art, the visual inertia calibration of the mobile device is generally achieved by manually holding the device to create shaking, and matching the feature points of the previous and next frames obtained by the image sensor with the motion trajectory of the inertial measurement unit. However, this method has poor calibration accuracy and affects the anti-shake effect. Summary of the invention

[0005] The purpose of the embodiments of the present application is to provide a visual-inertial calibration method and device thereof, which can solve the problem of poor calibration accuracy existing in the visual-inertial calibration method of the prior art.

[0006] In a first aspect, an embodiment of the present application provides a visual inertial calibration method, the method comprising:

[0007] In the process of driving the electronic device to rotate according to the target motion parameter by the translation stage, acquiring image data collected by the image sensor of the electronic device and posture data collected by the inertial measurement unit of the electronic device;

[0008] Based on the set constraint conditions reflecting the relationship between the image data, the posture data and the target motion parameters, the parameter value of the calibration parameter is determined according to the image data, the posture data and the target motion parameters; wherein the calibration parameter is a parameter for performing anti-shake compensation on the image data collected by the image sensor through the posture data collected by the inertial measurement unit;

[0009] The determined parameter value is stored in the electronic device for performing the anti-shake compensation.

[0010] In a second aspect, an embodiment of the present application provides a visual inertial calibration device, the device comprising:

[0011] An acquisition module, used to acquire image data collected by an image sensor of the electronic device and posture data collected by an inertial measurement unit of the electronic device in the process of driving the electronic device to rotate according to the target motion parameter through the translation stage;

[0012] A determination module, configured to determine a parameter value of a calibration parameter according to the image data, the posture data and the target motion parameter based on a set constraint condition reflecting the relationship between the image data, the posture data and the target motion parameter; wherein the calibration parameter is a parameter for performing anti-shake compensation on the image data collected by the image sensor through the posture data collected by the inertial measurement unit;

[0013] The storage module is used to store the determined parameter value in the electronic device for performing the anti-shake compensation.

[0014] In a third aspect, an embodiment of the present application provides an electronic device, which includes a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the program or instructions are executed by the processor, the steps of the method described in the first aspect are implemented.

[0015] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented.

[0016] In a fifth aspect, an embodiment of the present application provides a chip, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the method described in the first aspect.

[0017] In a sixth aspect, an embodiment of the present application provides a computer program product, which is stored in a storage medium and is executed by at least one processor to implement the method described in the first aspect.

[0018] In an embodiment of the present application, an electronic device is controlled to rotate according to target motion parameters through a translation stage. Based on set constraints reflecting the relationship between image data, posture data and target motion parameters, parameter values ​​of calibration parameters are determined according to image data collected by an image sensor of the electronic device, posture data collected by an inertial measurement unit of the electronic device and target motion parameters of the translation stage. In this way, the posture data of the electronic device is matched with the known target motion parameters of the translation stage, and the motion trajectory of feature points in the image data is matched with the known target motion parameters of the translation stage. This can achieve visual inertial calibration of the electronic device and obtain high-precision calibration results. Furthermore, anti-shake compensation is performed on the image data collected by the image sensor according to the obtained calibration parameters, which can improve the anti-shake effect of the electronic device. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 It is a flowchart of a visual inertial calibration method provided in an embodiment of the present application;

[0020] Figure 2 is a schematic diagram of the relative relationship between the image sensor coordinate system, the inertial measurement unit coordinate system and the translation stage coordinate system provided in an embodiment of the present application;

[0021] Figure 3 is a structural schematic diagram of a visual inertial calibration device provided in an embodiment of the present application;

[0022] Figure 4 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application;

[0023] Figure 5 It is a schematic diagram of the structure of another electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0024] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0025] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0026] In the process of shooting videos, electronic devices will inevitably shake, which will affect the shooting effect. In this regard, an inertial measurement unit (IMU) is usually used for motion estimation, and anti-shake processing is performed based on the estimation results to solve the anti-shake problem of mobile devices.

[0027] However, there is a certain angle between the coordinate system of the inertial measurement unit and the coordinate system of the image sensor, and there is a certain time delay between the time when the inertial measurement unit collects attitude data and the time when the image sensor collects image data. Therefore, in order to improve the anti-shake effect of electronic equipment, visual inertial calibration is required before the electronic equipment leaves the factory, that is, the synchronization calibration of the coordinate system of the inertial measurement unit and the image sensor, as well as the synchronization calibration of the timestamp of the attitude data collected by the inertial measurement unit and the timestamp of the image data collected by the image sensor.

[0028] In the related art, the visual inertial calibration of electronic devices is usually carried out outdoors, where people hold the device in their hands to create jitter, and match the feature points of the previous and next frames obtained by the image sensor with the motion trajectory of the inertial measurement unit to achieve visual inertial calibration of the mobile device. However, this method has poor calibration accuracy and affects the anti-shake effect.

[0029] In order to solve the above problems, an embodiment of the present application provides a visual-inertial calibration method, which controls the electronic device to rotate according to the target motion parameters through a translation stage, and determines the parameter values ​​of the calibration parameters based on the image data, posture data and target motion parameters based on the set constraints reflecting the relationship between the image data, posture data and the target motion parameters to complete the visual-inertial calibration. A high-precision calibration result can be obtained, thereby improving the anti-shake effect of the electronic device.

[0030] The visual inertial calibration method provided in the embodiment of the present application is described in detail below through specific embodiments and their application scenarios in conjunction with the accompanying drawings.

[0031] See also Figure 1, which is a visual inertial calibration method provided in an embodiment of the present application. The method may include steps 1100 to 1300, which are described in detail below.

[0032] Step 1100 , in the process of driving the electronic device to rotate according to the target motion parameter by the translation stage, image data collected by the image sensor of the electronic device and posture data collected by the inertial measurement unit of the electronic device are obtained.

[0033] In this embodiment, the translation stage can be used to drive the electronic device to rotate according to the target motion parameter. The translation stage can be, for example, a six-axis hexapod translation stage. Exemplarily, the translation stage can drive the electronic device to perform rotational motion at different frequencies and angles. The target motion parameter can be set according to the jitter data of the electronic device. The target motion parameter of the translation stage can be, for example, the rotation angle of the translation stage.

[0034] The image data collected by the image sensor of the electronic device may be a video captured by the image sensor of the electronic device, that is, a video frame sequence. The image data may include N frames of images, where N is a positive integer greater than 1.

[0035] The posture data collected by the inertial measurement unit of the electronic device can be used to indicate the posture of the image sensor of the electronic device when collecting image data.

[0036] Step 1200, based on the set constraints reflecting the relationship between the image data, the posture data and the target motion parameters, determine the parameter value of the calibration parameter according to the image data, the posture data and the target motion parameters; wherein the calibration parameter is a parameter for anti-shake compensation of the image data collected by the image sensor through the posture data collected by the inertial measurement unit.

[0037] In this embodiment, setting the constraint condition may reflect the conversion relationship between the image data collected by the image sensor of the electronic device, the posture data collected by the inertial measurement unit of the electronic device, and the target motion parameters of the translation stage.

[0038] The calibration parameters may be parameters for anti-shake compensation of image data collected by the image sensor using the posture data collected by the inertial measurement unit. Exemplarily, the calibration parameters may include coordinate conversion parameters between the image data and the posture data, and time delay parameters between the image data and the posture data.

[0039] In this embodiment, the coordinate conversion parameters between the image data and the posture data can be determined based on the coordinate conversion parameters between the target motion parameters of the translation stage and the posture data collected by the inertial measurement unit, and the coordinate conversion parameters between the image data collected by the image sensor and the target motion parameters of the translation stage. Figure 2 , which is a schematic image of the relative relationship between the image sensor coordinate system, the inertial measurement unit coordinate system and the translation stage coordinate system provided in an embodiment of the present application.

[0040] Exemplarily, the coordinate conversion parameters between the target motion parameters of the translation stage and the posture data collected by the inertial measurement unit may be, for example, the rotation transformation matrix R DOF-IMU The coordinate transformation parameters between the image data collected by the image sensor and the target motion parameters of the translation stage can be, for example, the rotation transformation matrix R Cam-DOF ; The coordinate conversion parameter between image data and posture data is R Cam-DOF *R DOF-IMU .

[0041] In some embodiments of the present application, the set constraint condition includes a first constraint condition reflecting a first distance between the posture data and the target motion parameter; and a second constraint condition reflecting a second distance between the target motion parameter and the image data.

[0042] In this embodiment, the first constraint condition may reflect a first distance between the posture data collected by the inertial measurement unit and the target motion parameter of the translation stage.

[0043] In this embodiment, a first loss function can be constructed according to the first constraint condition. The first loss function is shown in formula (1):

[0044] S=∑||R DOF-IMU *A IMU -A DOF || 2 (1)

[0045] Among them, S is the first loss value of the first loss function, R DOF-IMU A is the rotation transformation matrix of the inertial measurement unit coordinate system relative to the translation stage coordinate system, IMU A is the attitude data collected by the inertial measurement unit. DOF is the target motion parameter of the translation stage.

[0046] In this embodiment, the posture data collected by the inertial measurement unit may be rotational posture information obtained by integral calculation. For example, the calculation formula (2) of the posture data is as follows:

[0047] A IMU =∑ω i *Δt1 (2)

[0048] Among them, A IMU is the attitude data collected by the inertial measurement unit, ω iis the instantaneous angular velocity of the electronic device, and Δt1 is the time interval between two adjacent acquisitions of the angular velocity of the electronic device.

[0049] In this embodiment, the rotation transformation matrix of the inertial measurement unit coordinate system relative to the translation platform coordinate system can be obtained according to the three-axis angle conversion of the inertial measurement unit coordinate system relative to the translation platform coordinate system. Among them, the three-axis angle of the inertial measurement unit coordinate system relative to the translation platform coordinate system is: θ DOF-IMU =(θ x ,θ y ,θ z ). For example, the calculation formula (3) to (6) of the rotation transformation matrix of the inertial measurement unit coordinate system relative to the translation stage coordinate system is as follows:

[0050] R DOF-IMu =R z R y R x (3)

[0051]

[0052]

[0053]

[0054] Among them, R DOF-IMU is the rotation transformation matrix of the inertial measurement unit coordinate system relative to the translation stage coordinate system, R x is the rotation transformation matrix of the inertial measurement unit coordinate system relative to the x-axis of the translation stage coordinate system, R y is the rotation transformation matrix of the inertial measurement unit coordinate system relative to the y-axis of the translation stage coordinate system, R z is the rotation transformation matrix of the inertial measurement unit coordinate system relative to the z-axis of the translation stage coordinate system, θ x is the angle between the inertial measurement unit coordinate system and the x-axis of the translation stage coordinate system, θ y is the angle between the inertial measurement unit coordinate system and the y-axis of the translation stage coordinate system, θ y is the angle between the inertial measurement unit coordinate system and the z-axis of the translation stage coordinate system.

[0055] Exemplarily, the first constraint condition may be that the first distance between the posture data collected by the inertial measurement unit and the target motion parameter of the displacement stage is less than or equal to the first threshold. Referring to formula (1), the first loss value S of the first loss function is less than or equal to the first threshold. Exemplarily, the first constraint condition may be that the first distance between the posture data collected by the inertial measurement unit and the target motion parameter of the displacement stage is a minimum value.

[0056] In this embodiment, the second constraint condition may reflect the second distance between the target motion parameter and the image data. The image data may include the current frame image and the next frame image, that is, the second constraint condition may reflect the second distance between the motion trajectory of the target feature point in the current frame image and the next frame image and the motion trajectory of the translation stage. The target feature point may be a feature point successfully matched in the current frame image and the next frame image. The current frame image and the next frame image are matched for feature points to determine the target feature point.

[0057] Exemplarily, a second loss function may be constructed according to the second constraint condition. The second loss function is shown in formula (7):

[0058] M=∑||x j-ro -x i || 2 (7)

[0059] Where M is the second loss value of the second loss function; x i is the image coordinate of the target feature point in the current frame image, that is, x i =(x i ,y i ); x j-ro is the positional relationship between the target feature points in the next frame image and the target feature points in the current frame image.

[0060] In this embodiment, the calculation formula (8) for the positional relationship between the target feature point in the next frame image and the target feature point in the current frame image is as follows:

[0061] x j-ro =K(R j R Cam-DOF )(R i R Cam-DOF ) T K -1 x i (8)

[0062] Among them, x j-ro is the positional relationship between the target feature point in the next frame image and the target feature point in the current frame image; R Cam-DOF is the rotation transformation matrix of the stage coordinate system relative to the image sensor coordinate system; R i is the rotation angle of the translation stage corresponding to the acquisition time of the current frame image acquired by the image sensor. Specifically, the time delay parameter Δt between the image data and the posture data can be used to find the rotation angle corresponding to x i The corresponding rotation angle R of the stage output i ; R jis the rotation angle of the translation stage corresponding to the acquisition time of the next frame of image collected by the image sensor. Specifically, the time delay parameter Δt between the image data and the posture data can be used to find the rotation angle corresponding to x j The corresponding rotation angle R of the stage output j ; K is the intrinsic parameter of the camera of the electronic device.

[0063] In this embodiment, the rotation transformation matrix of the translation stage coordinate system relative to the image sensor coordinate system can be obtained according to the three-axis angle conversion of the translation stage coordinate system relative to the image sensor coordinate system. Among them, the three-axis angle of the translation stage coordinate system relative to the image sensor coordinate system is: θ Cam-DOF =(θ x ′,θ y ′,θ z ′), where θ x ′ is the angle between the stage coordinate system and the x-axis of the image sensor coordinate system, θ y ′ is the angle between the stage coordinate system and the y-axis of the image sensor coordinate system, θ z ' is the angle between the translation stage coordinate system and the z-axis of the image sensor coordinate system.

[0064] Exemplarily, the calculation formula (9) of the rotation transformation matrix of the translation stage coordinate system relative to the image sensor coordinate system is as follows:

[0065] R Cam-DOF =R z ′R y ′R x ′ (9)

[0066] Among them, R Cam-DOF is the rotation transformation matrix of the stage coordinate system relative to the image sensor coordinate system, R x ′ is the rotation transformation matrix of the stage coordinate system relative to the x-axis of the image sensor coordinate system, R y ′ is the rotation transformation matrix of the stage coordinate system relative to the y-axis of the image sensor coordinate system, R z ' is the rotation transformation matrix of the stage coordinate system relative to the z-axis of the image sensor coordinate system.

[0067] Exemplarily, the second constraint condition may be that the second distance between the target motion parameter and the image data is less than or equal to the second threshold. Referring to formula (7), the second loss value M of the second loss function is less than or equal to the second threshold. Exemplarily, the second constraint condition may be that the second distance between the target motion parameter and the image data is a minimum value.

[0068] In this embodiment, by setting the first constraint condition, the posture data collected by the inertial measurement unit of the electronic device can be made as close as possible to the target motion parameters of the translation stage. By setting the second constraint condition, the motion trajectory of the feature points in the image data can be made as close as possible to the motion trajectory of the translation stage. This allows the calibration of the image sensor and the inertial measurement unit of the electronic device to be achieved with high calibration accuracy.

[0069] In some embodiments of the present application, the set constraint condition is that the weighted sum of the first constraint condition and the second constraint condition satisfies a set convergence condition.

[0070] In this embodiment, the convergence condition may be that the weighted sum of the first distance between the posture data collected by the inertial measurement unit and the target motion parameter of the translation stage and the second distance between the target motion parameter and the image data is less than or equal to the third threshold.

[0071] Exemplarily, the convergence condition can be expressed by the following formula (10):

[0072] w1S+w2M (10)

[0073] Among them, S is a first loss function reflecting the first distance between the posture data collected by the inertial measurement unit and the target motion parameter of the displacement stage, M is a second loss function reflecting the second distance between the target motion parameter and the image data, and w1 and w2 are weight coefficients. For example, the weight coefficient w1 of the first loss function is 1, and the weight coefficient w2 of the first loss function is 2. It should be noted that the weight coefficient w1 of the first loss function and the weight coefficient w2 of the first loss function can be set according to actual needs, and this embodiment does not specifically limit this.

[0074] In some embodiments of the present application, determining the parameter value of the calibration parameter according to the image data, the posture data and the target motion parameters based on the set constraints reflecting the relationship between the image data, the posture data and the target motion parameters may further include: determining the parameter value of a first parameter corresponding to the first distance and the parameter value of a second parameter corresponding to the second distance according to the image data, the posture data and the target motion parameters; obtaining the parameter value of the calibration parameter according to the parameter value of the first parameter and the parameter value of the second parameter.

[0075] In this embodiment, the first parameter corresponding to the first distance may be the three-axis angle between the inertial measurement unit coordinate system and the translation stage coordinate system, and the second parameter corresponding to the second distance may be the three-axis angle between the translation stage coordinate system and the image sensor coordinate system.

[0076] The calibration parameters include coordinate conversion parameters between the image data and the posture data, and time delay parameters between the image data and the posture data. The parameter value of the coordinate conversion parameter between the image data and the posture data can be the product of the rotation transformation matrix of the inertial measurement unit coordinate system relative to the translation stage coordinate system and the rotation transformation matrix of the translation stage coordinate system relative to the image sensor coordinate system.

[0077] In one embodiment, obtaining the parameter value of the calibration parameter according to the parameter value of the first parameter and the parameter value of the second parameter may include: determining the rotation transformation matrix of the inertial measurement unit coordinate system relative to the translation stage coordinate system according to the three-axis angle between the inertial measurement unit coordinate system and the translation stage coordinate system; determining the rotation transformation matrix of the translation stage coordinate system relative to the image sensor coordinate system according to the three-axis angle between the translation stage coordinate system and the image sensor coordinate system; determining the parameter value of the calibration parameter according to the rotation transformation matrix of the inertial measurement unit coordinate system relative to the translation stage coordinate system and the rotation transformation matrix of the translation stage coordinate system relative to the image sensor coordinate system.

[0078] Taking formula (10) as an example, the first loss function and the second loss function are optimized so that the weighted sum of the first loss value and the second loss value is the minimum value, and the three-axis angle of the inertial measurement unit coordinate system relative to the displacement stage coordinate system and the three-axis angle of the displacement stage coordinate system relative to the image sensor coordinate system corresponding to the minimum value are obtained; according to the three-axis angle of the inertial measurement unit coordinate system relative to the displacement stage coordinate system, the rotation transformation matrix of the inertial measurement unit coordinate system relative to the displacement stage coordinate system is determined; according to the three-axis angle of the displacement stage coordinate system relative to the image sensor coordinate system, the rotation transformation matrix of the displacement stage coordinate system relative to the image sensor coordinate system is determined; the product of the rotation transformation matrix of the inertial measurement unit coordinate system relative to the displacement stage coordinate system and the rotation transformation matrix of the displacement stage coordinate system relative to the image sensor coordinate system is taken as the parameter value of the calibration parameter.

[0079] After step 1200, step 1300 is executed to store the determined parameter value in the electronic device for performing the anti-shake compensation.

[0080] In an embodiment of the present application, the electronic device is controlled to rotate according to preset target motion parameters through a translation stage. Based on set constraints reflecting the relationship between image data, posture data and target motion parameters, parameter values ​​of calibration parameters are determined according to image data collected by an image sensor of the electronic device, posture data collected by an inertial measurement unit of the electronic device and target motion parameters of the translation stage. In this way, the posture data of the electronic device is matched with the known target motion parameters of the translation stage, and the motion trajectory of feature points in the image data is matched with the known target motion parameters of the translation stage. This can achieve visual inertial calibration of the electronic device and obtain high-precision calibration results. Furthermore, anti-shake compensation is performed on the image data collected by the image sensor according to the obtained calibration parameters, which can improve the anti-shake effect of the electronic device.

[0081] The visual inertial calibration method provided in the embodiment of the present application can be executed by a visual inertial calibration device. In the embodiment of the present application, the visual inertial calibration method performed by the visual inertial calibration device is taken as an example to illustrate the visual inertial calibration device provided in the embodiment of the present application.

[0082] See also Figure 3 The embodiment of the present application further provides a visual inertial calibration device 300 , which includes an acquisition module 301 , a determination module 302 and a storage module 303 .

[0083] The acquisition module 301 is used to acquire image data collected by an image sensor of the electronic device and posture data collected by an inertial measurement unit of the electronic device during the process of driving the electronic device to rotate according to the target motion parameter by the translation stage;

[0084] The determination module 302 is used to determine the parameter value of the calibration parameter according to the image data, the posture data and the target motion parameter based on the set constraint condition reflecting the relationship between the image data, the posture data and the target motion parameter; wherein the calibration parameter is a parameter for performing anti-shake compensation on the image data collected by the image sensor through the posture data collected by the inertial measurement unit;

[0085] The storage module 303 is used to store the determined parameter value in the electronic device for performing the anti-shake compensation.

[0086] Optionally, the calibration parameters include coordinate conversion parameters between the image data and the posture data, and time delay parameters between the image data and the posture data.

[0087] Optionally, the set constraint condition includes a first constraint condition reflecting a first distance between the posture data and the target motion parameter; and a second constraint condition reflecting a second distance between the target motion parameter and the image data.

[0088] Optionally, the set constraint condition is that the weighted sum of the first constraint condition and the second constraint condition satisfies a set convergence condition.

[0089] Optionally, the determination module includes: a first determination unit, used to determine the parameter value of a first parameter corresponding to the first distance and the parameter value of a second parameter corresponding to the second distance based on the image data, the posture data and the target motion parameters; a parameter acquisition unit, used to obtain the parameter value of the calibration parameter based on the parameter value of the first parameter and the parameter value of the second parameter.

[0090] In an embodiment of the present application, an electronic device is controlled to rotate according to target motion parameters through a translation stage. Based on set constraints reflecting the relationship between image data, posture data and target motion parameters, parameter values ​​of calibration parameters are determined according to image data collected by an image sensor of the electronic device, posture data collected by an inertial measurement unit of the electronic device and target motion parameters of the translation stage. In this way, the posture data of the electronic device is matched with the known target motion parameters of the translation stage, and the motion trajectory of feature points in the image data is matched with the known target motion parameters of the translation stage. This can achieve visual inertial calibration of the electronic device and obtain high-precision calibration results. Furthermore, anti-shake compensation is performed on the image data collected by the image sensor according to the obtained calibration parameters, which can improve the anti-shake effect of the electronic device.

[0091] The visual inertial calibration device in the embodiment of the present application can be an electronic device or a component in an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal or other devices other than a terminal. Exemplarily, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, a vehicle-mounted electronic device, a mobile Internet device (Mobile Internet Device, MID), an augmented reality (augmented reality, AR) / virtual reality (virtual reality, VR) device, a robot, a wearable device, an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), a netbook or a personal digital assistant (personal digital assistant, PDA), etc. It can also be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (personal computer, PC), a television (television, TV), a teller machine or a self-service machine, etc., which is not specifically limited in the embodiment of the present application.

[0092] The visual inertial calibration device in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0093] The visual inertial calibration device provided in the embodiment of the present application can achieve Figure 1 To avoid repetition, the various processes implemented by the method embodiment are not described here.

[0094] Alternatively, if Figure 4 As shown, an embodiment of the present application also provides an electronic device 400, including a processor 401 and a memory 402, wherein the memory 402 stores programs or instructions that can be executed on the processor 401, and when the program or instructions are executed by the processor 401, the various steps of the above-mentioned visual inertial calibration method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, they are not repeated here.

[0095] Figure 5 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of the present application.

[0096] The electronic device 500 includes but is not limited to: a radio frequency unit 501, a network module 502, an audio output unit 503, an input unit 504, a sensor 505, a display unit 506, a user input unit 507, an interface unit 508, a memory 509, and a processor 510.

[0097] Those skilled in the art will appreciate that the electronic device 500 may also include a power source (such as a battery) for supplying power to various components, and the power source may be logically connected to the processor 510 through a power management system, thereby implementing functions such as managing charging, discharging, and power consumption management through the power management system. Figure 5 The electronic device structure shown in the figure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently, which will not be described in detail here.

[0098] Among them, the processor 510 is used to obtain image data collected by the image sensor of the electronic device and posture data collected by the inertial measurement unit of the electronic device in the process of driving the electronic device to rotate according to the target motion parameters through the displacement stage; based on the set constraints reflecting the relationship between the image data, the posture data and the target motion parameters, the parameter value of the calibration parameter is determined according to the image data, the posture data and the target motion parameters; wherein the calibration parameter is a parameter for anti-shake compensation of the image data collected by the image sensor through the posture data collected by the inertial measurement unit; the memory 509 is used to save the determined parameter value in the electronic device for the anti-shake compensation.

[0099] Optionally, the calibration parameters include coordinate conversion parameters between the image data and the posture data, and time delay parameters between the image data and the posture data.

[0100] Optionally, the set constraint condition includes a first constraint condition reflecting a first distance between the posture data and the target motion parameter; and a second constraint condition reflecting a second distance between the target motion parameter and the image data.

[0101] Optionally, the set constraint condition is that the weighted sum of the first constraint condition and the second constraint condition satisfies a set convergence condition.

[0102] Optionally, when determining the parameter value of the calibration parameter based on the image data, the posture data and the target motion parameters, the processor 510 is used to: determine the parameter value of a first parameter corresponding to the first distance and the parameter value of a second parameter corresponding to the second distance based on the image data, the posture data and the target motion parameters; and obtain the parameter value of the calibration parameter based on the parameter value of the first parameter and the parameter value of the second parameter.

[0103] In an embodiment of the present application, an electronic device is controlled to rotate according to target motion parameters through a translation stage. Based on set constraints reflecting the relationship between image data, posture data and target motion parameters, parameter values ​​of calibration parameters are determined according to image data collected by an image sensor of the electronic device, posture data collected by an inertial measurement unit of the electronic device and target motion parameters of the translation stage. In this way, the posture data of the electronic device is matched with the known target motion parameters of the translation stage, and the motion trajectory of feature points in the image data is matched with the known target motion parameters of the translation stage. This can achieve visual inertial calibration of the electronic device and obtain high-precision calibration results. Furthermore, anti-shake compensation is performed on the image data collected by the image sensor according to the obtained calibration parameters, which can improve the anti-shake effect of the electronic device.

[0104] It should be understood that in the embodiment of the present application, the input unit 504 may include a graphics processor (GPU) 5041 and a microphone 5042, and the graphics processor 5041 processes the image data of a static picture or video obtained by an image capture device (such as a camera) in a video capture mode or an image capture mode. The display unit 506 may include a display panel 5061, and the display panel 5061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 507 includes a touch panel 5071 and at least one of other input devices 5072. The touch panel 5071 is also called a touch screen. The touch panel 5071 may include two parts: a touch detection device and a touch controller. Other input devices 5072 may include, but are not limited to, a physical keyboard, function keys (such as a volume control button, a switch button, etc.), a trackball, a mouse, and a joystick, which will not be repeated here.

[0105] The memory 509 can be used to store software programs and various data. The memory 509 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, an application program or instructions required for at least one function (such as a sound playback function, an image playback function, etc.), etc. In addition, the memory 509 may include a volatile memory or a non-volatile memory, or the memory 509 may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDRSDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM) and a direct memory bus random access memory (DRRAM). The memory 509 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.

[0106] The processor 510 may include one or more processing units; optionally, the processor 510 integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to an operating system, a user interface, and application programs, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It is understandable that the modem processor may not be integrated into the processor 510.

[0107] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the various processes of the above-mentioned visual inertial calibration method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0108] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0109] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned visual inertial calibration method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0110] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0111] An embodiment of the present application provides a computer program product, which is stored in a storage medium. The program product is executed by at least one processor to implement the various processes of the above-mentioned visual inertial calibration method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0112] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0113] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0114] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. A visual inertial calibration method, characterized in that: The method comprises: In the process of driving the electronic device to rotate according to the target motion parameter by the translation stage, acquiring image data collected by the image sensor of the electronic device and posture data collected by the inertial measurement unit of the electronic device; Based on the set constraint conditions reflecting the relationship between the image data, the posture data and the target motion parameters, the parameter value of the calibration parameter is determined according to the image data, the posture data and the target motion parameters; wherein the calibration parameter is a parameter for anti-shake compensation of the image data collected by the image sensor through the posture data collected by the inertial measurement unit, and the set constraint conditions include a first constraint condition reflecting a first distance between the posture data and the target motion parameters, and a second constraint condition reflecting a second distance between the motion trajectory of the target feature point in the current frame image and the next frame image and the motion trajectory of the displacement stage; Saving the determined parameter value in the electronic device for performing the anti-shake compensation; Determining the parameter value of the calibration parameter according to the image data, the posture data and the target motion parameter comprises: Determine, according to the image data, the posture data and the target motion parameter, a parameter value of a first parameter corresponding to the first distance and a parameter value of a second parameter corresponding to the second distance; wherein the first parameter of the first distance is a three-axis angle between an inertial measurement unit coordinate system and a translation stage coordinate system, the second parameter of the second distance is a three-axis angle between the translation stage coordinate system and an image sensor coordinate system, and the first parameter of the first distance and the second parameter of the second distance are determined according to the set constraint condition; Determine a rotation transformation matrix of the inertial measurement unit coordinate system relative to the translation stage coordinate system according to the three-axis angle of the inertial measurement unit coordinate system relative to the translation stage coordinate system; Determine a rotation transformation matrix of the translation stage coordinate system relative to the image sensor coordinate system according to the three-axis angle of the translation stage coordinate system relative to the image sensor coordinate system; The parameter values ​​of the calibration parameters are determined according to the rotation transformation matrix of the inertial measurement unit coordinate system relative to the translation stage coordinate system and the rotation transformation matrix of the translation stage coordinate system relative to the image sensor coordinate system.

2. The method according to claim 1, characterized in that The calibration parameters include coordinate conversion parameters between the image data and the posture data, and time delay parameters between the image data and the posture data.

3. The method according to claim 1, characterized in that The set constraint condition is that the weighted sum of the first constraint condition and the second constraint condition satisfies a set convergence condition.

4. A visual inertial calibration device, characterized in that: The device comprises: An acquisition module, used to acquire image data collected by an image sensor of the electronic device and posture data collected by an inertial measurement unit of the electronic device in the process of driving the electronic device to rotate according to the target motion parameter through the translation stage; A determination module, configured to determine a parameter value of a calibration parameter according to the image data, the posture data and the target motion parameter based on a set constraint condition reflecting the relationship between the image data, the posture data and the target motion parameter; wherein the calibration parameter is a parameter for performing anti-shake compensation on the image data collected by the image sensor through the posture data collected by the inertial measurement unit, and the set constraint condition includes a first constraint condition reflecting a first distance between the posture data and the target motion parameter, and a second constraint condition reflecting a second distance between a motion trajectory of a target feature point in a current frame image and a next frame image and a motion trajectory of the displacement stage; A storage module, used for storing the determined parameter value in the electronic device for performing the anti-shake compensation; The determining module comprises: a first determination unit, configured to determine a parameter value of a first parameter corresponding to the first distance and a parameter value of a second parameter corresponding to the second distance according to the image data, the posture data and the target motion parameter; wherein the first parameter of the first distance is a three-axis angle between an inertial measurement unit coordinate system and a translation stage coordinate system, the second parameter of the second distance is a three-axis angle between the translation stage coordinate system and an image sensor coordinate system, and the first parameter of the first distance and the second parameter of the second distance are determined according to the set constraint condition; A parameter acquisition unit is used to determine a rotation transformation matrix of the inertial measurement unit coordinate system relative to the translation stage coordinate system according to a three-axis angle between the inertial measurement unit coordinate system and the translation stage coordinate system; determine a rotation transformation matrix of the translation stage coordinate system relative to the image sensor coordinate system according to a three-axis angle between the translation stage coordinate system and the image sensor coordinate system; and determine parameter values ​​of calibration parameters according to the rotation transformation matrix of the inertial measurement unit coordinate system relative to the translation stage coordinate system and the rotation transformation matrix of the translation stage coordinate system relative to the image sensor coordinate system.

5. The device according to claim 4, characterized in that The calibration parameters include coordinate conversion parameters between the image data and the posture data, and time delay parameters between the image data and the posture data.

6. The device according to claim 4, characterized in that The set constraint condition is that the weighted sum of the first constraint condition and the second constraint condition satisfies a set convergence condition.

Images