Pose compensation method, device and equipment of intelligent glasses camera and storage medium

Abstract

The invention provides an intelligent glasses camera pose compensation method, device and equipment and a storage medium, and the method comprises the steps: calculating a first rotation matrix corresponding to the current relative pose of at least two cameras on intelligent glasses according to the current relative angle data of the two cameras, therefore, the relative pose variation between every two cameras can be conveniently determined; according to the first rotation matrix and a second rotation matrix corresponding to an initial relative pose between every two cameras, calculating a third rotation matrix corresponding to a relative pose change relation between every two cameras, and according to the first rotation matrix or the third rotation matrix, calculating a relative pose change amount between every two cameras, therefore, the current pose of the camera is corrected, and the precision of positioning and map construction by using multiple cameras is improved. Or the display content in the binocular display lens is adjusted according to the pose change, so that the problems of image ghosting and the like are avoided, and the image display quality of the intelligent glasses is improved.

Description

Technical Field

[0001] The present application relates to the field of smart wearable technology, and in particular to a posture compensation method, device, equipment and storage medium for a smart glasses camera. Background Art

[0002] Smart glasses, also known as smart mirrors, are like smart phones, with an independent operating system. Users can install software, games and other programs provided by software service providers.

[0003] By installing a camera on smart glasses, it is possible to shoot videos and photos from a first-person perspective, thereby improving the user experience. However, when installing a camera on smart glasses, the camera is generally installed on the frame of the smart glasses. When the frame of the smart glasses is permanently deformed or deformed by external forces, the relative positions of multiple cameras or optical waveguide display lenses on the frame change relative to the initial relative positions before leaving the factory, resulting in large errors when using multiple cameras for positioning and map construction, or causing double images in the binocular display screen, etc., resulting in low display quality of the smart glasses, which leads to a poor user experience.

[0004] Therefore, how to improve the picture display quality of smart glasses has become a technical problem that needs to be solved urgently. Summary of the invention

[0005] The present application provides a posture compensation method, device, equipment and storage medium for a smart glasses camera, aiming to improve the picture display quality of the smart glasses.

[0006] In a first aspect, the present application provides a posture compensation method for a smart glasses camera, comprising:

[0007] Based on current relative angle data of at least two cameras of the smart glasses in a Cartesian coordinate system, calculating a first rotation matrix corresponding to current relative postures between each of the cameras;

[0008] Obtaining a second rotation matrix corresponding to the initial relative postures between each of the cameras;

[0009] Based on the first rotation matrix and the second rotation matrix, calculating a third rotation matrix corresponding to a relative posture change relationship, wherein the relative posture change relationship is a posture change relationship of the current relative posture relative to the initial relative posture;

[0010] Based on the first rotation matrix or the third rotation matrix, posture compensation is performed on the current relative postures between each of the cameras.

[0011] In a second aspect, the present application also provides a posture compensation device for a smart glasses camera, comprising:

[0012] A first rotation matrix calculation module, used to calculate a first rotation matrix corresponding to the current relative posture between each of the at least two cameras of the smart glasses based on the current relative angle data of the at least two cameras in the Cartesian coordinate system;

[0013] A second rotation matrix acquisition module is used to acquire a second rotation matrix corresponding to the initial relative postures between each of the cameras;

[0014] A third rotation matrix calculation module, used to calculate a third rotation matrix corresponding to a relative posture change relationship based on the first rotation matrix and the second rotation matrix, wherein the relative posture change relationship is a posture change relationship of the current relative posture relative to the initial relative posture;

[0015] A posture compensation module is used to perform posture compensation on the current relative postures between each of the cameras based on the first rotation matrix or the third rotation matrix.

[0016] In a third aspect, the present application also provides a computer device, comprising a processor, a memory, and a computer program stored in the memory and executable by the processor, wherein when the computer program is executed by the processor, the steps of the posture compensation method of the smart glasses camera as described above are implemented.

[0017] In a fourth aspect, the present application also provides a computer-readable storage medium, on which a computer program is stored, wherein when the computer program is executed by a processor, the steps of the posture compensation method of the smart glasses camera as described above are implemented.

[0018] The present invention provides a posture compensation method, device, equipment and storage medium for smart glasses cameras. The method comprises calculating a first rotation matrix corresponding to the current relative posture between each pair of cameras based on current relative angle data of at least two cameras of the smart glasses in a Cartesian coordinate system; obtaining a second rotation matrix corresponding to the initial relative posture between each pair of cameras; calculating a third rotation matrix corresponding to a relative posture change relationship based on the first rotation matrix and the second rotation matrix, wherein the relative posture change relationship is a posture change relationship of the current relative posture relative to the initial relative posture; and performing posture compensation on the current relative posture between each pair of cameras based on the first rotation matrix or the third rotation matrix. In the above manner, according to the current relative angle data of at least two cameras on the smart glasses, the first rotation matrix corresponding to the current relative posture of the two cameras is calculated, so as to facilitate the determination of the relative posture change between the cameras; and according to the first rotation matrix and the second rotation matrix corresponding to the initial relative posture between the cameras, the third rotation matrix corresponding to the relative posture change relationship between the cameras is calculated, and then according to the first rotation matrix or the third rotation matrix, the relative posture change between the cameras is calculated, and then the posture compensation of the camera is performed, and the current posture of the camera is corrected, so as to improve the accuracy of positioning and building maps using multiple cameras. Or according to the posture change, the display content in the binocular display lens is adjusted to avoid problems such as image ghosting, and improve the image display quality of the smart glasses. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0020] Figure 1 A schematic diagram of a flow chart of a first embodiment of a posture compensation method for a smart glasses camera provided in an embodiment of the present application;

[0021] Figure 2 A schematic diagram of a coordinate system of a pair of smart glasses provided in an embodiment of the present application when they are worn and used;

[0022] Figure 3 A schematic flow chart of a second embodiment of a posture compensation method for a smart glasses camera provided in an embodiment of the present application;

[0023] Figure 4 A schematic flow chart of a third embodiment of a posture compensation method for a smart glasses camera provided in an embodiment of the present application;

[0024] Figure 5 It is a structural schematic diagram of a first embodiment of a posture compensation device for a smart glasses camera provided by the present application;

[0025] Figure 6 It is a schematic block diagram of the structure of a computer device provided in an embodiment of the present application.

[0026] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0027] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0028] The flowcharts shown in the accompanying drawings are only examples and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may also be decomposed, combined or partially merged, so the actual execution order may change according to actual conditions.

[0029] In conjunction with the accompanying drawings, some embodiments of the present application are described in detail below. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0030] It is understandable that the posture compensation method of the smart glasses camera provided in this application can be applied to the posture compensation of two cameras, and can also be applied to the posture compensation of multiple (more than two) cameras. The posture compensation method of multiple cameras is usually implemented based on the posture compensation between the cameras. For example, taking any camera as a reference, the posture of each camera is adjusted according to the relative posture change of each other camera relative to the reference camera, thereby realizing the relative posture compensation of multiple cameras. Therefore, in order to facilitate the understanding of those skilled in the art, the embodiment of the present application takes the posture compensation of two cameras on smart glasses as an example to explain the posture compensation method of a smart glasses camera provided in this application, but it is not used as a limiting condition.

[0031] Please refer to Figure 1 , Figure 1 A flowchart of a first embodiment of a posture compensation method for a smart glasses camera provided in an embodiment of the present application.

[0032] like Figure 1 As shown, the posture compensation method of the smart glasses camera includes steps S101 to S104.

[0033] Step S101, based on current relative angle data of at least two cameras of the smart glasses in a Cartesian coordinate system, calculating a first rotation matrix corresponding to current relative postures between each of the cameras;

[0034] In one embodiment, the relative angle relationship between two cameras is calculated by acquiring three-dimensional angle data of the cameras in a Cartesian coordinate system.

[0035] In one embodiment, the Cartesian coordinate system may be a three-dimensional rectangular coordinate system.

[0036] For example, Figure 2 As shown, when the user wears and uses the smart glasses, the user's head and the smart glasses as a whole can have the horizontal right direction as the positive direction of the x-axis, the horizontal forward direction as the positive direction of the y-axis, and the vertical upward direction as the positive direction of the z-axis.

[0037] In one embodiment, based on a magnetometer installed on the side of each of the cameras, current three-axis magnetometer data is collected, and based on the current three-axis magnetometer data, the current azimuth data of the camera in the Cartesian coordinate system is calculated; or based on a magnetometer and an inertial measurement unit installed on the side of each of the cameras, current three-axis accelerometer data, three-axis gyroscope data and three-axis magnetometer data are collected, and based on the current three-axis accelerometer data, three-axis gyroscope data and three-axis magnetometer data, the current azimuth data of each of the cameras in the Cartesian coordinate system is calculated; based on the current azimuth data, the current relative angle data between each of the cameras is calculated.

[0038] Among them, the magnetometer (Magnetic, M-Sensor), also known as geomagnetism and magnetic sensor, can be used to test the strength and direction of the magnetic field and locate the orientation of the device. The principle of the magnetometer is similar to that of the compass, and can measure the angle between the current device and the four directions of east, south, west and north.

[0039] The magnetometer is used to measure the magnetic field data of the location to calculate the current posture of the device and the azimuth of the geomagnetic direction. It can be understood that the earth's magnetic field is like a bar magnet, pointing from the magnetic south pole to the magnetic north pole. The geomagnetic field is a vector, and for a fixed location, this vector can be decomposed into three mutually perpendicular components. Two of the components are horizontal to the ground, and the other is perpendicular to the ground. If the electronic compass is parallel to the horizontal plane, the three axes of the magnetometer will correspond to these three components respectively. By looking at the data on the two horizontal components, the azimuth of the electronic compass relative to the magnetic north direction can be calculated.

[0040] An inertial measurement unit (IMU) usually contains a gyroscope and an accelerometer. The gyroscope measures the angular velocity, which is the speed at which an object rotates. By multiplying the velocity by time, we can get the angle that the object rotates in a certain period of time.

[0041] In one embodiment, the magnetometer and the camera are installed adjacent to each other, and the adjacent camera and the magnetometer use the same coordinate system, thereby synchronizing the posture of the magnetometer and the camera, and then the azimuth angle data of the camera can be collected by the magnetometer.

[0042] In one embodiment, the magnetometer can be embedded in the frame of the smart glasses, or fixedly installed on the frame of the smart glasses.

[0043] In one embodiment, any two Cartesian coordinate systems in space, that is, orthogonal coordinate systems that comply with the right-hand rule, can be transformed into one another by rotating around three axes in sequence. Rotation matrices and Euler angles can be converted and calculated mutually.

[0044] In one embodiment, when a magnetometer is used to collect angle data, an inertial measurement unit (IMU) can also be provided to collect three-axis acceleration data, three-axis gyroscope data and three-axis magnetometer data, and then the data of the magnetometer and the inertial measurement unit are fused respectively to obtain a three-degree-of-freedom 3DoF (Degree of freedom) posture, that is, the 3DoF posture of the cameras on both sides.

[0045] Among them, 3DoF posture mainly refers to the orientation, that is, the three rotation directions are called pitch, yaw and roll.

[0046] Among them, the inertial measurement unit is mainly used to detect and measure acceleration, tilt, shock, vibration, rotation and multi-degree-of-freedom motion. It usually refers to a device that uses accelerometers and gyroscopes to measure the single-axis, dual-axis or tri-axis attitude angle (or angular rate) and acceleration of an object.

[0047] In one embodiment, when the magnetometer and inertial measurement unit data are fused, the initial 3DoF pose is first calculated based on the magnetometer data, and then the attitude is calculated based on the three-axis accelerometer (acceleration) and three-axis gyroscope data (angular velocity).

[0048] Exemplarily, the process of data fusion of a magnetometer and an inertial measurement unit may include steps such as data correction, coordinate conversion, and attitude fusion / filtering.

[0049] Data correction includes acceleration data correction, magnetometer data correction and gyroscope data correction.

[0050] The acceleration data correction process includes: collecting the acceleration sensor value of the inertial measurement unit in a stationary state, obtaining acceleration sample data, and then using the least squares method to calculate the various parameters of the ellipse equation, and further calculating the center of the sphere (offset) and the axis length to achieve the correction of the acceleration data.

[0051] The magnetometer data correction process includes extracting sample data of the magnetometer, using the least square method to find out the parameters of the ellipsoid equation, and further finding out the center (offset) and axis length to achieve correction of the magnetometer data.

[0052] The correction of gyroscope data is to average the gyroscope data of the inertial measurement unit when it is stationary, and then the offset can be obtained.

[0053] Among them, in the spatial rectangular coordinate system, the surface represented by the elliptical equation x2 / a2+y2 / b2+z2 / c2=1 is called an ellipsoid, or an elliptical surface, where a, b, and c are arbitrary normal numbers, and it is usually assumed that a≥b≥c>0.

[0054] By correcting the acceleration data, magnetometer data and gyroscope data, the accuracy of the data fusion of the magnetometer and inertial measurement unit can be improved, thereby improving the accuracy of the camera posture data.

[0055] The base coordinate system of the magnetometer and the base coordinate system of the accelerometer are not necessarily the same. For example, the x-axis of the base coordinate system of the accelerometer points to the east, while that of the magnetometer points to the north. Therefore, the magnetometer coordinate system needs to be converted to the accelerometer coordinate system in order to calculate the correct attitude through the accelerometer / magnetometer solution.

[0056] When performing attitude fusion of the magnetometer and the inertial measurement unit, it is assumed that the gravity acceleration vector measured by the accelerometer is u g , the magnetic flux measured by the magnetometer is v m .

[0057] First, we need to set v m The projection equation from the ellipsoid to the sphere is r m =Av m . r m is the projected vector, and A is a 3×3 matrix.

[0058] Secondly, find r m Convert to the rotation matrix R of the accelerometer coordinate system to get u m =Rr m Regardless of the situation, the magnetic flux u m and the gravitational acceleration u g The angle is fixed, that is, u g *u m =|ug |*|u m |cosθ, because the coordinate system fusion only cares about the direction and not the size, so the fixed value |u g |*|u m |cosθ can be normalized. In summary, we have u g R m =1, let B = RA, then we have u g Bv m = 1. The ellipsoidal transformation, spherical transformation and rotation transformation are already included in the matrix B. The least square method can be used again to find the elements of B.

[0059] Then, various attitude fusion algorithms (such as quaternion method, first-order complementary algorithm and Kalman filter algorithm, etc.) are used for attitude fusion. For example, the Kalman filter algorithm uses the gyroscope to estimate the quaternion at time t, and then calculates the gravity acceleration and magnetic flux vector under the estimated state, and then uses the gravity acceleration and magnetic flux vector measured by the accelerometer / magnetometer to correct according to the Kalman filter correction equation.

[0060] In one embodiment, the representation methods of 3DoF posture mainly include rotation matrix, rotation vector, quaternion, and Euler angle, which can be converted into each other. After calculating the 3DoF postures of the two cameras, the rotation matrix of the relative posture of the two cameras can be calculated.

[0061] In one embodiment, in their respective coordinate systems, the data measured by the two magnetometers are respectively [x L ,y L ,z L ] T ,[x R ,y R ,z R ] T , then the rotation matrix of the left magnetometer relative to the right magnetometer is R M , are matrices with 3 rows and 3 columns respectively.

[0062] In one embodiment, the calculation formula of the first rotation matrix is:

[0063] R M *[x R ,y R ,z R ] T =[x L ,y L ,z L ] T

[0064] Among them, x L ,y L ,z L is the three-axis magnetometer data collected by the first magnetometer, xR ,y R ,z R is the three-axis magnetometer data collected by the second magnetometer. Through the above calculation formula, the rotation matrix of the relative position of the two cameras can be calculated.

[0065] In one embodiment, Euler angles are a mechanism for creating general rotations based on three simpler rotations, called pitch, roll, and yaw. Pitch, roll, and yaw are rotations around the x-axis, y-axis, and z-axis, respectively. Assuming that the angles of rotation around the x, y, and z axes are α, β, and γ, respectively, the rotation matrices for the three rotations are calculated as follows:

[0066]

[0067]

[0068]

[0069] The product of these three matrices can be used to represent general rotations, so the general rotation matrix M in 3D space can be expressed as the product of three rotation matrices:

[0070]

[0071] In one embodiment, according to the internal rotation method, the zyx rotation order (first around its own axis z, then around its own axis y, and finally around its own axis x) can be obtained as the rotation matrix (internal rotation is right multiplication):

[0072] R1=R z (γ)*R y (β)*R x (α)

[0073] In one embodiment, according to the external rotation method, the xyz rotation order (first around the fixed axis x, then around the fixed axis y, and finally around the fixed axis z) can be obtained as the rotation matrix (external rotation is left multiplication):

[0074] R2=R z (γ)*R y (β)*R x (α)

[0075] Among them, R1=R2. This conclusion shows that the internal rotation of the ZYX sequence is equivalent to the external rotation of the XYZ sequence.

[0076] Step S102, obtaining a second rotation matrix corresponding to the initial relative postures between each of the cameras;

[0077] In one embodiment, the initial three-dimensional angle data of the smart glasses camera calibrated at the factory is first obtained, and then the initial relative posture of the two cameras of the smart glasses is obtained, and then the rotation matrix of the initial relative posture of the two cameras, that is, the second rotation matrix, is calculated.

[0078] In one embodiment, the second rotation matrix is ​​calculated in the same manner as the first rotation matrix, that is, according to the initial 3DoF poses of the two cameras, the rotation matrix of the pose of one camera relative to the pose of the other camera is calculated, that is, the second rotation matrix R C .

[0079] Step S103: calculating a third rotation matrix corresponding to a relative posture change relationship based on the first rotation matrix and the second rotation matrix, wherein the relative posture change relationship is a posture change relationship of the current relative posture relative to the initial relative posture;

[0080] In one embodiment, the relative angle relationship of the cameras on both sides of the smart glasses (i.e., the first rotation matrix R M ) relative to the initial relative angle relationship obtained by factory calibration (the second rotation matrix, denoted as R0) of the three-axis angle changes (respectively denoted as α, β, γ, and the corresponding third rotation matrix R C );

[0081] In one embodiment, the third rotation matrix calculation method corresponding to the relative posture change relationship is:

[0082] R C *R0=R M

[0083] Thus, the third rotation matrix R can be obtained C And the corresponding three-axis angle changes α, β, and γ.

[0084] Step S104: Based on the first rotation matrix or the third rotation matrix, perform posture compensation on the current relative postures between each pair of the cameras.

[0085] In one embodiment, a first rotation matrix R is used M Or the third rotation matrix R C Compensate for the relative angle relationship between the cameras on both sides.

[0086] In one embodiment, when posture compensation is performed based on the relative posture of the two cameras, posture compensation is performed on the current relative posture of the cameras based on the first rotation matrix.

[0087] For example, when using multiple cameras for real-time positioning and map construction, it is necessary to know the relative position relationship between the cameras. In this case, the first rotation matrix R can be used.M Calculate the relative posture change of the current relative posture relative to the initial relative posture, and then compensate.

[0088] In one embodiment, data correction is performed on the current relative posture according to the change in relative posture, such as position coordinates and deflection angle, and on the basis of the current relative posture, addition and subtraction operations are performed on the data of the current relative posture according to the change in relative posture, so that the relative posture data of the two cameras are restored to the relative posture data corresponding to the initial relative posture, thereby realizing posture compensation of the two cameras.

[0089] Among them, simultaneous localization and mapping (SLAM) is used for the parallel construction of the environment model and the state estimation of the mobile robot in it. The three-dimensional data of the surrounding environment is collected through the binocular camera, and the positioning is performed by matching the feature points of the three-dimensional data. Then, based on the three-dimensional positioning, the complete three-dimensional data is solved and matched.

[0090] In another embodiment, when performing posture compensation based on the relative posture change of the two cameras, posture compensation is performed on the current relative posture of the two cameras based on the third rotation matrix.

[0091] In one embodiment, one or more of the Euler angles α, β, and γ may be used for relative posture compensation, and the Euler angles and the third rotation matrix may be converted to each other.

[0092] In one embodiment, based on the third rotation matrix, the current Euler angles corresponding to each of the cameras are calculated; the initial Euler angles corresponding to each of the cameras are obtained; and based on the initial Euler angles and the current Euler angles, posture compensation is performed on the current relative postures between each of the cameras.

[0093] In one embodiment, a mapping relationship between the relative angle change and the relative position change can be established. When the relative angle change is used to calculate the relative position change, and then relative angle and relative position compensation is performed, one or more of the Euler angles α, β, and γ need to be used, and these Euler angles are R C It is calculated that at this time, the third rotation matrix R is needed C .

[0094] In one embodiment, the Euler angle uses three angles to describe the rotation in three-dimensional space. The Euler angle includes three angles, which respectively represent the angles of rotation around three coordinate axes (X, Y, Z), such as (α, β, γ), where α represents the angle of rotation around the X axis, β represents the angle of rotation around the Y axis, and γ represents the angle of rotation around the Z axis. The rotation matrix is ​​a way to describe the rotation of three-dimensional space. It is a 3×3 matrix that represents the result of rotating a vector around three coordinate axes. In the coordinate system, each column of the rotation matrix represents the direction of the coordinate axis after rotation.

[0095] For example, if the third rotation matrix R is given C The Euler angles are (α, β, γ); the third rotation matrix can be expressed as:

[0096]

[0097] At this point, the sequence of rotations, first around the X axis, then around the Y axis, and finally around the Z axis, can be expressed as a matrix product:

[0098]

[0099]

[0100] Given a rotation matrix R, we can multiply each element in R by the matrix R z (γ), R y (β), R x The corresponding elements in (α) are equivalent, and the Euler angles α, β, and γ can be calculated. From this, nine equations can be obtained to calculate the Euler angles. For example, R 31 = -sinβ, and by reversing the equation we get β = -asin(R 31 ), and so on, to find the current Euler angles α, β, γ.

[0101] Furthermore, based on the initial Euler angle and the current Euler angle, the angle change of each camera is calculated; based on the angle change of each camera, the current posture of each camera is compensated to achieve posture compensation for the current relative posture between each camera.

[0102] In one embodiment, the initial Euler angle of the camera when it leaves the factory is fixed. At this time, the relative posture relationship between the cameras is also fixed. When the camera angle shifts, its Euler angle also changes, that is, the current Euler angle shifts relative to the initial Euler angle, and the offset is the angle change of each camera.

[0103] In one embodiment, when calculating the angle change, the angle difference between the initial Euler angle and the current Euler angle corresponding to each axis can be used to calculate the offset of the Euler angle of each axis, thereby obtaining the angle changes Δα, Δβ, and Δγ of the three axes.

[0104] In one embodiment, each rotation matrix has corresponding camera pose data, and the current Euler angles corresponding to each camera can be calculated in sequence according to the camera pose data corresponding to the rotation matrix. The initial Euler angles corresponding to each camera are factory calibrated, so that the current Euler angles of each camera can correspond to the initial Euler angles.

[0105] In one embodiment, when the angle change of the camera is calculated, the current Euler angle of the camera is added or subtracted according to the angle change, so that the Euler angle data of each camera is transformed from the current Euler angle to the initial Euler angle. When the Euler angle data of all cameras are corrected to the initial Euler angle, the relative angle relationship between the cameras is also restored to the initial state, thereby achieving posture compensation for the relative posture of each camera.

[0106] The present embodiment provides a posture compensation method for a camera of a smart glasses. The method calculates a first rotation matrix corresponding to the current relative posture of the two cameras based on the current relative angle data of the cameras on both sides of the smart glasses, so as to facilitate determining the relative posture change between the two cameras; and calculates a third rotation matrix corresponding to the relative posture change relationship between the two cameras based on the first rotation matrix and the second rotation matrix corresponding to the initial relative posture of the two cameras, and then calculates the relative posture change between the two cameras based on the first rotation matrix or the third rotation matrix, and then performs posture compensation on the camera, corrects the current posture of the camera, and then adjusts the image captured by the camera to avoid problems such as image ghosting, thereby improving the user experience.

[0107] Please refer to Figure 3 , Figure 3 A flowchart of a second embodiment of a posture compensation method for a smart glasses camera provided in an embodiment of the present application.

[0108] like Figure 3 As shown, based on the above Figure 1 In the illustrated embodiment, the posture compensation of the current relative posture of the two cameras based on the third rotation matrix specifically includes:

[0109] Step S201, obtaining initial posture data and current posture data of each camera;

[0110] In one embodiment, the initial posture data includes initial azimuth data and initial position data.

[0111] In one embodiment, the initial posture data of the camera includes the three-dimensional azimuth data and position data calibrated before the smart glasses leave the factory. Because the camera is installed in the frame, when the frame of the smart glasses is deformed, the azimuth data and position data of the camera will change, that is, the current posture data is changed relative to the initial posture data.

[0112] Among them, the initial posture data is the initial calibration data of the camera of the smart glasses device before leaving the factory, and the current posture data is the camera posture collected during use, including angle and position.

[0113] In one embodiment, the relative angle relationship can be measured by a device such as a magnetometer, and the relative position relationship can be measured by a displacement sensor.

[0114] Step S202, calculating the relative angle change between each of the cameras based on the third rotation matrix, the initial posture data and the current posture data;

[0115] In one embodiment, based on the current posture data and the initial posture data, the initial relative angle relationship and the current relative angle relationship between the two cameras can be calculated respectively. The third rotation matrix represents the two relative angle relationships, thereby calculating the relative angle change between the two cameras.

[0116] In a specific embodiment, based on the initial posture data of the two cameras, the initial axial relative angles of the two cameras in three axes can be calculated, thereby obtaining the initial relative angle between the two cameras; similarly, based on the current posture data of the two cameras, the initial axial relative angles of the two cameras in three axes can be calculated, thereby obtaining the current relative angle between the two cameras. The third rotation matrix represents the three-axis angle change of the current relative angle of the two cameras relative to the initial relative angle. Therefore, by substituting the initial relative angle and the current relative angle into the third rotation matrix, the relative angle change between the two cameras can be calculated.

[0117] Step S203: determining the relative position changes between the cameras based on the preset posture mapping relationship and the relative angle changes;

[0118] Step S204: performing posture compensation on the current relative posture based on the relative angle change and the relative position change.

[0119] In one embodiment, the posture mapping relationship includes a mapping relationship between a relative angle change and a relative position change.

[0120] In one embodiment, before the smart glasses leave the factory, the relative change in position between the cameras on both sides can be determined based on the three-axis angle change, and a mapping relationship between the relative angle change and the relative position change can be established.

[0121] In one embodiment, the relative change in position is obtained according to the change in the relative angle relationship of the current relative angle relationship of the cameras relative to the initial relative angle relationship, and then the change in the position relationship of the cameras on both sides is compensated. That is, according to the relative change in position, the data of the current relative positions of the two cameras are corrected, and the position data of the current relative position is restored to the position data of the initial relative position by addition and subtraction, so as to achieve compensation for the relative positions of the two cameras.

[0122] This embodiment maps the relative position change relationship through the relative angle change relationship of the binocular cameras according to the posture mapping relationship, and then calculates the relative position change amount according to the relative angle change amount of the cameras, thereby achieving simultaneous compensation for the relative angle and relative position of the binocular cameras, thereby improving the posture compensation accuracy of the binocular cameras.

[0123] Furthermore, if Figure 4 As shown, based on the above Figure 3 In the illustrated embodiment, after step S203, the following steps are specifically included:

[0124] Step S205: determining a relative position change of the camera based on the relative angle change and the relative position change;

[0125] In one embodiment, the posture of the camera includes an angle and a position, and the relative posture change includes a relative angle change and a relative position change.

[0126] Step S206, calculating the central posture change of the binocular lens display area of ​​the smart glasses based on a preset posture change mapping relationship and the relative posture change of the camera, wherein the posture change mapping relationship includes a posture change mapping relationship between the central posture of the binocular lens display area and the posture of the camera;

[0127] It is understandable that the relative posture compensation method of the binocular camera can be used for optical waveguide image adjustment. In smart glasses, the display area on the binocular waveguide lens will be imaged before leaving the factory so that the images seen by both eyes overlap, thereby avoiding problems such as ghosting and virtual images and improving the image display quality. In actual use, due to changes in external forces, the frame of the smart glasses may be temporarily or permanently deformed, which may cause the same display content on the binocular waveguide lens observed by the user to separate, affecting the user experience.

[0128] Among them, an optical waveguide is a medium device that guides light waves to propagate in it, also known as a dielectric optical waveguide. An optical waveguide is a guiding structure composed of a light-transparent medium (such as quartz glass) that transmits electromagnetic waves of optical frequencies. The transmission principle of an optical waveguide is that at the interface of media with different refractive indices, the total reflection of electromagnetic waves causes the light waves to propagate confined to the waveguide and a limited area around it.

[0129] In one embodiment, before the smart glasses leave the factory, a mapping relationship between the binocular waveguide lens posture and the two side camera posture can be established based on the distance relationship between the center point of the display area on the binocular waveguide lens and the distance between the two side cameras.

[0130] Exemplarily, the two lenses of the binocular lens correspond to a display area respectively, the two display areas are of the same size, and the display screen is also the same, but each lens corresponds to the camera on the same side, and the display area of ​​the two lenses of the binocular lens is the overlapping area of ​​the two camera capture screens. At this time, the central posture of the display area is synchronized with the camera posture. If the relative posture of the two cameras remains unchanged, the relative posture of the two display areas remains unchanged, and the screen display is normal; if the relative posture of the two cameras changes, for example, the posture of the camera on one side remains unchanged, and the posture of the camera on the other side is offset, because of the posture change mapping relationship, the posture of one display area remains unchanged, while the posture of the other display area is synchronously offset with the corresponding camera posture. At this time, the display screen will have problems such as ghosting and virtual images. At this time, according to the relative posture change of the camera and the posture change mapping relationship, the central posture change of the display area can be calculated, and then according to the central posture change, the central posture of the display area is compensated, and the central posture of the display area is corrected, so that the central posture of the display area is restored to the initial posture state, and the problems of ghosting and virtual images can be eliminated.

[0131] Step S207: Based on the central posture change, perform posture compensation on the central posture of the binocular lens display area.

[0132] In one embodiment, the position change of the center point of the binocular waveguide lens display area is calculated according to the relative position change of the camera, and the display content is adjusted accordingly so that the content observed by the user's two eyes overlaps when wearing it.

[0133] Exemplarily, the posture data of the center point of the display area is corrected according to the relative posture change through addition and subtraction. For example, if the current position coordinates of the center point are offset relative to the initial position coordinates, the current position coordinates of the center point are added / subtracted according to the relative position change, so that the current position coordinates of the center point are restored to the initial position coordinates, thereby achieving compensation for the position coordinates of the center point. If the relative angles of the two display areas are offset, that is, when the center points of the two display areas coincide, the images do not coincide, then according to the relative angle change, the current three-axis angle data of the two display areas are added and subtracted, so that the display areas coincide, that is, compensation for the relative angles of the display areas is achieved.

[0134] In this embodiment, by establishing a posture mapping relationship between the posture of the center point of the display screen of the binocular waveguide lens and the posture of the camera, the posture change of the center point of the display screen is calculated according to the posture change mapping of the camera, and then the display screen of the smart glasses is adjusted, so that the screen display of the smart glasses is clearer, avoiding problems such as ghosting, and improving the user experience.

[0135] See also Figure 5 , Figure 5 It is a structural schematic diagram of a first embodiment of a posture compensation device for a smart glasses camera provided in the present application, and the posture compensation device for a smart glasses camera is used to execute the aforementioned posture compensation method for a smart glasses camera.

[0136] like Figure 5 As shown, the posture compensation device 300 of the smart glasses camera includes: a first rotation matrix calculation module 301, a second rotation matrix acquisition module 302, a third rotation matrix calculation module 303 and a posture compensation module 304. Among them:

[0137] A first rotation matrix calculation module 301 is used to calculate a first rotation matrix corresponding to the current relative posture between the two cameras based on the current relative angle data of at least two cameras of the smart glasses in a Cartesian coordinate system;

[0138] A second rotation matrix acquisition module 302 is used to acquire a second rotation matrix corresponding to the initial relative postures between the cameras;

[0139] A third rotation matrix calculation module 303 is used to calculate a third rotation matrix corresponding to a relative posture change relationship based on the first rotation matrix and the second rotation matrix, wherein the relative posture change relationship is a posture change relationship of the current relative posture relative to the initial relative posture;

[0140] The posture compensation module 304 is used to perform posture compensation on the current relative postures between each of the cameras based on the first rotation matrix or the third rotation matrix.

[0141] In one embodiment, the posture compensation module 304 is also used to obtain the initial posture data and the current posture data of each of the cameras; based on the third rotation matrix, the initial posture data and the current posture data, calculate the relative angle change between each of the cameras; based on the preset posture mapping relationship and the relative angle change, determine the relative position change between each of the cameras; based on the relative angle change and the relative position change, perform posture compensation on the current relative posture.

[0142] In one embodiment, the initial posture data includes initial azimuth data and initial position data, and the posture mapping relationship includes a mapping relationship between relative angle change and relative position change.

[0143] In one embodiment, the posture compensation module 304 is further used to determine the relative posture change between the cameras based on the relative angle change and the relative position change; calculate the center posture change of the binocular display area of ​​the smart glasses based on a preset posture change mapping relationship and the relative posture change between the cameras, wherein the posture change mapping relationship includes a posture change mapping relationship between the center posture of the binocular display area and the camera posture; based on the center posture change, perform posture compensation on the center posture of the binocular display area.

[0144] In one embodiment, the posture compensation module 304 is also used to calculate the current Euler angle corresponding to each of the cameras based on the third rotation matrix; obtain the initial Euler angle corresponding to each of the cameras; and perform posture compensation on the current relative posture between each of the cameras based on the initial Euler angle and the current Euler angle.

[0145] In one embodiment, the posture compensation module 304 is also used to calculate the angle change of each camera based on the initial Euler angle and the current Euler angle; based on the angle change of each camera, the current posture of each camera is compensated to achieve posture compensation for the current relative posture between each camera.

[0146] In one embodiment, the posture compensation device 300 of the smart glasses camera also includes a data acquisition module, which is used to collect current three-axis magnetometer data based on the magnetometer installed on the side of each camera, and calculate the current azimuth data of the camera in the Cartesian coordinate system based on the current three-axis magnetometer data; or based on the magnetometer and inertial measurement unit installed on the side of each camera, collect the current three-axis accelerometer data, three-axis gyroscope data and three-axis magnetometer data, and calculate the current azimuth data of each camera in the Cartesian coordinate system based on the current three-axis accelerometer data, three-axis gyroscope data and three-axis magnetometer data; based on the current azimuth data, calculate the current relative angle data between each camera.

[0147] It should be noted that technicians in the relevant field can clearly understand that for the convenience and conciseness of description, the specific working process of the above-described device and each module can refer to the corresponding process in the aforementioned shooting method embodiment based on the split camera module, and will not be repeated here.

[0148] The apparatus provided in the above embodiment may be implemented in the form of a computer program. The computer program may be Figure 6 Runs on the computer device shown.

[0149] See also Figure 6 , Figure 6 1 is a schematic block diagram of a computer device provided in an embodiment of the present application. The computer device may be a server.

[0150] See also Figure 6 The computer device includes a processor, a memory and a network interface connected through a system bus, wherein the memory may include a non-volatile storage medium and an internal memory.

[0151] The non-volatile storage medium can store an operating system and a computer program. The computer program includes program instructions, and when the program instructions are executed, the processor can execute any shooting method based on the split camera module.

[0152] The processor is used to provide computing and control capabilities and support the operation of the entire computer equipment.

[0153] The internal memory provides an environment for the operation of the computer program in the non-volatile storage medium. When the computer program is executed by the processor, the processor can execute any shooting method based on the split camera module.

[0154] The network interface is used for network communication, such as sending assigned tasks, etc. Those skilled in the art will understand that Figure 6The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0155] It should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc.

[0156] In one embodiment, the processor is used to run a computer program stored in the memory to implement the following steps:

[0157] Based on current relative angle data of at least two cameras of the smart glasses in a Cartesian coordinate system, calculating a first rotation matrix corresponding to current relative postures between each of the cameras;

[0158] Obtaining a second rotation matrix corresponding to the initial relative postures between each of the cameras;

[0159] Based on the first rotation matrix and the second rotation matrix, calculating a third rotation matrix corresponding to a relative posture change relationship, wherein the relative posture change relationship is a posture change relationship of the current relative posture relative to the initial relative posture;

[0160] Based on the first rotation matrix or the third rotation matrix, posture compensation is performed on the current relative postures between each of the cameras.

[0161] In one embodiment, when the processor implements the posture compensation of the current relative posture between each of the cameras based on the third rotation matrix, it is used to implement:

[0162] Obtaining initial posture data and current posture data of each camera;

[0163] Based on the third rotation matrix, the initial posture data and the current posture data, calculating the relative angle change between each of the cameras;

[0164] Determine the relative position change between each of the cameras based on the preset posture mapping relationship and the relative angle change;

[0165] Based on the relative angle change and the relative position change, posture compensation is performed on the current relative posture.

[0166] In one embodiment, the initial posture data includes initial azimuth data and initial position data, and the posture mapping relationship includes a mapping relationship between relative angle change and relative position change.

[0167] In one embodiment, after the processor determines the relative position change between each of the cameras based on the preset posture mapping relationship and the relative angle change, it is further used to implement:

[0168] Determine the relative position change between the two cameras based on the relative angle change and the relative position change;

[0169] Based on a preset posture change mapping relationship and the relative posture change between the cameras, the center posture change of the binocular lens display area of ​​the smart glasses is calculated, wherein the posture change mapping relationship includes a posture change mapping relationship between the center posture of the binocular lens display area and the camera posture;

[0170] Based on the central posture change, posture compensation is performed on the central posture of the binocular lens display area.

[0171] In one embodiment, when the processor implements the posture compensation of the current relative posture between each of the cameras based on the third rotation matrix, it is also used to implement:

[0172] Based on the third rotation matrix, calculate the current Euler angle corresponding to each of the cameras;

[0173] Obtaining the initial Euler angles corresponding to each of the cameras;

[0174] Based on the initial Euler angles and the current Euler angles, posture compensation is performed on the current relative postures between each pair of the cameras.

[0175] In one embodiment, when the processor implements the posture compensation of the current relative posture between each of the cameras based on the initial Euler angle and the current Euler angle, it is used to implement:

[0176] Based on the initial Euler angle and the current Euler angle, calculating the angle change of each camera;

[0177] Based on the angle change of each camera, the current posture of each camera is compensated to achieve posture compensation for the current relative posture between each camera.

[0178] In one embodiment, the processor is further used to implement the following steps before implementing the current relative angle data of the at least two cameras based on the smart glasses in the Cartesian coordinate system and calculating the first rotation matrix corresponding to the current relative posture between the two cameras:

[0179] Based on the magnetometer installed on the side of each camera, current three-axis magnetometer data is collected, and based on the current three-axis magnetometer data, current azimuth data of the camera in the Cartesian coordinate system is calculated;

[0180] Alternatively, based on a magnetometer and an inertial measurement unit installed on the side of each camera, current three-axis accelerometer data, three-axis gyroscope data, and three-axis magnetometer data are collected, and based on the current three-axis accelerometer data, three-axis gyroscope data, and three-axis magnetometer data, current azimuth angle data of each camera in the Cartesian coordinate system is calculated;

[0181] Based on the current azimuth angle data, the current relative angle data between each of the cameras is calculated.

[0182] A computer-readable storage medium is also provided in an embodiment of the present application, wherein the computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and the processor executes the program instructions to implement any one of the shooting methods based on the split camera module provided in the embodiments of the present application.

[0183] The computer-readable storage medium may be an internal storage unit of the computer device described in the above embodiment, such as a hard disk or memory of the computer device. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, a smart memory card (SmartMediaCard, SMC), a secure digital (SecureDigital, SD) card, a flash card (FlashCard), etc. equipped on the computer device.

[0184] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A posture compensation method for a smart glasses camera, characterized in that: The method comprises: Based on current relative angle data of at least two cameras of the smart glasses in a Cartesian coordinate system, calculating a first rotation matrix corresponding to current relative postures between each of the cameras; Obtaining a second rotation matrix corresponding to the initial relative postures between each of the cameras; Based on the first rotation matrix and the second rotation matrix, calculating a third rotation matrix corresponding to a relative posture change relationship, wherein the relative posture change relationship is a posture change relationship of the current relative posture relative to the initial relative posture; Based on the first rotation matrix or the third rotation matrix, posture compensation is performed on the current relative postures between each of the cameras.

2. The method for compensating the position and posture of the smart glasses camera according to claim 1, characterized in that: The performing posture compensation on the current relative postures between the cameras based on the third rotation matrix comprises: Obtaining initial posture data and current posture data of each camera; Based on the third rotation matrix, the initial posture data and the current posture data, calculating the relative angle change between each of the cameras; Determine the relative position change between each of the cameras based on the preset posture mapping relationship and the relative angle change; Based on the relative angle change and the relative position change, posture compensation is performed on the current relative posture.

3. The method for compensating the position and posture of a smart glasses camera according to claim 2, characterized in that: The initial posture data includes initial azimuth data and initial position data, and the posture mapping relationship includes a mapping relationship between a relative angle change and a relative position change.

4. The method for compensating the position and posture of a smart glasses camera according to claim 2, characterized in that: After determining the relative position changes between the two cameras based on the preset posture mapping relationship and the relative angle changes, the method further includes: Determine the relative position change between the two cameras based on the relative angle change and the relative position change; Based on a preset posture change mapping relationship and the relative posture change between the cameras, the center posture change of the binocular lens display area of ​​the smart glasses is calculated, wherein the posture change mapping relationship includes a posture change mapping relationship between the center posture of the binocular lens display area and the camera posture; Based on the central posture change, posture compensation is performed on the central posture of the binocular lens display area.

5. The method for compensating the position and posture of a smart glasses camera according to claim 1, characterized in that: The performing posture compensation on the current relative postures between the cameras based on the third rotation matrix further includes: Based on the third rotation matrix, calculate the current Euler angle corresponding to each of the cameras; Obtaining the initial Euler angles corresponding to each of the cameras; Based on the initial Euler angles and the current Euler angles, posture compensation is performed on the current relative postures between each pair of the cameras.

6. The method for compensating the position and posture of a smart glasses camera according to claim 5, characterized in that: The performing posture compensation on the current relative postures between each pair of the cameras based on the initial Euler angles and the current Euler angles comprises: Based on the initial Euler angle and the current Euler angle, calculating the angle change of each camera; Based on the angle change of each camera, the current posture of each camera is compensated to achieve posture compensation for the current relative posture between each camera.

7. The method for compensating the position and posture of a smart glasses camera according to any one of claims 1 to 6, characterized in that: Before calculating the first rotation matrix corresponding to the current relative postures between the two cameras based on the current relative angle data of the at least two cameras of the smart glasses in the Cartesian coordinate system, the method further includes: Based on the magnetometer installed on the side of each camera, current three-axis magnetometer data is collected, and based on the current three-axis magnetometer data, current azimuth data of the camera in the Cartesian coordinate system is calculated; Alternatively, based on a magnetometer and an inertial measurement unit installed on the side of each camera, current three-axis accelerometer data, three-axis gyroscope data, and three-axis magnetometer data are collected, and based on the current three-axis accelerometer data, three-axis gyroscope data, and three-axis magnetometer data, current azimuth angle data of each camera in the Cartesian coordinate system is calculated; Based on the current azimuth angle data, the current relative angle data between each of the cameras is calculated.

8. A posture compensation device for a smart glasses camera, characterized in that: The posture compensation device of the smart glasses camera includes: A first rotation matrix calculation module, used to calculate a first rotation matrix corresponding to the current relative posture between each of the at least two cameras of the smart glasses based on the current relative angle data of the at least two cameras in the Cartesian coordinate system; A second rotation matrix acquisition module is used to acquire a second rotation matrix corresponding to the initial relative postures between each of the cameras; A third rotation matrix calculation module, used to calculate a third rotation matrix corresponding to a relative posture change relationship based on the first rotation matrix and the second rotation matrix, wherein the relative posture change relationship is a posture change relationship of the current relative posture relative to the initial relative posture; A posture compensation module is used to perform posture compensation on the current relative postures between each of the cameras based on the first rotation matrix or the third rotation matrix.

9. A computer device, characterized in that: The computer device includes a processor, a memory, and a computer program stored in the memory and executable by the processor, wherein when the computer program is executed by the processor, the steps of the posture compensation method of the smart glasses camera as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, the steps of the posture compensation method for the smart glasses camera as described in any one of claims 1 to 7 are implemented.

Images