Camera posture compensation method, device, electronic device and readable storage medium

Abstract

This application discloses a camera pose compensation method, device, electronic device, and readable storage medium. The camera pose compensation method includes: when determining that smart glasses are in a wearing state, dynamically detecting the relative change in the camera's pose, wherein the relative change in pose is the change in the camera's wearing pose parameters relative to the reference pose parameters, and dynamically compensating the reference pose parameters based on the relative change in pose. This application provides a dynamic compensation method for camera pose changes caused by different people wearing smart glasses, dynamically compensating the camera's pose change parameters into factory calibration data, thereby improving the pose positioning accuracy of 6DoF tracking.

Description

Technical Field

[0001] The present application relates to the technical field of virtual reality / augmented reality, and in particular to a camera posture compensation method, device, electronic device, and readable storage medium. Background Art

[0002] VR (Virtual Reality) glasses or AR (Augmented Reality) glasses are rapidly developing and gaining popularity. Many smart glasses on the market, such as AR and VR glasses, often require integrated 6DoF (degree of freedom) tracking systems. 6DoF tracking provides users with unprecedented interactive experiences and control over the virtual world. It can be said that 6DoF tracking is the foundation for establishing a connection between the virtual and real worlds, and the quality of 6DoF tracking directly affects the user experience of AR / VR devices.

[0003] At present, the 6DoF tracking system based on binocular fisheye has been adopted by different AR / VR glasses manufacturers because it provides real-time map initialization, powerful tracking and accurate measurement of environmental scale. Compared with the monocular fisheye 6DoF tracking system, the binocular fisheye camera can expand the scene faster and measure the environmental scale more accurately to prevent drift. The binocular fisheye 6DoF tracking system includes cameras respectively arranged on both sides of the glasses, and an IMU (Inertial measurement unit) sensor arranged at the center of the glasses. Since the 6DoF tracking algorithm calculates the head posture by fusing the data collected by the dual cameras and the IMU sensor, the initial wearing posture (displacement and angle) of the camera directly affects the accuracy of dynamic monitoring of the real-time head posture. Although these smart glasses are calibrated for the initial relative pose of the camera and IMU sensor before leaving the factory, due to differences in head shape and size, the degree of outward expansion of the temples of different people when wearing smart glasses often varies, causing the hinges of the smart glasses to expand to different degrees. This causes the initial wearing pose of the cameras distributed on both sides of the glasses to change, which deviates from the factory calibration value, thereby reducing the pose positioning accuracy of 6DoF tracking. Summary of the Invention

[0004] The main purpose of this application is to provide a camera posture compensation method, device, electronic device and readable storage medium, aiming to solve the technical problem that the posture positioning accuracy of smart glasses is reduced due to changes in the initial wearing posture of the camera.

[0005] To achieve the above objectives, the present application provides a camera posture compensation method, which is applied to smart glasses. The smart glasses include a glasses frame and cameras distributed on both sides of the glasses frame. The camera posture compensation method includes:

[0006] When it is determined that the smart glasses are in a wearing state, dynamically detecting a relative change in the posture of the camera, wherein the relative change in posture is a change in the posture of the camera between the wearing posture parameter and the calibration posture parameter;

[0007] Dynamically compensate the calibration pose parameters according to the relative change in pose.

[0008] Optionally, when determining that the smart glasses are in a wearing state, the step of dynamically detecting a relative change in the position posture of the camera includes:

[0009] When it is determined that the smart glasses are in a wearing state, dynamically detecting an outward deformation amount of a hinge of the smart glasses;

[0010] If the outward deformation amount is greater than a preset deformation threshold, the relative change amount of the camera's posture is dynamically detected.

[0011] Optionally, the smart glasses further include a circuit board independently provided from the glasses frame, and an IMU sensor provided on the circuit board. Before the step of dynamically detecting the relative change in the position and posture of the camera when determining that the smart glasses are in a worn state includes:

[0012] When the smart glasses are not worn, calibrating a relative spatial position vector of the camera relative to the IMU sensor;

[0013] The relative spatial position vector is used as the calibration pose parameter of the camera.

[0014] Optionally, the step of calibrating the relative spatial position vector of the camera relative to the IMU sensor includes:

[0015] The position coordinates of the IMU sensor are used as the origin of the three-axis direction to construct the body coordinate system of the smart glasses;

[0016] Determine the coordinate vector of the camera in the body coordinate system, and use the coordinate vector as the relative spatial position vector of the camera relative to the IMU sensor.

[0017] Optionally, the relative change in posture includes a relative change in position and an angle of the camera, and the step of dynamically compensating the calibration posture parameters based on the relative change in posture includes:

[0018] Determine a first vector parameter corresponding to the relative position change according to the relative position change, and determine a second vector parameter corresponding to the relative angle change according to the relative angle change;

[0019] The first vector parameter and the second vector parameter are dynamically compensated to the calibration pose parameter.

[0020] Optionally, the smart glasses include a Hall sensor and an acceleration sensor, and the step of dynamically detecting the relative change in the position and posture of the camera includes:

[0021] The relative position change is dynamically detected based on the Hall sensor, and the relative angle change is dynamically detected based on the acceleration sensor.

[0022] Optionally, the camera posture compensation method further includes:

[0023] Obtaining a calibration compensation value input by a user, and obtaining an actual compensation value for dynamically compensating the calibration attitude parameters;

[0024] Calculating a deviation between the calibration compensation value and the actual compensation value to obtain a compensation error value for dynamically compensating the calibration attitude parameter;

[0025] Training a self-learning model using the calculated compensation error value as a learning sample;

[0026] The step of dynamically compensating the calibration pose parameters according to the relative change in pose further comprises:

[0027] If the number of the learning samples reaches a preset threshold, a compensation error coefficient for dynamically compensating the calibration pose parameters is obtained based on the prediction of the self-learning model;

[0028] determining a compensation correction coefficient according to the compensation error coefficient;

[0029] Dynamically compensate the calibration pose parameters according to the pose change amount and the compensation correction coefficient.

[0030] The present application also provides a camera posture compensation device, which is applied to smart glasses. The smart glasses include a glasses frame and cameras distributed on both sides of the glasses frame. The camera posture compensation device includes:

[0031] a detection module, which dynamically detects a relative change in the camera's posture when determining that the smart glasses are in a wearing state, wherein the relative change in posture is a change in the camera's wearing posture parameter relative to the calibration posture parameter;

[0032] The compensation module dynamically compensates the calibration pose parameters according to the relative change of the pose.

[0033] The present application also provides an electronic device, which is a physical device, and includes: a memory, a processor, and a program of the camera posture compensation method stored in the memory and runnable on the processor. When the program of the camera posture compensation method is executed by the processor, the steps of the camera posture compensation method as described above can be implemented.

[0034] The present application also provides a readable storage medium, which is a computer-readable storage medium. The computer-readable storage medium stores a program for implementing the camera posture compensation method. The program for implementing the camera posture compensation method is executed by a processor to implement the steps of the camera posture compensation method as described above.

[0035] The present application also provides a computer program product, comprising a computer program, which, when executed by a processor, implements the steps of the camera posture compensation method as described above.

[0036] When the user wears the smart glasses, the temples expand outward, which causes a certain degree of deformation on both sides of the frame. Since different people have different head shapes and sizes, the temples expand outward to different degrees after wearing the smart glasses, resulting in different local deformation on both sides of the frame, which in turn causes the relative posture drift of the camera following the local areas on both sides of the frame to be often different. Therefore, the present application dynamically detects the relative posture change of the camera when determining that the smart glasses are in the wearing state, wherein the relative posture change is the posture change of the camera's wearing posture parameter relative to the reference posture parameter. Each time the user wears the smart glasses, the relative posture drift generated by the camera is dynamically detected, and then the reference posture parameter is dynamically compensated based on the relative posture change, so that the user can wear the smart glasses. After wearing the glasses, the relative posture drift generated by the camera is dynamically compensated to the calibration posture parameters pre-calibrated before leaving the factory, so as to avoid the relative posture drift generated by the camera causing the camera's wearing posture parameters to deviate from the calibration posture parameters pre-calibrated before leaving the factory, thereby resulting in inaccurate head posture calculated by the 6DoF tracking algorithm, thereby improving the posture positioning accuracy of 6DoF tracking and improving the accuracy of dynamic monitoring of real-time head posture. This application provides a dynamic compensation method for changes in camera posture (displacement and rotation) caused by different people wearing smart glasses, and dynamically compensates the camera's posture change parameters to factory calibration data, thereby overcoming the technical problem that the posture positioning accuracy of smart glasses is reduced due to changes in the initial wearing posture of the camera. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0038] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0039] Figure 1 This is a flow chart of the first embodiment of the camera posture compensation method of the present application;

[0040] Figure 2 This is a schematic diagram of the camera's posture drift caused by the hinge's extension in an embodiment of the present application;

[0041] Figure 3 This is a partial enlarged view of the camera generating posture drift in the embodiment of the present application;

[0042] Figure 4 This is a flow chart of the second embodiment of the camera posture compensation method of the present application;

[0043] Figure 5 This is a schematic diagram of the device structure of the hardware operating environment involved in the camera posture compensation device in the embodiment of the present application.

[0044] The purpose, features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0045] To make the above-mentioned objects, features, and advantages of the present invention more clearly understood, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making any creative efforts shall fall within the scope of protection of the present invention.

[0046] In this embodiment, the smart glasses of the present application can be, for example, mixed reality (MR) glasses, augmented reality (AR) glasses, virtual reality (VR) glasses, extended reality (XR) glasses, or some combination thereof.

[0047] SLAM (simultaneous localization and mapping) technology has become increasingly mature. Modern, popular visual SLAM systems can be broadly divided into a front-end and a back-end. The front-end performs data association, equivalent to visual odometry (VO). It processes images captured by the camera on smart glasses, analyzes the transformation relationship between frames, and performs real-time pose tracking and pose change calculations. When smart glasses are equipped with an IMU sensor to collect pose information, it can also participate in fusion calculations (this is the approach of visual inertial odometry (VIO)). The back-end primarily optimizes the front-end output, using filtering or optimization theory to perform tree or graph optimization to obtain the optimal pose estimate and map. With the widespread application of SLAM technology, the combination of 6DoF (degree of freedom) tracking and SLAM has made immersive VR (virtual reality) or AR (augmented reality) experiences possible. Users can break free from the constraints of landmarks and perform real-time pose localization and map construction, resulting in a more immersive and realistic experience with virtual content.

[0048] However, the current 6DoF tracking algorithm with higher pose positioning accuracy calculates the head pose (i.e., VIO) by fusing data collected by the camera and the IMU (Inertial Measurement Unit) sensor. Therefore, the initial wearing pose (position and angle) of the camera directly affects the accuracy of dynamic monitoring of the user's head pose. Although these smart glasses calibrate the initial relative pose of the camera and IMU sensor before leaving the factory, due to the differences in head shape and size, the degree of outward expansion of the temples of the smart glasses often varies when different people wear smart glasses, causing the degree of outward expansion of the hinge of the smart glasses to vary. As a result, the initial wearing pose of the cameras distributed on both sides of the glasses changes, which deviates from the factory calibration value, thereby reducing the pose positioning accuracy of 6DoF tracking.

[0049] Example 1

[0050] Based on this, please refer to Figure 1 This embodiment provides a camera posture compensation method. The smart glasses include a glasses frame and cameras distributed on both sides of the glasses frame. The camera posture compensation method includes:

[0051] Step S10: When it is determined that the smart glasses are in a wearing state, dynamically detecting a relative change in the camera's posture, wherein the relative change in posture is a change in the wearing posture parameter of the camera relative to the calibration posture parameter;

[0052] In this embodiment, the wearing posture parameter of the camera refers to the current posture parameter of the camera when it is determined that the smart glasses are in a wearing state, and the calibration posture parameter can be obtained by pre-calibrating the posture parameter of the camera when the smart glasses are not in a wearing state. Among them, those skilled in the art can understand that the posture parameter may include position and angle, and the posture relative change is the position relative change and angle relative change of the wearing posture parameter of the camera relative to the calibration posture parameter. Specifically, the position relative change can be detected by a displacement sensor of the smart glasses, such as a potentiometer displacement sensor, an inductive displacement sensor, a self-synchronous angle machine, a capacitive displacement sensor, an eddy current displacement sensor or a Hall sensor, and the angle relative change can be detected by an angle sensor of the smart glasses, such as a photoelectric angle sensor, a gyroscope or an acceleration sensor.

[0053] It is understood that since the cameras of this embodiment are distributed on both sides of the glasses frame, the cameras of this embodiment can be binocular cameras or quad cameras, etc. In addition, the cameras of this embodiment can perform 6DoF tracking of the user's head posture based on SLAM (simultaneous localization and mapping) technology.

[0054] In this embodiment, the camera can be an ordinary camera, an RGBD (Red Green Blue Depth, three primary colors depth) camera, or a fisheye camera, which is not specifically limited in this embodiment. The smart glasses can be VR glasses or AR glasses. It should be noted that the smart glasses include a glasses frame, and cameras distributed on both sides of the glasses frame. Due to the differences in head shape and size, when different people wear smart glasses such as AR / VR, the degree of outward expansion of the temples of the smart glasses is often different, and the outward expansion of the temples will cause local deformation on both sides of the glasses frame, causing the initial wearing posture of the cameras distributed on both sides of the glasses frame to change, that is, the camera follows the local area on both sides of the glasses frame to produce relative movement, resulting in relative posture drift. At this time, the wearing posture parameters of the camera deviate from the calibration posture parameters pre-calibrated before leaving the factory, which in turn leads to a decrease in the posture positioning accuracy of 6DoF tracking and requires recalibration.

[0055] Step S20: dynamically compensate the calibration pose parameters according to the relative pose change.

[0056] In the embodiment, since the temples expand outward when the user wears the smart glasses, the two sides of the frame are deformed to a certain extent, resulting in the camera following the local areas on both sides of the frame to produce relative posture drift. It is easy for those skilled in the art to understand that this embodiment, after determining the relative change in the camera's posture (that is, determining the relative posture drift generated by the camera when the user wears the smart glasses), can dynamically compensate for the calibration posture parameters pre-calibrated before leaving the factory based on the relative change in posture. That is to say, the relative change in the posture drift of the camera when the user wears the smart glasses can be dynamically compensated to the relative change in the posture of the calibration posture parameters.

[0057] When the user wears the smart glasses, the temples expand outward, which causes a certain degree of deformation on both sides of the frame. Since different people have different head shapes and sizes, the temples expand outward to different degrees after wearing the smart glasses, resulting in different local deformation on both sides of the frame, which in turn causes the relative posture drift of the camera following the local areas on both sides of the frame to be often different. Therefore, this embodiment dynamically detects the relative posture change of the camera when determining that the smart glasses are in the wearing state, wherein the posture relative change is the posture change of the camera's wearing posture parameter relative to the reference posture parameter. Each time the user wears the smart glasses, the relative posture drift generated by the camera is dynamically detected, and then the reference posture parameter is dynamically compensated based on the posture relative change, so that the user can wear the smart glasses. After wearing the glasses, the relative posture drift generated by the camera is dynamically compensated to the calibration posture parameters pre-calibrated before leaving the factory, so as to avoid the relative posture drift generated by the camera causing the wearing posture parameters of the camera to deviate from the calibration posture parameters pre-calibrated before leaving the factory, thereby causing the head posture calculated by the 6DoF tracking algorithm to be inaccurate, thereby improving the posture positioning accuracy of 6DoF tracking and improving the accuracy of dynamic monitoring of real-time head posture. This embodiment provides a dynamic compensation method for changes in camera posture (displacement and rotation) caused by different people wearing smart glasses, and dynamically compensates the camera posture change parameters to the factory calibration data, thereby overcoming the technical problem that the posture positioning accuracy of smart glasses is reduced due to changes in the initial wearing posture of the camera.

[0058] In one practicable manner, when determining that the smart glasses are in a wearing state, the step of dynamically detecting the relative change in the position of the camera includes:

[0059] Step A10: When it is determined that the smart glasses are in a wearing state, dynamically detecting the outward deformation of the hinge of the smart glasses;

[0060] In one embodiment, a pressure sensor disposed on the temple of the smart glasses can detect the pressure exerted on the temple when the user wears the smart glasses, and then determine the amount of hinge deformation of the smart glasses based on the pressure. Specifically, the hinge deformation amount corresponding to the pressure mapping can be obtained by querying a preset calibrated hinge deformation mapping table. The hinge deformation mapping table includes a one-to-one mapping relationship between the pressure exerted on the temple and the amount of hinge deformation.

[0061] In another embodiment, the hinge's outward deformation can be directly detected by a displacement-type photoelectric sensor. In yet another embodiment, the outward deformation of the temples of the smart glasses can also be detected by a displacement-type photoelectric sensor, and the hinge's outward deformation can be determined based on the outward deformation.

[0062] Step A20: If the outward deformation amount is greater than a preset deformation threshold, dynamically detect the relative change in the camera's posture.

[0063] In this embodiment, the preset deformation threshold is not specifically limited in this embodiment, and is used to better judge whether the outward deformation of the current hinge is sufficient to cause the camera to produce relative posture drift, or to judge whether the outward deformation of the current hinge causes the relative posture change of the camera to be greater than the preset value.

[0064] Because some users have smaller heads, the degree of outward expansion of the temples caused by these users wearing smart glasses is relatively small, resulting in a smaller outward deformation of the hinge of the smart glasses, or no outward deformation of the hinge of the smart glasses. This ensures that the camera's wearing posture parameters are the same or similar to the calibration posture parameters pre-calibrated before leaving the factory, and thus does not affect the posture positioning accuracy of the 6DoF tracking algorithm. In this case, blindly detecting and calculating the relative change in the camera's posture will increase the operating load of the smart glasses. Therefore, this embodiment dynamically detects the outward deformation of the hinge of the smart glasses. When the outward deformation is greater than a preset deformation threshold, the subsequent step of dynamically detecting the relative change in the camera's posture is performed. When it is determined that the current outward deformation of the hinge is sufficient to cause relative posture drift of the camera, or is sufficient to cause the relative posture change of the camera to be greater than a preset value, the relative posture change of the camera is detected and related calculations are performed, thereby reducing the operating load of the smart glasses.

[0065] In order to help understand the technical concept or working principle of this application, a specific embodiment is listed below:

[0066] Please refer to Figure 2 , Figure 2 This is a schematic diagram of the camera's posture drift caused by the hinge's extension. Figure 2In the present invention, the frame of smart glasses includes a front shell and a back shell that are spliced ​​and assembled. The cameras of the smart glasses are fixedly connected to both sides of the front shell. Since different people often have different degrees of expansion of the temples of smart glasses when wearing AR / VR and other smart glasses, the expansion of the temples will cause the hinges of the smart glasses to expand outward, thereby causing the two sides of the back shell to produce outward deformation. Then, the back shell will cause the two sides of the front shell that are integrally assembled with the back shell to produce outward deformation, thereby causing the cameras fixed on both sides of the back shell to follow and produce relative movement, resulting in relative posture drift. At this time, there is a deviation between the wearing posture parameters of the camera and the calibration posture parameters pre-calibrated before leaving the factory, which leads to a decrease in the posture positioning accuracy of 6DoF tracking and requires recalibration.

[0067] Please refer to Figure 3 , Figure 3 This is a partial enlarged view of the camera's posture drift. The camera drifts from the original point to the drift point due to the deformation of the front shell. The relative position change (displacement) and angle change (rotation) generated by the camera drifting from the original point to the drift point are the relative posture change of the camera. Figure 3 In the equation, “Δθ” is the relative change in the camera angle, and the magnitude of the straight line vector from the original point to the drift point is the relative change in the camera position.

[0068] It should be noted that the above specific embodiments are only used to help understand the technical concept or working principle of the present application, and do not constitute a limitation on the hardware structure of the smart glasses of the present application.

[0069] In one practicable manner, please refer to Figure 4 The smart glasses further include a circuit board independently provided from the glasses frame, and an IMU sensor provided on the circuit board. Before the step of dynamically detecting the relative change in the position of the camera when determining that the smart glasses are in a wearing state includes:

[0070] Step S30, when the smart glasses are not worn, calibrating the relative spatial position vector of the camera relative to the IMU sensor;

[0071] Step S40: Using the relative spatial position vector as a calibration pose parameter of the camera.

[0072] In this embodiment, the IMU sensor may include an accelerometer and a gyroscope, and the user's head posture can be monitored in real time through the IMU sensor. Therefore, the smart glasses of this embodiment can calculate the user's head posture by fusing the data collected by the camera and the IMU (Inertial measurement unit) sensor, and perform real-time tracking, thereby improving the accuracy of 6DoF tracking of the user's head posture, thereby creating a more immersive and realistic experience in virtual content.

[0073] In this embodiment, the circuit board and the IMU sensor can be arranged in the middle of the glasses frame to improve the accuracy of the IMU sensor in 6DoF tracking of the user's head posture.

[0074] It is easy to understand that since the camera is fixed on both sides of the glasses frame of the smart glasses, the camera will follow the local areas on both sides of the glasses frame to produce relative movement, resulting in relative posture drift. However, the IMU sensor in this embodiment is fixed on a circuit board independent of the glasses frame. Therefore, the IMU sensor will not follow the glasses frame to produce relative movement, resulting in relative posture drift. Therefore, this embodiment can improve the objectivity and accuracy of calibrating the camera's calibration posture parameters by pre-calibrating the relative spatial position vector of the camera relative to the IMU sensor when the smart glasses are not worn, and using the relative spatial position vector as the camera's calibration posture parameter. In addition, since the subsequent 6DoF tracking algorithm also needs to use the relative spatial position vector of the camera relative to the IMU sensor in the process of calculating the user's head posture, the robustness of the camera posture compensation method of this embodiment is improved.

[0075] Furthermore, the step of calibrating the relative spatial position vector of the camera relative to the IMU sensor includes:

[0076] Step B10: Using the position coordinates of the IMU sensor as the origin of the three-axis direction to construct the body coordinate system of the smart glasses;

[0077] Step B20: Determine the coordinate vector of the camera in the body coordinate system, and use the coordinate vector as the relative spatial position vector of the camera relative to the IMU sensor.

[0078] In this embodiment, it is easy to understand that the origin of the three-axis direction represents the origin on the OXS, OYS, and OZS axes. This embodiment constructs the body coordinate system of the smart glasses by using the position coordinates of the IMU sensor as the origin of the three-axis direction, and determines the coordinate vector of the camera in the body coordinate system. The coordinate vector is used as the relative spatial position vector of the camera relative to the IMU sensor, thereby more intuitively and conveniently calibrating the relative spatial position of the camera relative to the IMU sensor. The calibration process is simple, convenient, and easy to operate, reducing the complexity of calibrating the relative spatial position.

[0079] In one practicable manner, the relative change in posture includes a relative change in position and an angle of the camera, and the step of dynamically compensating the calibration posture parameters based on the relative change in posture includes:

[0080] Step C10, determining a first vector parameter corresponding to the relative position change according to the relative position change, and determining a second vector parameter corresponding to the relative angle change according to the relative angle change;

[0081] Step C20: Dynamically compensate the first vector parameter and the second vector parameter to the calibration pose parameter.

[0082] This embodiment determines the first vector parameter corresponding to the relative change in position based on the relative change in position, and determines the second vector parameter corresponding to the relative change in angle based on the relative change in angle, and then dynamically compensates the first vector parameter and the second vector parameter to the calibration pose parameter, thereby accurately compensating the relative change in pose generated by the camera when the user wears smart glasses to the pre-calibrated calibration pose parameter, thereby improving the robustness of the camera pose compensation method of this embodiment.

[0083] Furthermore, the smart glasses include a Hall sensor and an acceleration sensor, and the step of dynamically detecting the relative change in the position of the camera includes:

[0084] Step D10 : dynamically detecting the relative position change based on the Hall sensor, and dynamically detecting the relative angle change based on the acceleration sensor.

[0085] Since the Hall sensor can more accurately detect the relative change in the camera's position, and the acceleration sensor can more sensitively detect the relative change in the camera's angle, this embodiment dynamically detects the relative change in position through the Hall sensor and dynamically detects the relative change in angle through the acceleration sensor, thereby improving the accuracy of detecting the relative change in the camera's posture when the user wears smart glasses.

[0086] Example 2

[0087] Based on the first embodiment of the present application, in another embodiment of the present application, the same or similar contents as those in the above-mentioned embodiment 1 can be referred to the above introduction and will not be repeated hereafter. On this basis, the camera posture compensation method further includes:

[0088] Step E10, obtaining a calibration compensation value input by a user, and obtaining an actual compensation value for dynamically compensating the calibration pose parameters;

[0089] Step E20, calculating a deviation between the calibration compensation value and the actual compensation value to obtain a compensation error value for dynamically compensating the calibration attitude parameter;

[0090] Step E30, training a self-learning model based on the calculated compensation error value as a learning sample;

[0091] The step of dynamically compensating the calibration pose parameters according to the relative change in pose further comprises:

[0092] Step E40: If the number of the learning samples reaches a preset threshold, a compensation error coefficient for dynamically compensating the calibration pose parameters is obtained based on the self-learning model prediction;

[0093] Step E50, determining a compensation correction coefficient according to the compensation error coefficient;

[0094] Step E60: dynamically compensate the calibration pose parameters according to the pose change amount and the compensation correction coefficient.

[0095] Since, in the above embodiment, the actual compensation value for dynamically compensating the calibration pose parameters according to the relative change in posture may have a certain compensation error, thereby affecting the posture positioning accuracy of 6DoF tracking, and further affecting the user's immersive experience of the smart glasses, this embodiment can calibrate the calibration compensation value belonging to the individual through continuous debugging and experimentation by the amount of deformation generated by the temples when the individual wears the smart glasses, or the relative posture drift generated by the camera, and obtain the compensation error value for dynamically compensating the calibration pose parameters through the deviation value between the calculated calibration compensation value and the actual compensation value, and then calculate the compensation error value according to the calculation result. The calculated compensation error value is used as a learning sample to train the self-learning model. If the number of learning samples reaches a preset threshold, the compensation error coefficient for dynamically compensating the calibration pose parameters is predicted based on the self-learning model, and the compensation correction coefficient is determined according to the compensation error coefficient. Finally, according to the pose change and the compensation correction coefficient, the calibration pose parameters are dynamically compensated, so that the relative pose drift generated by the camera is more accurately compensated to the calibration pose parameters pre-calibrated before leaving the factory, further improving the pose positioning accuracy of 6DoF tracking and the accuracy of dynamic monitoring of real-time head pose.

[0096] Example 3

[0097] An embodiment of the present invention further provides a camera posture compensation device, which is applied to smart glasses. The smart glasses include a glasses frame and cameras distributed on both sides of the glasses frame. The camera posture compensation device includes:

[0098] a detection module, which dynamically detects a relative change in the camera's posture when determining that the smart glasses are in a wearing state, wherein the relative change in posture is a change in the camera's wearing posture parameter relative to the calibration posture parameter;

[0099] The compensation module dynamically compensates the calibration pose parameters according to the relative change of the pose.

[0100] Optionally, the detection module is further configured to:

[0101] When it is determined that the smart glasses are in a wearing state, dynamically detecting an outward deformation amount of a hinge of the smart glasses;

[0102] If the outward deformation amount is greater than a preset deformation threshold, the relative change amount of the camera's posture is dynamically detected.

[0103] Optionally, the smart glasses further include a circuit board provided independently of the glasses frame, and an IMU sensor provided on the circuit board. The smart glasses further include a calibration module, wherein the calibration module is configured to:

[0104] When the smart glasses are not worn, calibrating a relative spatial position vector of the camera relative to the IMU sensor;

[0105] The relative spatial position vector is used as the calibration pose parameter of the camera.

[0106] Optionally, the calibration module is further configured to:

[0107] The position coordinates of the IMU sensor are used as the origin of the three-axis direction to construct the body coordinate system of the smart glasses;

[0108] Determine the coordinate vector of the camera in the body coordinate system, and use the coordinate vector as the relative spatial position vector of the camera relative to the IMU sensor.

[0109] Optionally, the compensation module is further configured to:

[0110] Determine a first vector parameter corresponding to the relative position change according to the relative position change, and determine a second vector parameter corresponding to the relative angle change according to the relative angle change;

[0111] The first vector parameter and the second vector parameter are dynamically compensated to the calibration pose parameter.

[0112] Optionally, the compensation module is further configured to:

[0113] The relative position change is dynamically detected based on the Hall sensor, and the relative angle change is dynamically detected based on the acceleration sensor.

[0114] Optionally, the compensation module is further configured to:

[0115] Obtaining a calibration compensation value input by a user, and obtaining an actual compensation value for dynamically compensating the calibration attitude parameters;

[0116] Calculating a deviation between the calibration compensation value and the actual compensation value to obtain a compensation error value for dynamically compensating the calibration attitude parameter;

[0117] Training a self-learning model using the calculated compensation error value as a learning sample;

[0118] If the number of the learning samples reaches a preset threshold, a compensation error coefficient for dynamically compensating the calibration pose parameters is obtained based on the prediction of the self-learning model;

[0119] determining a compensation correction coefficient according to the compensation error coefficient;

[0120] Dynamically compensate the calibration pose parameters according to the pose change amount and the compensation correction coefficient.

[0121] The camera posture compensation device provided in an embodiment of the present invention employs the camera posture compensation method of the first or second embodiment above, solving the technical problem of reduced posture positioning accuracy in smart glasses due to changes in the initial wearing posture of the camera. Compared with the prior art, the beneficial effects of the camera posture compensation device provided in an embodiment of the present invention are the same as those of the camera posture compensation method provided in the above embodiment. Other technical features of the camera posture compensation device are the same as those disclosed in the above embodiment and are not further described here.

[0122] Example 4

[0123] An embodiment of the present invention provides an electronic device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the camera posture compensation method in the above-mentioned embodiment one.

[0124] Reference below Figure 5 , which shows a schematic diagram of the structure of an electronic device suitable for implementing the embodiments of the present disclosure. The electronic device in the embodiments of the present disclosure may include, but is not limited to, mixed reality (MR) glasses, augmented reality (AR) glasses, virtual reality (VR) glasses, extended reality (XR) glasses, or some combination thereof, such as smart glasses. Figure 5 The electronic device shown is only an example and should not limit the functions and scope of use of the embodiments of the present disclosure.

[0125] like Figure 5 As shown, the electronic device may include a processing device (such as a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) or a program loaded from a storage device into a random access memory (RAM). In the RAM, various programs and data required for the operation of the electronic device are also stored. The processing device, ROM, and RAM are connected to each other via a bus. An input / output (I / O) interface is also connected to the bus.

[0126] Typically, the following systems can be connected to the I / O interface: input devices such as a touch screen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output devices such as a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices such as a magnetic tape, hard disk, etc.; and communication devices. The communication device can allow the electronic device to communicate with other devices wirelessly or by wire to exchange data. Although the figures show electronic devices with various systems, it should be understood that it is not required to implement or have all of the systems shown. More or fewer systems may be implemented or have instead.

[0127] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via a communication device, or installed from a storage device, or installed from a ROM. When the computer program is executed by a processing device, the above-mentioned functions defined in the method of the embodiment of the present disclosure are performed.

[0128] The electronic device provided by the present invention employs the camera posture compensation method of the first or second embodiment above, solving the technical problem of reduced posture positioning accuracy in smart glasses due to changes in the initial wearing posture of the camera. Compared with the prior art, the beneficial effects of the electronic device provided by the embodiment of the present invention are the same as those of the camera posture compensation method provided by the first embodiment above, and the other technical features of the electronic device are the same as those disclosed in the method of the previous embodiment, and are not further described here.

[0129] It should be understood that various parts of the present disclosure can be implemented with hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in an appropriate manner.

[0130] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present invention should be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims.

[0131] Example 5

[0132] An embodiment of the present invention provides a computer-readable storage medium having computer-readable program instructions stored thereon, and the computer-readable program instructions are used to execute the camera posture compensation method in the above-mentioned embodiment 1.

[0133] The computer-readable storage medium provided in the embodiment of the present invention can be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination thereof. More specific examples of computer-readable storage media can include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this embodiment, the computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium can be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination thereof.

[0134] The computer-readable storage medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0135] The above-mentioned computer-readable storage medium carries one or more programs. When the above-mentioned one or more programs are executed by an electronic device, the electronic device: when determining that the smart glasses are in a wearing state, dynamically detects the relative change in the camera's posture, wherein the relative change in posture is the posture change of the camera's wearing posture parameters relative to the reference posture parameters; and dynamically compensates the reference posture parameters based on the relative change in posture.

[0136] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on the remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0137] The flow charts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, program segment, or a part of code, and the module, program segment, or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0138] The modules involved in the embodiments described in this disclosure may be implemented in software or hardware, wherein the name of a module does not necessarily limit the unit itself.

[0139] The computer-readable storage medium provided by the present invention stores computer-readable program instructions for executing the aforementioned camera pose compensation method, thereby resolving the technical problem of reduced pose positioning accuracy in smart glasses due to changes in the initial camera pose. Compared to the prior art, the beneficial effects of the computer-readable storage medium provided by the embodiments of the present invention are the same as those of the camera pose compensation methods provided by the first or second embodiments above, and are not further elaborated here.

[0140] Example 6

[0141] An embodiment of the present invention further provides a computer program product, comprising a computer program, which, when executed by a processor, implements the steps of the camera posture compensation method as described above.

[0142] The computer program product provided in this application solves the technical problem of reduced position positioning accuracy in smart glasses due to changes in the initial camera position. Compared to the prior art, the beneficial effects of the computer program product provided in the embodiments of the present invention are the same as those of the camera position compensation method provided in the first or second embodiments above, and are not further elaborated here.

[0143] The above are only preferred embodiments of the present application and do not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent processing scope of the present application.

Claims

1. A camera posture compensation method, characterized in that: The camera posture compensation method is applied to smart glasses, which include a glasses frame and cameras distributed on both sides of the glasses frame. The camera posture compensation method includes: When it is determined that the smart glasses are in a wearing state, dynamically detecting the outward deformation of the hinge of the smart glasses by using a displacement photoelectric sensor or a pressure sensor provided on the temple of the smart glasses; If the outward deformation is greater than a preset deformation threshold, the relative change in the camera's posture is dynamically detected, wherein the relative change in posture is the change in the camera's wearing posture parameter relative to the calibration posture parameter, and the wearing posture parameter is the camera's posture parameter after the relative posture drift occurs due to the outward extension of the hinge; Dynamically compensate the calibration pose parameters according to the relative change in pose.

2. The camera posture compensation method according to claim 1, wherein: The smart glasses further include a circuit board provided independently of the glasses frame, and an IMU sensor provided on the circuit board. Before the step of dynamically detecting the outward deformation amount of the hinge of the smart glasses using a displacement photoelectric sensor or a pressure sensor provided on the temple of the smart glasses when determining that the smart glasses are in a worn state comprises: When the smart glasses are not worn, calibrating a relative spatial position vector of the camera relative to the IMU sensor; The relative spatial position vector is used as the calibration pose parameter of the camera.

3. The camera posture compensation method according to claim 2, wherein: The step of calibrating the relative spatial position vector of the camera relative to the IMU sensor includes: The position coordinates of the IMU sensor are used as the origin of the three-axis direction to construct the body coordinate system of the smart glasses; Determine the coordinate vector of the camera in the body coordinate system, and use the coordinate vector as the relative spatial position vector of the camera relative to the IMU sensor.

4. The camera posture compensation method according to claim 1, wherein: The relative change in posture includes the relative change in position and the relative change in angle of the camera. The step of dynamically compensating the calibration posture parameters based on the relative change in posture includes: Determine a first vector parameter corresponding to the relative position change according to the relative position change, and determine a second vector parameter corresponding to the relative angle change according to the relative angle change; The first vector parameter and the second vector parameter are dynamically compensated to the calibration pose parameter.

5. The camera posture compensation method according to claim 4, wherein: The smart glasses include a Hall sensor and an acceleration sensor, and the step of dynamically detecting the relative change in the position of the camera includes: The relative position change is dynamically detected based on the Hall sensor, and the relative angle change is dynamically detected based on the acceleration sensor.

6. The camera posture compensation method according to claim 1, wherein: The camera posture compensation method further includes: Obtaining a calibration compensation value input by a user, and obtaining an actual compensation value for dynamically compensating the calibration attitude parameters; Calculating a deviation between the calibration compensation value and the actual compensation value to obtain a compensation error value for dynamically compensating the calibration attitude parameter; Training a self-learning model using the calculated compensation error value as a learning sample; The step of dynamically compensating the calibration pose parameters according to the relative change in pose further comprises: If the number of the learning samples reaches a preset threshold, a compensation error coefficient for dynamically compensating the calibration pose parameters is obtained based on the prediction of the self-learning model; determining a compensation correction coefficient according to the compensation error coefficient; Dynamically compensate the calibration pose parameters according to the pose change amount and the compensation correction coefficient.

7. A camera posture compensation device, characterized in that: The camera posture compensation device is applied to smart glasses, which include a glasses frame and cameras distributed on both sides of the glasses frame. The camera posture compensation device includes: a detection module, when determining that the smart glasses are in a worn state, dynamically detecting the outward deformation of the hinge of the smart glasses through a displacement-type photoelectric sensor or a pressure sensor provided on the temple of the smart glasses; if the outward deformation is greater than a preset deformation threshold, dynamically detecting a relative change in the camera's posture, wherein the relative change in posture is a change in a wearing posture parameter of the camera relative to a reference posture parameter, and the wearing posture parameter is a posture parameter of the camera after relative posture drift due to the outward extension of the hinge; The compensation module dynamically compensates the calibration pose parameters according to the relative change of the pose.

8. An electronic device, characterized in that: The electronic device comprises: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the steps of the camera pose compensation method described in any one of claims 1 to 6.

9. A readable storage medium, characterized in that: The readable storage medium is a computer-readable storage medium, on which is stored a program for implementing the camera posture compensation method. The program for implementing the camera posture compensation method is executed by a processor to implement the steps of the camera posture compensation method as described in any one of claims 1 to 6.

Images