Pose correction method, device and equipment

Abstract

The invention discloses a pose correction method, device and equipment, and the method comprises the steps: obtaining a video signal comprising a to-be-calibrated object and a first pose, and enabling the first pose to be obtained through an inertial measurement unit disposed in the to-be-calibrated object; calibrating the first pose based on the video signal; or; and correcting a video pose determined based on the video signal based on the first pose.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of electronic technology, and relate to but are not limited to a posture correction method, device and equipment. Background Art

[0002] In the naked-eye 3D experience scenario, the ring, as an interactive controller, can realize functions such as interface control selection, page turning, model manipulation (movement, zooming, rotation), etc. This requires the ring to accurately and stably measure its own motion state, position, posture and other information. At present, smart rings can rely on their own integrated inertial measurement unit (IMU) to achieve posture measurement, in which the accelerometer calculates the acceleration information in the carrier coordinate system and the gyroscope calculates the angular velocity information in the navigation coordinate system. These are relative positioning and posture information, not absolute posture information. After a long time of use, the result will drift, and under the influence of noise, the absolute posture calculated based on the relative result will produce cumulative errors, which will affect the interaction accuracy and experience. In the interactive scenario, there are certain requirements for the position and posture of the ring. However, when the user wears the ring for a long time to perform interactive operations or handle daily things at hand, the initial wearing position of the ring will be changed, and the change in wearing position will also affect the normal interactive function of the ring.

[0003] During gesture interaction, the user's hands will shake unconsciously. This physiological shaking is inevitable. Especially in the scene where the hand position and posture are visualized, the user thinks that he has not moved his hands, but the position of his hands is shaking. This shaking will be recognized as a shaking signal, which will cause the interactive object (the model to be controlled in the naked-eye 3D device) to shake, causing confusion to the user and affecting the interactive experience. Therefore, it is necessary to remove the visual impact caused by physiological shaking during low-speed hand movement.

[0004] How to correct the position of the ring during use to ensure that it can always obtain a stable position that meets the interaction requirements is a problem to be solved. Summary of the invention

[0005] In view of this, embodiments of the present application provide a posture correction method, device and equipment.

[0006] The technical solution of the embodiment of the present application is implemented as follows:

[0007] In a first aspect, an embodiment of the present application provides a posture correction method, comprising:

[0008] Acquire a video signal and a first pose of the object to be calibrated, wherein the first pose is acquired by using an inertial measurement unit disposed in the object to be calibrated;

[0009] Calibrate the first pose based on the video signal;

[0010] or;

[0011] The video pose determined based on the video signal is corrected based on the first pose.

[0012] In a second aspect, an embodiment of the present application provides a posture correction device, comprising:

[0013] An acquisition module, used for acquiring a video signal and a first pose of the object to be calibrated, wherein the first pose is acquired by using an inertial measurement unit arranged in the object to be calibrated;

[0014] A calibration module, for calibrating the first pose based on the video signal;

[0015] The correction module is used to correct the video posture determined based on the video signal based on the first posture.

[0016] In a third aspect, an embodiment of the present application provides an electronic device, including a memory, a processor, and a camera device, wherein the camera device is used to acquire a video signal including an object to be calibrated, the memory stores a computer program that can be run on the processor, and the processor is used to acquire the video signal and a first pose acquired by the camera device, wherein the first pose is acquired using an inertial measurement unit arranged in the object to be calibrated; calibrate the first pose based on the video signal; or; correct the video pose determined based on the video signal based on the first pose.

[0017] In a fourth aspect, an embodiment of the present application provides a storage medium storing executable instructions for implementing the above method when executed by a processor.

[0018] In a fifth aspect, a computer program product includes a computer program or instructions, and when the computer program or instructions are executed by a processor, the steps in the above method are implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 A schematic diagram of an implementation flow of a posture correction method provided in an embodiment of the present application;

[0020] Figure 2A A schematic diagram of a process flow for implementing the first position calibration provided in an embodiment of the present application;

[0021] Figure 2B A schematic diagram of a smart ring with a logo provided in an embodiment of the present application;

[0022] Figure 3A A schematic diagram of wearing a smart ring provided in an embodiment of the present application;

[0023] Figure 3BA schematic diagram of wearing a smart ring provided in an embodiment of the present application;

[0024] Figure 3C A schematic diagram of wearing a smart ring provided in an embodiment of the present application;

[0025] Figure 4 A schematic diagram of an implementation process of correcting video posture provided in an embodiment of the present application;

[0026] Figure 5 A schematic diagram of the structure of a posture correction device provided in an embodiment of the present application;

[0027] Figure 6 A hardware entity schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0028] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the specific technical solution of the embodiments of the present application will be further described in detail below in conjunction with the drawings in the embodiments of the present application. The following embodiments are used to illustrate the present application, but are not used to limit the scope of the present application.

[0029] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0030] In the following description, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0031] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0032] The present application embodiment provides a posture correction method, such as Figure 1 As shown, the method includes:

[0033] Step S110, acquiring a video signal and a first pose of the object to be calibrated, wherein the first pose is acquired by using an inertial measurement unit disposed in the object to be calibrated;

[0034] Here, the video signal may be collected using a camera disposed on an electronic device; for example, an image may be acquired using a camera integrated in a naked-eye 3D device.

[0035] The pose may include information such as the motion state, position, and posture of the object to be calibrated.

[0036] Here, an inertial measurement unit (IMU) is set in the object to be calibrated to measure the first position of the object to be calibrated. The accelerometer of the IMU can be used to calculate the acceleration information in the carrier coordinate system, and the gyroscope can be used to calculate the angular velocity information in the navigation coordinate system.

[0037] The object to be calibrated may be a smart wearable device that needs to be calibrated. For example, the object to be calibrated may be any smart device that needs to be calibrated, such as a smart ring, a smart watch, or smart glasses.

[0038] Step S120, calibrating the first posture based on the video signal;

[0039] In some embodiments, the two-dimensional position coordinates of the device to be calibrated in the video signal may be identified, and the first pose may be calibrated using the two-dimensional position coordinates.

[0040] In some embodiments, when it is determined that a large deviation occurs in the first pose, the first pose may be calibrated using the relative pose between the object to be calibrated and the operating body obtained by the video signal.

[0041] or;

[0042] Step S130: Correct the video posture determined based on the video signal based on the first posture.

[0043] This scene of correcting the video position is used to remove the visual impact caused by physiological shaking during low-speed hand movement.

[0044] During implementation, the first position of the object to be calibrated (wearable sensor) can be used to reversely correct the video position determined by the video signal, so as to filter out the natural shaking of the hand under low-speed movement.

[0045] In an embodiment of the present application, a video signal and a first pose of the object to be calibrated are first obtained, wherein the first pose is obtained by using an inertial measurement unit disposed in the object to be calibrated; then the first pose is calibrated based on the video signal; or, based on the first pose, the video pose determined based on the video signal is corrected. In this way, the first pose can be calibrated based on the video signal based on different application scenarios; or, the video pose determined based on the video signal can be corrected based on the first pose.

[0046] In some embodiments, the above step S120 "calibrating the first posture based on the video signal" is as follows Figure 2A As shown, this can be achieved by following the steps below:

[0047] Step S210, identifying a target image in the video signal to obtain a second posture of the object to be calibrated;

[0048] Here, the captured video signal may be first acquired, and then a plurality of frames of images may be captured from the video signal, and finally a target image in the video signal may be determined.

[0049] In the implementation process, an image recognition method can be used to identify each frame of an image to identify whether the image includes the object to be calibrated. The image including the object to be calibrated is used as the target image. For example, an image in which a user is identified to be wearing a smart ring can be used as the target image.

[0050] In the implementation process, the computer vision (CV) method can be used to detect the spatial pose information of the object to be calibrated, and filter out the data with high confidence as the second pose of the object to be calibrated. Here, computer vision refers to the technical field of using cameras, image sensors and other devices to obtain images or videos, and processing and analyzing these images or videos through computer algorithms and models to achieve the understanding and recognition of objects, scenes and their attributes in the images or videos. CV can be used for image recognition to determine the attributes of objects in the image.

[0051] Step S220: When it is determined that the error between the second posture and the first posture is greater than an error threshold, calibrate the first posture using the second posture.

[0052] In the implementation process, the accelerometer of the IMU can be used to calculate the acceleration information in the carrier coordinate system, and the gyroscope can be used to calculate the angular velocity information in the navigation coordinate system. These are relative positioning and attitude information, not absolute posture information. Over time, the results will drift, and under the influence of noise, the absolute posture calculated based on the relative results will produce cumulative errors, which will affect the interaction accuracy and experience.

[0053] In some embodiments, the posture information may also be the wearing position information of the object to be calibrated. For example, when a user wears a smart ring, the posture information may be the position information, posture information and / or orientation information of the smart ring when worn on the finger. When the user wears the ring for a long time to perform interactive operations or handle daily things at hand, the initial wearing position or posture of the ring may be changed, which may affect the normal interactive function of the ring.

[0054] When it is determined that the error between the first posture and the second posture identified is greater than an error threshold, the first posture may be calibrated using the second posture. Here, the error threshold may be set according to actual needs.

[0055] In the embodiment of the present application, a target image including the object to be calibrated is first obtained; then the target image is identified to obtain the second posture of the object to be calibrated; finally, when it is determined that the error between the second posture and the first posture is greater than the error threshold, the first posture is calibrated using the second posture. In this way, when it is determined that the error between the second posture and the first posture of the object to be calibrated obtained by image recognition is greater than the error threshold, the first posture can be calibrated in time, thereby improving the efficiency and accuracy of calibrating the first posture, and ensuring that a stable first posture that meets the interaction requirements can always be obtained.

[0056] In some embodiments, the above step S210 "identifying the target image in the video signal to obtain the second posture of the object to be calibrated" can be implemented by the following steps:

[0057] Step 211, identifying the target image to obtain three sets of pixel coordinates corresponding to at least three marks set on the object to be calibrated;

[0058] Here, at least three mutually distinguishable marks can be set on the object to be calibrated. For example, multiple marking feature points can be evenly distributed on the appearance of the smart ring. The feature points need to be significant, easy to detect, and different from each other for easy distinction. The mark can be an indicator light with different patterns, different colors or different frequencies. Figure 2B A schematic diagram of a smart ring with a logo provided in an embodiment of the present application, such as Figure 2B As shown, the smart ring is provided with 5 logos with different patterns.

[0059] In the implementation process, the target image can be identified by CV to obtain three sets of pixel coordinates corresponding to at least three identifiers. Here, the pixel coordinates can be the position identifier of each point (or pixel) in the digital image or graphical interface. In a two-dimensional image, each pixel can be represented by a coordinate pair, usually represented as (x, y).

[0060] Step 212: Pair at least three groups of pixel coordinates with at least three groups of three-dimensional coordinates corresponding to at least three pre-stored identifiers to obtain an identifier pair information set;

[0061] Here, the three-dimensional coordinates corresponding to all the markers set on the intelligent controller can be acquired and stored in advance. For example, high-precision three-dimensional model geometric information of a ring can be stored in advance, and each feature point in the model and its three-dimensional information can be recorded as {P a(x, y, z), P b (x, y, z), P c (x, y, z)…}.

[0062] In the implementation process, when multiple (n>3) feature points (markers) can be accurately detected, the pixel coordinate information of the feature points detected at this time is recorded, that is, three or more groups of pixel coordinates can be marked as {C a (u,v),C b (u,v),C c (u, v)…}.

[0063] In the implementation process, the pixel coordinates and three-dimensional coordinates corresponding to the same identifier can be paired to obtain an identifier pair set. The identifier pair information set can be recorded as D:{{C a (u,v)~P a (x, y, z)}, {C b (u,v)~P b (x, y, z)}, {C c (u,v)~P c (x, y, z)}, …}.

[0064] Step 213: Fit the second pose using the identification pair information set and camera parameters, wherein the camera parameters are parameters of a camera used to acquire the target image.

[0065] Here, the camera integrated in the electronic device can be precisely calibrated to determine the camera parameters, which include the camera intrinsic parameters M intrinsic , distortion parameter M distortion and camera extrinsic parameter M extrinsic wait.

[0066] In the implementation process, the information set D can be used by using the identifier: a (u,v)~P a (x, y, z)}, {C b (u,v)~P b (x, y, z)}, {C c (u,v)~P c (x, y, z)}, …}, camera intrinsic parameter M intrinsic , distortion parameter M distortion and external reference M extrinsic The second pose of the intelligent control device is obtained by fitting.

[0067] In the embodiment of the present application, the target image is first identified to obtain three sets of pixel coordinates corresponding to at least three markers set on the object to be calibrated; then the at least three sets of pixel coordinates are paired with at least three sets of three-dimensional coordinates corresponding to at least three pre-stored markers to obtain a marker pair information set; finally, the marker pair information set and camera parameters are used to fit the second posture. In this way, it is possible to obtain a highly accurate and effective second posture by fitting the pixel coordinates corresponding to the markers set on the object to be calibrated in real time and the pre-stored high-precision three-dimensional coordinates.

[0068] In some embodiments, the above step 223 "fitting the second pose using the identification pair information set and camera parameters" can be implemented by the following process:

[0069] The second pose is fitted based on the identification pair information set and the camera parameters using an N-point perspective pose solving algorithm.

[0070] Here, the Perspective-n-Point (PnP) algorithm can estimate the pose (including position and direction) through the correspondence between multiple known three-dimensional space points in the image and their two-dimensional projection points on the image.

[0071] During the implementation process, when the camera intrinsic parameters (focal length, optical center, etc.) and distortion coefficients are known, the position and posture (rotation matrix R and translation vector t) of the object to be calibrated in the world coordinate system can be solved through the correspondence between N two-dimensional points on the image and the corresponding N three-dimensional points in the world coordinate system.

[0072] An algorithm (such as an N-point perspective pose solving algorithm) is used to fit the pre-stored three-dimensional model information of the object to be calibrated, and the spatial pose information (second pose) of the object to be calibrated is solved as follows:

[0073] (position, orientation) cv =F(D,M intrinsic ,M distortion ,M extrinsic ) (1);

[0074] Among them, position is the position information, orientation is the direction information, D is the set of identification pair information, and M intrinsic Camera intrinsic parameter, M distortion is the distortion parameter, M extrinsic is the camera extrinsic parameter.

[0075] In an embodiment of the present application, an N-point perspective pose solving algorithm can be used to obtain a second pose for calibrating the pose based on a set of identification pair information and camera parameter fitting.

[0076] In some embodiments, the above step S210 “identifying the target image in the video signal to obtain the second posture of the object to be calibrated” further includes the following steps:

[0077] Step 214: determining the confidence of the second pose information based on at least one of the following: the number of the identification points, the detection accuracy of the identification points, and the rationality of the postures of other objects in the target image;

[0078] Here, the number threshold of identification points can be set according to actual needs to determine whether the number of identifiable identification points is greater than the number threshold of identification points. For example, the number threshold can be set to 3 to determine whether the identified identification points are greater than or equal to 3.

[0079] Detection accuracy can be measured by a series of indicators, such as precision, recall, F1 score, and intersection over union (IoU). Different indicator thresholds are set for each identification point according to actual needs to determine whether each indicator is greater than the indicator threshold.

[0080] According to actual needs, the rationality of the postures of other objects in the target image can be identified. For example, whether the posture of a hand wearing a smart ring is reasonable can be identified, and the rationality of the postures of other objects can also be identified.

[0081] During implementation, the confidence of the second posture information may be determined based on at least one of the above indicators, or by combining different indicators.

[0082] Step 215: Determine whether the confidence level is greater than a confidence level threshold.

[0083] During implementation, a confidence threshold (T_confidence) may be set based on actual needs. When it is determined that the confidence is greater than the confidence threshold, it may be determined that the second posture is valid and available. When it is determined that the confidence is less than the confidence threshold, it may be determined that the second posture is unavailable.

[0084] When the confidence is greater than the confidence threshold, the second position (position, orientation) of the object to be calibrated is calculated. cv With the first position (position, orientation) IMU When the error E is greater than the error threshold (T_precision), the second pose is used to realign and calibrate the first pose.

[0085] In the embodiment of the present application, the confidence of the second posture information is first determined based on at least one of the following: the number of the identification points, the detection accuracy of the identification points, and the rationality of the postures of other objects in the target image; and then it is determined that the confidence is greater than the confidence threshold. In this way, the second posture can be used to correct the first posture when it is determined that the second posture is true and valid.

[0086] In some embodiments, the posture correction method further includes the following steps:

[0087] Step S230, identifying the target image in the video signal, and acquiring a third posture of the operating body wearing the object to be calibrated;

[0088] Here, the operating body wearing the object to be calibrated can be a finger wearing a smart ring, a wrist wearing a smart bracelet, or a head wearing smart glasses.

[0089] During implementation, the target image can be recognized through CV to obtain the third posture of the operating body.

[0090] Figure 3A A schematic diagram of wearing a smart ring provided in an embodiment of the present application, such as Figure 3A As shown, the schematic diagram includes a smart ring 31 and a hand 32 wearing the smart ring. When the smart ring is correctly configured, the ring logo can be facing upward relative to the palm, and the smart ring is worn at a relatively fixed position and orientation on the finger.

[0091] Step S240, determining a current relative posture between the third posture and the second posture;

[0092] Here, the relative posture between the third posture and the second posture may be a relative posture generated by the user wearing the smart device for a long time and rotating or moving the smart device.

[0093] Figure 3B A schematic diagram of wearing a smart ring provided in an embodiment of the present application, such as Figure 3B As shown, the schematic diagram includes a smart ring 31 and a hand 32 wearing the smart ring. When the user operates for a long time, there is a situation where the smart ring is rotated to change the direction of the mark on the smart ring. Figure 3B The smart ring 31 shown is rotated by a certain angle compared to the smart ring 31 shown in FIG. 3 .

[0094] Figure 3C A schematic diagram of wearing a smart ring provided in an embodiment of the present application, such as Figure 3C As shown, the schematic diagram includes a smart ring 31 and a hand 32 wearing the smart ring. When the user operates for a long time, there is a situation where the smart ring is turned and the posture of the smart ring is changed. Figure 3CThe wearing angle of the smart ring 31 shown in FIG. 3 is changed compared to the smart ring 31 shown in FIG. 3 .

[0095] Correspondingly, in the above step S130, "calibrating the first posture using the second posture" can be achieved by the following process:

[0096] When it is determined that a relative posture difference between the current relative posture and the target relative posture is greater than a first relative posture difference threshold, the first posture is corrected using the relative posture difference, wherein the target relative posture is a preset relative posture.

[0097] Here, the preset relative posture can be the relative posture of the object to be calibrated relative to the operating body when the object to be calibrated is correctly worn by the user. For example, as shown in FIG3 , it is the relative posture between the smart ring 31 and the hand 32 wearing the smart ring when the user wears the smart ring 31 correctly.

[0098] For example, during the interaction, the change δ between the relative relationship between the hand posture and the smart ring posture is determined. When δ is greater than the set first relative posture difference threshold, it can be determined that the current ring wearing position has changed. At this time, the ring posture correction program is started, and the first posture is corrected using the relative posture difference. All subsequent signals given by the ring are corrected signals, ensuring the correctness of the ring signal.

[0099] In some embodiments, Figure 3B As shown, the angle difference can be used to compensate for the rotation angle of the smart ring and correct the posture data of the smart ring.

[0100] In the embodiment of the present application, the target image is first identified to obtain the third posture of the operating body wearing the object to be calibrated; then the current relative posture between the third posture and the second posture is determined; finally, when it is determined that the relative posture difference between the current relative posture and the target relative posture is greater than the first relative posture difference threshold, the first posture is corrected using the relative posture difference. In this way, it is possible to judge whether the wearing position of the current object to be calibrated has changed. At this time, the posture correction program is started, and the first posture is corrected using the relative posture difference. All subsequent signals are corrected signals, which ensures the correctness of the signal of the object to be calibrated. Since the correction of the posture of the object to be calibrated after the position of the object to be calibrated changes is a user-insensitive process, the correctness of the signal of the object to be calibrated is guaranteed without affecting the user's interactive experience. The performance of long-term interaction when wearing the object to be calibrated is improved, and the user's interactive behavior is maximized and protected.

[0101] In some embodiments, the above posture correction method further includes the following steps:

[0102] Step S250: When it is determined that the relative posture difference is greater than a second relative posture difference threshold, output a prompt message to prompt the user to adjust the wearing of the object to be calibrated, wherein the second phase posture difference threshold is greater than the first relative difference threshold.

[0103] During the implementation process, when the δ value is greater than the alarm threshold (the second relative displacement difference threshold), the interactive interface reminds the user to adjust the position of the object to be calibrated by the user interface (UI) design, and records the user's current interaction behavior at the same time. After the user adjusts the position of the object to be calibrated, the interaction process continues.

[0104] Here, the second phase-pose difference threshold is set to be greater than the first relative difference threshold. That is, when the posture cannot be corrected by the correction program due to the large difference in phase-pose, the user can be prompted to adjust the configuration of the object to be calibrated.

[0105] In the embodiment of the present application, when it is determined that the relative posture difference is greater than the second relative posture difference threshold, a prompt message is output to prompt the user to adjust the wearing of the object to be calibrated. In this way, when the position of the object to be calibrated changes significantly and triggers the alarm threshold, the current interaction behavior is recorded, and the previous interaction behavior can be continued after the posture of the object to be calibrated is adjusted. The performance of long-term interaction while wearing the object to be calibrated is improved, and the user's interaction behavior is maximized and protected.

[0106] In some embodiments, before step S230 of "identifying the target image in the video signal and acquiring the third posture of the operating body wearing the object to be calibrated", the posture correction method further includes the following steps:

[0107] Step S260, obtaining the relative position and posture of the target using a gesture posture estimation algorithm;

[0108] Here, the CV hand gesture estimation algorithm can be used to record the target relative position between the hand gesture and the smart ring posture when the user wears the smart ring correctly.

[0109] Step S270: store the relative position and posture of the target.

[0110] During implementation, the target relative position and posture is stored to determine whether the position and posture of the object to be calibrated needs to be corrected when the user subsequently uses the object to be calibrated for a long time.

[0111] In the embodiment of the present application, the relative position and posture of the target is first obtained by using a gesture posture estimation algorithm, and then the relative position and posture of the target is stored. In this way, when the user subsequently uses the object to be calibrated for a long time, the stored relative position and posture of the target can be used to determine whether the position and posture of the object to be calibrated needs to be corrected.

[0112] In some embodiments, the above step S130 "correcting the video posture determined based on the video signal based on the first posture" is as follows: Figure 4 As shown, this can be achieved by following the steps below:

[0113] Step S410: determining, based on the first posture, that a motion state of an operating body worn by the object to be calibrated is a target motion state, wherein a motion speed of the operating body in the target motion state is less than a speed threshold;

[0114] During the implementation process, it can be determined whether the operating body is in a low-speed motion state based on the first posture (IMU data), because normal hand tremors will affect the visual experience only when the user believes that there is no motion.

[0115] For example, the motion state of the finger wearing the smart ring can be determined according to the first posture, and when the motion speed of the finger is less than a speed threshold, the finger is determined to be in a low-speed motion state. Here, the motion speed may include linear motion speed, linear acceleration, angular velocity, angular acceleration, and other motion speeds.

[0116] Step S420: determining a time difference between the first posture and the video posture corresponding to the same time point based on motion parameters of the first posture and the video posture;

[0117] Here, since the IMU data delay is usually lower than that of the video pose (CV data), about milliseconds, that is, the electronic device obtains the IMU data first and then the CV data. Therefore, the dynamic time difference Δt between the IMU data and the VC data can be calculated based on the movement direction of the acquired IMU data sequence and CV data sequence.

[0118] Step S430: determining a compensation motion posture based on the first posture within a time window, wherein the time window is determined based on the time difference and a preset duration;

[0119] During implementation, the compensation motion posture ΔT can be calculated using the following formula (1):

[0120] ΔT = f(traj_IMU,Δt) (1);

[0121] Among them, traj_IMU is the IMU data in the time window.

[0122] Step S440: modify the video pose using the compensated motion pose.

[0123] In the implementation process, the hand posture P_adjust can be corrected using the following formula (2):

[0124] P_adjust = P_cv + ΔT (2);

[0125] Among them, P_cv is the original hand estimated pose (video pose) output by the CV signal.

[0126] In an embodiment of the present application, firstly, the motion state of the operating body wearing the object to be calibrated is determined to be the target motion state based on the first posture, wherein the motion speed of the operating body in the target motion state is less than the speed threshold; then, based on the motion parameters of the first posture and the video posture, the time difference corresponding to the same time point of the first posture and the video posture is determined; within the time window, the compensation motion posture is determined based on the first posture, wherein the time window is determined based on the time difference and the preset duration; finally, the video posture is modified using the compensation motion posture. In this way, the video posture estimated by the CV signal is reversely corrected using the first posture of the wearable sensor, so as to filter out the natural jitter of the hand under low-speed movement.

[0127] In some embodiments, the above step S110 "obtaining a video signal including the object to be calibrated and the first pose" can be implemented by the following steps:

[0128] Step S111, obtaining the calibration period of the object to be calibrated;

[0129] Here, the calibration period can be set according to actual needs. For example, 10 seconds can be set as the calibration period.

[0130] Step S112, acquiring a video signal and a first pose of the object to be calibrated within the calibration period;

[0131] In some embodiments, during the interaction process, the change δ of the relative relationship between the posture of the operating body and the posture of the object to be calibrated is determined at intervals of a certain time threshold (for example, 10 seconds), so that an image including the object to be calibrated can be acquired.

[0132] In some embodiments, when the user interacts with the object to be calibrated, when the object to be calibrated is less obstructed and the time ΔT that has passed since the last calibration moment is greater than the time threshold (T_time), IMU posture correction is started to obtain an image including the object to be calibrated.

[0133] Step S113: Determine an image of the object to be calibrated with the highest integrity in the video signal including the object to be calibrated as a target image.

[0134] Here, among all images including the object to be calibrated, the object to be calibrated included in each image can be identified using CV, and an image with the highest degree of integrity without occlusion and deformation can be determined as the target image. For example, in the acquired image, there may be a situation where the smart ring is blocked by another hand or an obstacle, and the image with the occlusion is filtered out to obtain the target image.

[0135] In the embodiment of the present application, the calibration period of the object to be calibrated is first obtained; then at least one image including the object to be calibrated is obtained within the calibration period; finally, the image of the object to be calibrated with the highest integrity among the at least one image including the object to be calibrated is determined as the target image. In this way, the correction program can be automatically started periodically to correct inaccurate postures in a timely manner. And the image of the object to be calibrated with the highest integrity obtained can effectively improve the reliability and accuracy of the calibration.

[0136] Based on the foregoing embodiments, an embodiment of the present application provides a posture correction device, which includes the modules included, each module includes sub-modules, each sub-module includes a unit, which can be implemented by a processor in an electronic device; of course, it can also be implemented by a specific logic circuit; in the implementation process, the processor can be a central processing unit (CPU), a microprocessor (MPU), a digital signal processor (DSP) or a field programmable gate array (FPGA), etc.

[0137] Figure 5 A schematic diagram of the structure of the posture correction device provided in the embodiment of the present application is shown in FIG. Figure 5 As shown, the device 500 includes:

[0138] An acquisition module 510 is used to acquire a video signal and a first pose of the object to be calibrated, wherein the first pose is acquired by using an inertial measurement unit arranged in the object to be calibrated;

[0139] A calibration module 520, configured to calibrate the first posture based on the video signal; or

[0140] The correction module 530 is used to correct the video posture determined based on the video signal based on the first posture.

[0141] In some embodiments, the calibration module 520 includes a first recognition submodule and a calibration submodule, wherein the first recognition submodule is used to recognize the target image in the video signal to obtain the second posture of the object to be calibrated; the calibration submodule is used to determine that the error between the second posture and the first posture is greater than the error threshold, and the first posture is calibrated using the second posture.

[0142] In some embodiments, the first recognition submodule includes a first recognition unit, a pairing unit and a fitting unit, wherein the first recognition unit is used to recognize the target image in the video signal to obtain three groups of pixel coordinates corresponding to at least three identifiers set on the object to be calibrated; the pairing unit is used to pair at least three groups of pixel coordinates with at least three groups of three-dimensional coordinates corresponding to at least three pre-stored identifiers to obtain an identifier pair information set; the fitting unit is used to fit the second pose using the identifier pair information set and camera parameters, wherein the camera parameters are parameters of a camera used to obtain the target image.

[0143] In some embodiments, the fitting unit is further used to fit the second pose based on the identification pair information set and the camera parameters using an N-point perspective pose solving algorithm.

[0144] In some embodiments, the first recognition submodule also includes a first determination unit and a second determination unit, wherein the first determination unit is used to determine the confidence of the second posture information based on at least one of: the number of the identification points, the detection accuracy of the identification points, and the rationality of the postures of other objects in the target image; and the second determination unit is used to determine that the confidence is greater than a confidence threshold.

[0145] In some embodiments, the calibration module 520 also includes a second identification submodule and a first determination submodule, wherein the second identification submodule is used to identify the target image in the video signal and obtain the third posture of the operating body wearing the object to be calibrated; the first determination submodule is used to determine the current relative posture between the third posture and the second posture; correspondingly, the calibration submodule is also used to correct the first posture using the relative posture difference when it is determined that the relative posture difference between the current relative posture and the target relative posture is greater than a first relative posture difference threshold, wherein the target relative posture is a preset relative posture.

[0146] In some embodiments, the calibration module 520 also includes an information output submodule for outputting a prompt message to prompt the user to adjust the wearing of the object to be calibrated when it is determined that the relative posture difference is greater than a second relative posture difference threshold, wherein the second phase posture difference threshold is greater than the first relative difference threshold.

[0147] In some embodiments, the calibration module 520 further includes a first acquisition submodule and a storage submodule, wherein the first acquisition submodule is used to acquire the target relative posture using a gesture posture estimation algorithm; and the storage submodule is used to store the target relative posture.

[0148] In some embodiments, the correction module 530 includes a second determination submodule, a third determination submodule, a fourth determination submodule and a correction submodule, wherein the second determination submodule is used to determine, based on the first posture, that the motion state of the operating body worn by the object to be calibrated is a target motion state, wherein the motion speed of the operating body in the target motion state is less than a speed threshold; the third determination submodule is used to determine, based on the motion parameters of the first posture and the video posture, the time difference between the first posture and the video posture corresponding to the same time point; the fourth determination submodule is used to determine the compensatory motion posture based on the first posture within a time window, wherein the time window is determined based on the time difference and a preset duration; the correction submodule is used to modify the video posture using the compensatory motion posture.

[0149] The description of the above device embodiment is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. For technical details not disclosed in the device embodiment of the present application, please refer to the description of the method embodiment of the present application for understanding.

[0150] It should be noted that in the embodiment of the present application, if the above method is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable an electronic device (which can be a mobile phone, a tablet computer, a laptop computer, a desktop computer, etc.) to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a U disk, a mobile hard disk, a read-only memory (ROM), a disk or an optical disk. In this way, the embodiment of the present application is not limited to any specific combination of hardware and software.

[0151] Correspondingly, an embodiment of the present application provides a storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps in the posture correction method provided in the above embodiment are implemented.

[0152] Correspondingly, an embodiment of the present application provides an electronic device, Figure 6 A hardware entity diagram of an electronic device provided in an embodiment of the present application, such as Figure 6 As shown, the hardware entity of the device 600 includes: a memory 601 and a processor 602, wherein the memory 601 stores a computer program that can be run on the processor 602, and when the processor 602 executes the program, the steps in the posture correction method provided in the above embodiment are implemented.

[0153] The memory 601 is configured to store instructions and applications executable by the processor 602, and can also cache data to be processed or processed by the processor 602 and various modules in the electronic device 600 (for example, image data, audio data, voice communication data, and video communication data), which can be implemented through flash memory (FLASH) or random access memory (Random Access Memory, RAM).

[0154] It should be noted here that the description of the above storage medium and device embodiments is similar to the description of the above method embodiments, and has similar beneficial effects as the method embodiments. For technical details not disclosed in the storage medium and device embodiments of this application, please refer to the description of the method embodiments of this application for understanding.

[0155] It should be understood that "one embodiment" or "an embodiment" mentioned throughout the specification means that specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that in various embodiments of the present application, the size of the sequence number of the above-mentioned processes does not mean the order of execution, and the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application. The above-mentioned sequence numbers of the embodiments of the present application are only for description and do not represent the advantages and disadvantages of the embodiments.

[0156] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0157] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0158] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units; they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0159] In addition, all functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may be a separate unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0160] A person skilled in the art can understand that all or part of the steps of implementing the above method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiment; and the aforementioned storage medium includes: mobile storage devices, read-only memories (ROM), magnetic disks or optical disks, etc., various media that can store program codes.

[0161] Alternatively, if the above-mentioned integrated unit of the present application is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application can be essentially or partly reflected in the form of a software product that contributes to the relevant technology. The computer software product is stored in a storage medium, including several instructions to enable an electronic device (which can be a mobile phone, a tablet computer, a laptop computer, a desktop computer, etc.) to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0162] The methods disclosed in several method embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments.

[0163] The features disclosed in several product embodiments provided in this application can be arbitrarily combined without conflict to obtain new product embodiments.

[0164] The features disclosed in several method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments or device embodiments.

[0165] The above is only an implementation method of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A posture correction method, the method include: Acquire a video signal and a first pose of the object to be calibrated, wherein the first pose is acquired by using an inertial measurement unit disposed in the object to be calibrated; calibrating the first pose based on the video signal; or; A video pose determined based on the video signal is corrected based on the first pose.

2. The method of claim 1, wherein the first pose is calibrated based on the video signal. include: Identifying a target image in the video signal to obtain a second posture of the object to be calibrated; When it is determined that the error between the second posture and the first posture is greater than an error threshold, the first posture is calibrated using the second posture.

3. The method according to claim 2, wherein the target image in the video signal is identified to obtain a second posture of the object to be calibrated, include: Identify the target image in the video signal to obtain three sets of pixel coordinates corresponding to at least three marks set on the object to be calibrated; Pairing at least three groups of pixel coordinates with at least three groups of three-dimensional coordinates corresponding to at least three pre-stored identifiers to obtain an identifier pair information set; The second pose is fitted using the identification pair information set and camera parameters, wherein the camera parameters are parameters of a camera used to acquire the target image.

4. The method of claim 3, wherein the second pose is fitted using the identification pair information set and camera parameters, include: The second pose is fitted based on the identification pair information set and the camera parameters using an N-point perspective pose solving algorithm.

5. The method according to claim 3, further comprising: include: Determining the confidence of the second pose information based on at least one of the following: the number of the identification points, the detection accuracy of the identification points, and the rationality of the postures of other objects in the target image; It is determined that the confidence level is greater than a confidence level threshold.

6. The method according to claim 2, further comprising: include: Identify the target image in the video signal, and obtain a third posture of the operating body wearing the object to be calibrated; determining a current relative pose between the third pose and the second pose; Correspondingly, the calibrating the first posture by using the second posture includes: When it is determined that a relative posture difference between the current relative posture and the target relative posture is greater than a first relative posture difference threshold, the first posture is corrected using the relative posture difference, wherein the target relative posture is a preset relative posture.

7. The method according to claim 6, further comprising: include: When it is determined that the relative posture difference is greater than a second relative posture difference threshold, a prompt message is output to prompt the user to adjust the wearing of the object to be calibrated, wherein the second relative posture difference threshold is greater than the first relative difference threshold.

8. The method according to claim 6, before identifying the target image in the video signal and obtaining the third posture of the operating body wearing the object to be calibrated, the method further include: Obtaining the relative position and posture of the target using a hand gesture estimation algorithm; The target relative pose is stored.

9. The method of claim 1, wherein the video pose determined based on the video signal is corrected based on the first pose. include: Determining, based on the first posture, that a motion state of an operating body worn by the object to be calibrated is a target motion state, wherein a motion speed of the operating body in the target motion state is less than a speed threshold; Determine, based on motion parameters of the first posture and the video posture, a time difference between the first posture and the video posture corresponding to the same time point; Determine a compensation motion posture based on the first posture within a time window, wherein the time window is determined based on the time difference and a preset duration; The video pose is modified using the compensated motion pose.

10. An electronic device, comprising a memory, a processor and a camera device, wherein the camera setting device is used to obtain a video signal including an object to be calibrated, the memory stores a computer program that can be run on the processor, and the processor is used to obtain the video signal and the first pose obtained by the camera device, in, The first posture is acquired by using an inertial measurement unit arranged in the object to be calibrated; the first posture is calibrated based on the video signal; or; the video posture determined based on the video signal is corrected based on the first posture.

Images