Method, device and electronic equipment for processing posture data

Abstract

The present application discloses a posture data processing method, device and electronic device, the method comprising: obtaining a first image, the first image being a two-dimensional image captured by an image acquisition device for a target object; obtaining a first posture of the target object relative to the image acquisition device at least based on the first image; processing the first image by means of synchronous positioning and mapping SLAM to obtain a second posture of the image acquisition device relative to a target coordinate system; and obtaining a target posture of the target object relative to the target coordinate system based on the first posture and the second posture.

Description

Technical Field

[0001] The present application relates to the field of positioning technology, and in particular to a method, device and electronic equipment for processing posture data. Background Art

[0002] In some application scenarios, it is necessary to locate the objects and estimate their position in three-dimensional space, which includes attitude, rotation and translation in six degrees of freedom.

[0003] Currently, SLAM-based technical solutions are mostly used to track objects in real time to reduce the amount of calculations generated by multiple posture recognitions. Summary of the invention

[0004] In view of this, the present application provides a posture data processing method, device and electronic device, including:

[0005] A method for processing posture data, comprising:

[0006] Obtaining a first image, where the first image is a two-dimensional image acquired by an image acquisition device for a target object;

[0007] At least based on the first image, obtaining a first pose of the target object relative to the image acquisition device;

[0008] Processing the first image by means of simultaneous localization and mapping (SLAM) to obtain a second pose of the image acquisition device relative to a target coordinate system;

[0009] A target pose of the target object relative to the target coordinate system is obtained according to the first pose and the second pose.

[0010] The above method preferably obtains the first pose of the target object relative to the image acquisition device at least based on the first image, including:

[0011] Using the first image, processing the third posture to obtain a first posture of the target object relative to the image acquisition device;

[0012] The third posture is obtained based on a second image, and the second image is a two-dimensional image captured by the image acquisition device for the target object at a moment before the first image is captured.

[0013] In the above method, preferably, the third posture is obtained by:

[0014] Obtaining two-dimensional feature points of the target object according to the second image;

[0015] Matching the three-dimensional feature points of the target object with the two-dimensional feature points to obtain a matching relationship between the three-dimensional feature points and the two-dimensional feature points; the three-dimensional feature points of the target object are obtained according to the three-dimensional scanning information of the target object;

[0016] A third posture of the target object relative to the image acquisition device is obtained based at least on the matching relationship and acquisition parameters of the image acquisition device.

[0017] The above method preferably uses the first image to process the third posture to obtain the first posture of the target object relative to the image acquisition device, including:

[0018] Obtaining two-dimensional feature points in the first image;

[0019] According to the matching relationship, obtaining three-dimensional feature points corresponding to the two-dimensional feature points in the first image;

[0020] The third posture data is adjusted according to the three-dimensional feature points corresponding to the two-dimensional feature points in the first image to obtain the first posture of the target object relative to the image acquisition device.

[0021] Preferably, in the above method, after obtaining the target position and posture of the target object relative to the target coordinate system, the method further comprises:

[0022] The target posture is processed using a Kalman filter to obtain a new target posture.

[0023] In the above method, preferably, the Kalman filter processes the target posture at least according to the third posture.

[0024] The above method, preferably, further comprises:

[0025] The target posture is used to update the filter parameters of the Kalman filter, and the updated filter parameters are used to process a fourth posture, where the fourth posture is the posture of the target object relative to the target coordinate system at the next moment.

[0026] In the above method, preferably, obtaining the target posture of the target object relative to the target coordinate system according to the first posture and the second posture includes:

[0027] The first posture and the second posture are vector-multiplied to obtain a target posture of the target object relative to the target coordinate system.

[0028] A posture data processing device, comprising:

[0029] An image acquisition unit, configured to acquire a first image, wherein the first image is a two-dimensional image acquired by an image acquisition device for a target object;

[0030] A first obtaining unit, configured to obtain a first posture of the target object relative to the image acquisition device at least based on the first image;

[0031] A second obtaining unit, configured to process the first image by means of SLAM to obtain a second posture of the image acquisition device relative to a target coordinate system;

[0032] A target acquisition unit is used to obtain a target posture of the target object relative to the target coordinate system according to the first posture and the second posture.

[0033] An electronic device, comprising:

[0034] An image acquisition device, used for acquiring images;

[0035] A processor is used to obtain a first image, which is a two-dimensional image captured by the image acquisition device for a target object; obtain a first pose of the target object relative to the image acquisition device based on at least the first image; process the first image by means of synchronous localization and mapping (SLAM) to obtain a second pose of the image acquisition device relative to a target coordinate system; and obtain a target pose of the target object relative to the target coordinate system based on the first pose and the second pose.

[0036] It can be seen from the above technical solutions that in a posture data processing method, device and electronic device disclosed in the present application, after obtaining the two-dimensional image captured by the image acquisition device for the target object, the first posture of the target object relative to the image acquisition device and the second posture of the image acquisition device relative to the target coordinate system obtained by SLAM are obtained using the two-dimensional image, and then the target posture of the target object relative to the target coordinate system is obtained based on these two postures. It can be seen that the present application integrates the posture obtained based on the two-dimensional image and the posture obtained based on the SLAM method, and can obtain the target posture of the target object in three-dimensional space regardless of whether the target object is in motion, thereby avoiding the situation where the posture deviation is caused by the movement of the object when the posture of the object is acquired based on the SLAM method alone, thereby improving the accuracy of the posture. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0038] Figure 1 A flowchart of a method for processing posture data provided in Example 1 of the present application;

[0039] Figure 2 This is an example diagram of an AR application scenario applicable to this application;

[0040] Figure 3 A partial flow chart of a method for processing posture data provided in Example 1 of the present application;

[0041] Figure 4 This is an example diagram of an implementation method of the embodiment of the present application;

[0042] Figure 5 and Figure 6 They are respectively another partial flow chart of a method for processing posture data provided in Example 1 of the present application;

[0043] Figure 7 and Figure 8 They are respectively another flow chart of a posture data processing method provided in the first embodiment of the present application;

[0044] Fig. 9 A schematic diagram of the structure of a posture data processing device provided in Embodiment 2 of the present application;

[0045] Fig.10 and Fig.11 They are respectively another structural schematic diagram of a posture data processing device provided in the second embodiment of the present application;

[0046] Fig.12 A schematic diagram of the structure of an electronic device provided in Example 3 of the present application. DETAILED DESCRIPTION

[0047] With the development of technology, the demand for object position positioning is also increasing. In many application scenarios, it is necessary to implement corresponding scene functions based on the position of the object. For example, in augmented reality AR (Augmented Reality) glasses, it is necessary to obtain the position of game players in three-dimensional space; in infant monitoring videos, it is necessary to obtain the position of infants, and so on.

[0048] Taking the application scenario of AR as an example, it is necessary to identify three-dimensional objects in the scene, such as game players or small animals, obtain their position in the three-dimensional space, and then superimpose various virtual information such as models, text, videos, etc. to achieve the display effect of augmented real world.

[0049] The inventor of the present application found in the research of tracking algorithm that: since the computational amount of identifying the position and posture of three-dimensional objects is relatively large, and the AR application scenario needs to obtain the position and posture of three-dimensional objects in real time, the current common practice is to use a real-time tracking algorithm to track the three-dimensional objects after identifying the three-dimensional objects. Among the tracking algorithms, there are two commonly used ones: the first tracking scheme is to use a pure visual three-dimensional object tracking algorithm, and the second tracking scheme is based on SLAM for real-time tracking. Although the first tracking scheme can track the three-dimensional objects in the field of view in real time, because it only relies on visual information, when the camera moves or rotates quickly, the image is prone to blur. At this time, the tracking algorithm is very likely to fail, resulting in a poor AR experience. Moreover, after the three-dimensional object moves out of the field of view and enters the field of view again, it also needs to re-recognize the position and posture, which will cause recognition delay. The second tracking scheme is based on visual and inertial measurement unit IMU (Inertial Measurement Unit) information to track the position and posture of the camera in the real world, which can solve the shortcomings of the pure visual tracking scheme, but it is only effective for stationary objects. Once the three-dimensional object is displaced in the three-dimensional space, the tracking scheme cannot be perceived, resulting in deviations when the virtual information is superimposed later.

[0050] Based on the above defects, the inventors of this application proposed a real-time tracking solution for three-dimensional objects that integrates visual information and SLAM, so as to avoid the posture deviation caused by the movement of three-dimensional objects when acquiring the posture of three-dimensional objects based on SLAM, thereby improving the accuracy of the posture.

[0051] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0052] refer to Figure 1 , is a flowchart of a method for processing posture data provided in the first embodiment of the present application, which can be applied to electronic devices with image acquisition devices, such as AR glasses, etc. The technical solution in this embodiment is mainly used to improve the accuracy of posture.

[0053] Specifically, the method in this embodiment may include the following steps:

[0054] Step 101: Obtain a first image.

[0055] The first image is a two-dimensional image captured by the image acquisition device for the target object.

[0056] For example, AR glasses use cameras to collect two-dimensional images of other game players' fields of view, such as Figure 2 As shown in , the camera can be a built-in camera in the AR glasses or an external camera set on the AR glasses housing.

[0057] Step 102: Obtain a first pose of the target object relative to the image acquisition device at least based on the first image.

[0058] In this embodiment, the first pose can be obtained by performing posture recognition on the three-dimensional object based on the first image, or, in this embodiment, the first pose can be obtained by performing posture tracking on the three-dimensional object based on the first image.

[0059] For example, the AR glasses perform posture recognition on the first image captured by the camera to obtain the first posture of the game player in the field of view relative to the AR glasses; or, the AR glasses use the first image captured by the camera to perform posture tracking based on the historical posture of the game player to obtain the first posture of the game player in the field of view relative to the AR glasses.

[0060] Step 103: Process the first image by means of SLAM to obtain a second posture of the image acquisition device relative to the target coordinate system.

[0061] The target coordinate system here can be a world coordinate system, which is a coordinate system in three-dimensional space. Specifically, in this embodiment, a SLAM algorithm can be used in combination with data collected by the IMU to perform image recognition on the first image to obtain a second posture of the image acquisition device relative to the target coordinate system. The IMU is set in the electronic device.

[0062] For example, the AR glasses use a SLAM algorithm to perform posture recognition on a first image captured by a camera to obtain a second posture of the AR glasses relative to a world coordinate system.

[0063] It should be noted that the execution order between step 102 and step 103 is not limited to the execution order shown in the accompanying drawings. Step 103 can be executed first and then step 102. Alternatively, step 102 and step 103 can be executed simultaneously. The different technical solutions formed belong to the same inventive concept and are all within the protection scope of this application.

[0064] Step 104: Obtain a target pose of the target object relative to the target coordinate system according to the first pose and the second pose.

[0065] Specifically, in this embodiment, a fusion algorithm of pose vectors may be used to perform pose fusion on the first pose and the second position to obtain the target position of the target object relative to the target coordinate system. For example, in this embodiment, the first pose and the second pose are vector-multiplied to obtain the target pose.

[0066] For example, the AR glasses perform vector multiplication of the posture of the game player in the field of view relative to the AR glasses and the posture of the AR glasses relative to the world coordinate system to obtain the posture of the game player in the field of view relative to the world coordinate system.

[0067] It can be seen from the above scheme that in a posture data processing method provided in the first embodiment of the present application, after obtaining the two-dimensional image captured by the image acquisition device for the target object, the first posture of the target object relative to the image acquisition device and the second posture of the image acquisition device relative to the target coordinate system obtained by SLAM are obtained using the two-dimensional image, and then the target posture of the target object relative to the target coordinate system is obtained based on these two postures. It can be seen that in the embodiment of the present application, the posture obtained based on the two-dimensional image and the posture obtained based on the SLAM method are integrated, and the target posture of the target object in the three-dimensional space can be obtained regardless of whether the target object is in motion, thereby avoiding the situation where the posture deviation is caused by the movement of the object when the posture of the object is acquired based on the SLAM method alone, thereby improving the accuracy of posture acquisition.

[0068] In one implementation, when the first pose is obtained in step 102, it can be implemented in the following manner: Figure 3 As shown in:

[0069] The third posture is processed using the first image to obtain a first posture of the target object relative to the image acquisition device.

[0070] The third posture here is a posture obtained based on the second image, and the second image is a two-dimensional image of the target object captured by the image acquisition device at a moment before the first image is captured.

[0071] That is to say, the first posture obtained in this embodiment is obtained by tracking the posture using the two-dimensional image of the target object based on the third posture of the target object at the previous moment.

[0072] In one implementation, the third posture may be a posture obtained by tracking the posture of the target object in the second image at an earlier time. Figure 4As shown in , the first moment is the current moment, i.e., the moment when the image acquisition device acquires the first image, the second moment is the moment before the first moment, i.e., the moment when the image acquisition device acquires the second image, and the third moment is the moment before the second moment. At the third moment, the target object has a corresponding historical posture. On this basis, the historical posture is tracked in combination with the second image acquired at the second moment to obtain the third posture of the target object relative to the image acquisition device at the second moment. On this basis, the third posture is tracked in combination with the first image acquired at the first moment to obtain the first posture of the target object relative to the image acquisition device at the first moment. For example, after the AR glasses obtain the historical posture of the game player relative to the AR glasses at the third moment, after the second image is acquired at the second moment, the second image is used to track the historical posture to obtain the third posture of the game player relative to the AR glasses at the second moment. After the first image is acquired at the first moment, the first image is used to track the third posture to obtain the first posture of the game player relative to the AR glasses at the first moment, i.e., the current moment.

[0073] In another implementation, the third posture is a posture obtained by performing posture recognition on the target object using the second image. In this embodiment, when performing posture recognition on the target object using the second image to obtain the third posture, it can be specifically implemented in the following manner, such as Figure 5 As shown in:

[0074] Step 501: Obtain two-dimensional feature points of the target object according to the second image.

[0075] The second image is a two-dimensional image of the target object captured by the image capture device before the first image is captured. Based on this, after the feature points of the second image are recognized, the two-dimensional feature points of the target object are obtained.

[0076] For example, the camera in the AR glasses captures a second image at a second moment (i.e., a moment before the moment when the camera captures the first image), performs feature point recognition on the second image, and obtains two-dimensional feature points of the target object at the second moment.

[0077] Step 502: Match the three-dimensional feature points of the target object with the two-dimensional feature points to obtain a matching relationship between the three-dimensional feature points and the two-dimensional feature points.

[0078] The three-dimensional feature points of the target object are obtained according to the three-dimensional scanning information of the target object. Specifically, in this embodiment, a scanner can be used to scan the target object to obtain the three-dimensional scanning information of the target object, and based on this, feature points are extracted from the three-dimensional scanning information to obtain the three-dimensional feature points of the target object. The three-dimensional feature points are represented by vectors.

[0079] For example, a scanner is configured in AR glasses, and the scanner is used to perform a 3D scan of the game player in the field of view, thereby obtaining three-dimensional scanning information, such as the posture information of the game player at any angle, and then feature points are extracted from the scanned 3D information, such as extracting feature points of the game player's facial features, limbs, etc., to obtain the three-dimensional feature points of the game player.

[0080] It should be noted that, in this embodiment, after the target object is three-dimensionally scanned for the first time, the obtained three-dimensional feature points of the target object may be cached so that the three-dimensional feature points can be directly read in the cache area when needed later.

[0081] In one implementation, in this embodiment, the two-dimensional feature points can be matched one-to-one with the scanned three-dimensional feature points, specifically by performing vector matching. After matching the corresponding three-dimensional feature points, a matching relationship between the three-dimensional feature points and the two-dimensional feature points is established.

[0082] The matching relationship between the three-dimensional feature points and the two-dimensional feature points can be represented by a vector group of the feature points.

[0083] For example, the AR glasses match the two-dimensional feature points of the game player identified in the second image with the three-dimensional feature points of the scanned game player to form a matching relationship between the three-dimensional feature points and the two-dimensional feature points of the game player, and each corresponding three-dimensional feature point and two-dimensional feature point forms a feature point group.

[0084] Step 503: Obtain a third posture of the target object relative to the image acquisition device at least based on the matching relationship and the acquisition parameters of the image acquisition device.

[0085] The acquisition parameters of the image acquisition device may be internal parameters of the image acquisition device, such as the memory of the camera, etc., which may include parameters such as focal length.

[0086] Based on this, in this embodiment, based on the known acquisition parameters and the matching relationship between the three-dimensional feature points and the two-dimensional feature points, the third posture of the target object relative to the image acquisition device is obtained through a corresponding algorithm. For example, based on the known camera internal parameters and matching relationship, AR glasses use an algorithm for estimating camera postures, such as the PNP (Perspective-n-Point) algorithm, to calculate the posture of the game player relative to the camera in the AR glasses and also relative to the AR glasses, that is, the third posture.

[0087] On this basis, after the image acquisition device acquires the first image at the first moment (the moment after the second moment, i.e., the current moment), the position and posture of the target object relative to the image acquisition device is tracked based on the third position and posture of the target object identified at the second moment using the first image, referring to Figure 4, to obtain the first position of the target object relative to the image acquisition device at the first moment. That is to say, in this embodiment, the position and posture of the target object are tracked by integrating the SLAM method on the basis of position and posture tracking based on visual information, so as to achieve the purpose of improving the accuracy of position and posture acquisition.

[0088] For example, a first image is captured at a first moment (the moment after the second moment, i.e., the current moment), and the AR glasses use the first image to perform posture tracking based on the third posture of the game player at the second moment, so as to obtain the first posture of the game player in the field of view relative to the AR glasses at the first moment. At the same time, the AR glasses use the SLAM algorithm to perform posture recognition on the first image to obtain the second posture of the AR glasses relative to the world coordinate system. Finally, the AR glasses perform vector multiplication of the two postures to obtain the posture of the game player in the field of view at the first moment relative to the world coordinate system.

[0089] In a specific implementation, in this embodiment, when the third posture is processed using the first image to obtain the first posture, a visual tracking algorithm can be used. The visual tracking algorithm is an algorithm initialized with the matching relationship between the two-dimensional feature points and the three-dimensional feature points of the target object. Specifically, the following steps can be performed, such as Figure 6 As shown in:

[0090] Step 601: Obtain two-dimensional feature points in a first image.

[0091] In this embodiment, feature point recognition may be performed on the first image to obtain two-dimensional feature points in the first image.

[0092] For example, the AR glasses use a feature point recognition algorithm to perform feature point recognition on a first image collected at a first moment, i.e., the current moment, to obtain two-dimensional feature points of the game player at the first moment.

[0093] Step 602: Obtain three-dimensional feature points corresponding to two-dimensional feature points in the first image according to the matching relationship.

[0094] Among them, the matching relationship is the matching relationship between the two-dimensional feature points of the second image used in the previous text and the three-dimensional feature points of the target object obtained by scanning, and this matching relationship is initialized to the visual tracking algorithm. The initialization here can be understood as storing the matching relationship in a storage path that can be read by the visual tracking algorithm. Based on this, in this embodiment, the matching relationship can be used to search and match the two-dimensional feature points in the first image for three-dimensional feature points, and then obtain the three-dimensional feature points corresponding to the two-dimensional feature points in the first image.

[0095] For example, after obtaining the two-dimensional feature points of the game player at the first moment, the AR glasses use the matching relationship between the two-dimensional feature points of the game player and the three-dimensional feature points to collect the three-dimensional feature points corresponding to the two-dimensional feature points of the game player at the first moment.

[0096] Step 603: Adjust the third posture data according to the three-dimensional feature points corresponding to the two-dimensional feature points in the first image to obtain the first posture of the target object relative to the image acquisition device.

[0097] For example, after obtaining the three-dimensional feature points corresponding to the two-dimensional feature points of the game player at the first moment, the AR glasses use the three-dimensional feature points to adjust the third posture of the game player at the second moment to track the first posture of the game player at the first moment, i.e., the current moment.

[0098] based on Figure 1 or Figure 3 In the scheme shown in FIG. 1 , after obtaining the target posture in step 104, the present embodiment may further include the following steps: Figure 7 As shown in:

[0099] Step 105: Process the target pose using a Kalman filter to obtain a new target pose.

[0100] Among them, the Kalman filter can perform posture smoothing processing on the target posture, so that when using a new target posture, it can avoid the situation where the target object jumps due to excessive posture changes, thereby improving the user experience of using electronic devices.

[0101] Specifically, the Kalman filter in this embodiment smoothes the current position and posture of the target object based on the position and posture of the target object at the previous moment. For example, in this embodiment, the third position and posture of the target object at the second moment is used to initialize or update the parameters of the Kalman filter, based on which the Kalman filter can smooth the target position and posture of the target object at the first moment at least based on the third position and posture.

[0102] In addition, after step 104, the present embodiment may further include the following steps: Figure 8 As shown in:

[0103] Step 106: Use the target pose to update the filter parameters of the Kalman filter.

[0104] The updated filter parameters are used to process the fourth pose, which is the pose of the target object relative to the target coordinate system at the next moment. For example, after the filter parameters of the Kalman filter are updated, after the fourth pose of the target object relative to the target coordinate system is obtained at the fourth moment, which is the next moment of the first moment, the Kalman filter can use the updated filter parameters to smooth the fourth pose.

[0105] Based on the above implementation scheme, after obtaining the pose of the target object relative to the target coordinate system at each moment, the Kalman filter can be used to smooth the pose. At the same time, the obtained pose can be used to update the filter parameters of the Kalman filter, so that the Kalman filter can smooth the pose obtained at the next moment with the updated filter parameters.

[0106] In a specific implementation, the filter parameters of the Kalman filter updated in this embodiment may include: the credibility of the Kalman filter in 6 degrees of freedom. In this embodiment, updating the filter parameters may include processing methods such as increase or decrease. For example, the corresponding filter parameters may be increased or decreased by the difference between the posture of the Kalman filter at the previous moment and the posture of the current moment.

[0107] refer to Fig. 9 , is a structural diagram of a posture data processing device provided in Embodiment 2 of the present application, which can be configured in an electronic device with an image acquisition device, such as AR glasses, etc. The technical solution in this embodiment is mainly used to improve the accuracy of posture.

[0108] Specifically, the device in this embodiment may include the following units:

[0109] An image acquisition unit 901 is used to acquire a first image, where the first image is a two-dimensional image acquired by an image acquisition device for a target object;

[0110] A first obtaining unit 902, configured to obtain a first pose of the target object relative to the image acquisition device at least based on the first image;

[0111] A second obtaining unit 903 is used to process the first image by means of SLAM to obtain a second posture of the image acquisition device relative to a target coordinate system;

[0112] The target obtaining unit 904 is used to obtain a target posture of the target object relative to the target coordinate system according to the first posture and the second posture.

[0113] It can be seen from the above scheme that in a posture data processing device provided in the second embodiment of the present application, after obtaining the two-dimensional image captured by the image acquisition device for the target object, the two-dimensional image is used to obtain the first posture of the target object relative to the image acquisition device and the second posture of the image acquisition device relative to the target coordinate system obtained by SLAM, and then the target posture of the target object relative to the target coordinate system is obtained based on these two postures. It can be seen that in the embodiment of the present application, the posture obtained based on the two-dimensional image and the posture obtained based on the SLAM method are integrated, and the target posture of the target object in the three-dimensional space can be obtained regardless of whether the target object is in motion, thereby avoiding the situation where the posture deviation is caused by the movement of the object when the posture of the object is acquired based on the SLAM method alone, thereby improving the accuracy of the posture.

[0114] In one implementation, the first acquisition unit 902 is specifically used to: use the first image to process the third posture to obtain the first posture of the target object relative to the image acquisition device; wherein the third posture is obtained based on the second image, and the second image is a two-dimensional image captured by the image acquisition device for the target object at a moment before the first image is captured.

[0115] Optionally, the third posture is obtained in the following manner: obtaining two-dimensional feature points of the target object based on the second image; matching the three-dimensional feature points of the target object with the two-dimensional feature points to obtain a matching relationship between the three-dimensional feature points and the two-dimensional feature points; the three-dimensional feature points of the target object are obtained based on three-dimensional scanning information of the target object; and obtaining the third posture of the target object relative to the image acquisition device based at least on the matching relationship and acquisition parameters of the image acquisition device.

[0116] Specifically, the first acquisition unit 902 is specifically used to: obtain two-dimensional feature points in the first image; obtain three-dimensional feature points corresponding to the two-dimensional feature points in the first image according to the matching relationship; adjust the third posture data according to the three-dimensional feature points corresponding to the two-dimensional feature points in the first image to obtain the first posture of the target object relative to the image acquisition device.

[0117] In one implementation, the device in this embodiment may further include the following units: Fig.10 As shown in:

[0118] The posture processing unit 905 is used to: after the target obtaining unit 904 obtains the target posture of the target object relative to the target coordinate system, use a Kalman filter to process the target posture to obtain a new target posture.

[0119] Optionally, the Kalman filter processes the target posture at least according to the third posture.

[0120] In one implementation, the device in this embodiment may further include the following units: Fig.11 As shown in:

[0121] The parameter updating unit 906 is used to update the filter parameters of the Kalman filter using the target posture, and the updated filter parameters are used to process the fourth posture, which is the posture of the target object relative to the target coordinate system at the next moment.

[0122] In one implementation, the target acquisition unit 904 is specifically configured to: perform vector multiplication processing on the first posture and the second posture to obtain a target posture of the target object relative to the target coordinate system.

[0123] It should be noted that the specific implementation of each unit in this embodiment can refer to the corresponding content in the previous text and will not be described in detail here.

[0124] refer to Fig.12 , is a schematic diagram of the structure of an electronic device provided in Embodiment 3 of the present application, and the electronic device may be an electronic device having an image acquisition device, such as AR glasses, etc. The technical solution in this embodiment is mainly used to improve the accuracy of posture.

[0125] Specifically, the electronic device in this embodiment may include the following structure:

[0126] An image acquisition device 1201, used for acquiring images;

[0127] The processor 1202 is used to obtain a first image, which is a two-dimensional image captured by the image acquisition device 1201 for the target object; obtain a first pose of the target object relative to the image acquisition device 1201 at least based on the first image; process the first image by means of synchronous localization and mapping (SLAM) to obtain a second pose of the image acquisition device 1201 relative to a target coordinate system; and obtain a target pose of the target object relative to the target coordinate system based on the first pose and the second pose.

[0128] It can be seen from the above scheme that in an electronic device provided in the third embodiment of the present application, after obtaining the two-dimensional image captured by the image acquisition device for the target object, the first posture of the target object relative to the image acquisition device and the second posture of the image acquisition device relative to the target coordinate system obtained by SLAM are obtained using the two-dimensional image, and then the target posture of the target object relative to the target coordinate system is obtained based on these two postures. It can be seen that in the embodiment of the present application, the posture obtained based on the two-dimensional image and the posture obtained based on the SLAM method are integrated, and the target posture of the target object in the three-dimensional space can be obtained regardless of whether the target object is in motion, thereby avoiding the situation where the posture deviation is caused by the movement of the object when the posture of the object is obtained based on the SLAM method alone, thereby improving the accuracy of the posture.

[0129] It should be noted that the electronic device in this embodiment may also include other components, such as a memory, a display screen, a touch screen, various sensors, etc., and the specific implementation of the processor can refer to the corresponding content in the previous text and will not be described in detail here.

[0130] In specific implementation, the technical solution in this application is applicable to various application scenarios where the position and posture of an object need to be positioned. In the embodiment of this application, the application scenario of AR glasses is taken as an example to illustrate the technical solution in this application, and the specific process is as follows:

[0131] 1. After entering the game application that needs to locate the position of three-dimensional objects in AR glasses, start the thread to run the SLAM algorithm and monitor the IMU sensor at the same time;

[0132] 2. Input a frame of image and pass it to the SLAM thread. If the 3D object recognition algorithm has been run, it can be determined that it is in tracking mode, that is, the mode of using images for posture tracking without posture recognition. At this time, jump to step 7. If the 3D object recognition algorithm has not been run, it can be determined that it is not in tracking mode and the posture recognition of the 3D object is required. At this time, execute step 3;

[0133] 3. Use the 3D object recognition algorithm to identify the 3D objects in the scene and output their 6DOF pose P in the camera coordinate system. tc ;

[0134] 4. Get the current SLAM algorithm result, that is, the camera's position P in the three-dimensional world cw ;

[0135] 5. Multiply the obtained results to calculate the position P of the three-dimensional object in the three-dimensional world tw , and at the same time, using the pose P tw Initialize the pose Kalman filter.

[0136] 6. Start a new thread and initialize the visual tracking algorithm. You can use the P obtained in step 5. tw Initialize, then switch to tracking mode, enter step 7, and execute step 12 at the same time to follow P tw Outputting corresponding images, such as images of three-dimensional objects after superimposing virtual information;

[0137] 7. Pass the newly acquired image to the visual tracking sub-thread;

[0138] 8. Synchronize the SLAM thread and the visual tracking thread to obtain the output results P of each thread. tc and P cw , at this time P tc is P at the previous moment tc The pose is tracked by visual tracking algorithm based on the

[0139] 9. Determine whether the visual tracking algorithm fails. If the visual tracking algorithm fails, that is, P tc If there is no output, the position and posture of the 3D object in the 3D world will not be updated, and step 12 will be executed. If there is no failure, step 10 will be executed.

[0140] 10. If both the SLAM thread and the visual tracking thread have output, then the outputs of both threads are P tc and P cw Multiply them together to calculate the position P of the three-dimensional object in the three-dimensional world tw ;

[0141] 11. P calculated based on the current frame tw The observation value is passed into the Kalman filter, which is used for smoothing, updates the filter parameters (state), and outputs the position and posture of the three-dimensional object in the three-dimensional world.

[0142] 12. P tw Output.

[0143] In summary, this application proposes a real-time tracking algorithm for three-dimensional objects that uses a Kalman filter to fuse visual tracking and SLAM. In this scheme, the visual tracking algorithm is run at a low frame rate, and combined with the real-time output of the system SLAM algorithm, it is fused through the Kalman filter, which can achieve stable tracking of static and moving objects, and smoothly output the position and posture of the object in three-dimensional space.

[0144] Based on this, the technical solution in this application has advantages: first, when the three-dimensional object is stationary, even if the camera shakes violently, the tracking algorithm can still work normally; second, when the three-dimensional object itself is moving, the technical solution of this application will also update the position and posture of the three-dimensional object in the three-dimensional world in a timely manner; then, after passing the three-dimensional object recognition algorithm, the amount of calculation can be reduced through visual tracking; in addition, after the three-dimensional object is recognized, even if it is removed from the field of view and enters the field of view again, there is no need to re-recognize it, and the tracking mode can be quickly entered; finally, for a video stream output scene, the output object posture is very stable. Of course, the advantages of the technical solution of this application can also be shown by the technical solutions of the above embodiments, which will not be described in detail here.

[0145] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0146] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0147] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0148] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for processing posture data, comprising: Obtaining a first image, where the first image is a two-dimensional image acquired by an image acquisition device for a target object; At least based on the first image, obtaining a first pose of the target object relative to the image acquisition device; Processing the first image by means of simultaneous localization and mapping (SLAM) to obtain a second pose of the image acquisition device relative to a target coordinate system; The first posture and the second posture are vector-multiplied to obtain a target posture of the target object relative to the target coordinate system.

2. The method according to claim 1, obtaining a first pose of the target object relative to the image acquisition device at least based on the first image, comprising: Using the first image, processing the third posture to obtain a first posture of the target object relative to the image acquisition device; The third posture is obtained based on a second image, and the second image is a two-dimensional image captured by the image acquisition device for the target object at a moment before the first image is captured.

3. The method according to claim 2, wherein the third posture is obtained by: Obtaining two-dimensional feature points of the target object according to the second image; Performing feature point matching on the three-dimensional feature points of the target object and the two-dimensional feature points to obtain a matching relationship between the three-dimensional feature points and the two-dimensional feature points; The three-dimensional feature points of the target object are obtained according to the three-dimensional scanning information of the target object; A third posture of the target object relative to the image acquisition device is obtained based at least on the matching relationship and acquisition parameters of the image acquisition device.

4. The method according to claim 3, using the first image, processing the third posture to obtain the first posture of the target object relative to the image acquisition device, comprising: Obtaining two-dimensional feature points in the first image; According to the matching relationship, obtaining three-dimensional feature points corresponding to the two-dimensional feature points in the first image; The third posture data is adjusted according to the three-dimensional feature points corresponding to the two-dimensional feature points in the first image to obtain the first posture of the target object relative to the image acquisition device.

5. The method according to claim 2, after obtaining the target position and posture of the target object relative to the target coordinate system, the method further comprises: The target posture is processed using a Kalman filter to obtain a new target posture.

6. According to the method of claim 5, the Kalman filter processes the target posture at least according to the third posture.

7. The method according to claim 5, further comprising: The target posture is used to update the filter parameters of the Kalman filter, and the updated filter parameters are used to process a fourth posture, where the fourth posture is the posture of the target object relative to the target coordinate system at the next moment.

8. A posture data processing device, comprising: An image acquisition unit, configured to acquire a first image, wherein the first image is a two-dimensional image acquired by an image acquisition device for a target object; A first obtaining unit, configured to obtain a first posture of the target object relative to the image acquisition device at least based on the first image; A second obtaining unit, configured to process the first image by means of SLAM to obtain a second posture of the image acquisition device relative to a target coordinate system; A target acquisition unit is used to perform vector multiplication processing on the first posture and the second posture to obtain a target posture of the target object relative to the target coordinate system.

9. An electronic device, comprising: An image acquisition device, used for acquiring images; A processor is used to obtain a first image, where the first image is a two-dimensional image captured by the image acquisition device for a target object; obtain a first pose of the target object relative to the image acquisition device based on at least the first image; process the first image by means of synchronous localization and mapping (SLAM) to obtain a second pose of the image acquisition device relative to a target coordinate system; and perform vector multiplication processing on the first pose and the second pose to obtain a target pose of the target object relative to the target coordinate system.

Images