Method, device and storage medium for obtaining posture information and determining object symmetry

Abstract

The present application discloses a method, device and storage medium for obtaining posture information and determining the symmetry of an object, and belongs to the field of posture detection technology. The method includes: obtaining the rotational symmetric degrees of freedom of a target object, and obtaining the posture information of the target object, and then adjusting the posture information of the target object according to the rotational symmetric degrees of freedom to obtain the adjusted posture information, and the adjusted posture information is used for the display of a virtual object, and the virtual object is an object associated with the target object. In this way, the posture information of the virtual object can be adaptively adjusted according to the rotational symmetric degrees of freedom of the target object, so that the display result of the virtual fusion can be the optimal display angle relative to the camera, thereby improving the user's viewing experience.

Description

Technical Field

[0001] The present application relates to the technical field of posture detection, and in particular to a method, device and storage medium for acquiring posture information and determining the symmetry of an object. Background Art

[0002] Object pose detection and tracking technology is a key technology in the field of environmental understanding. It can give machines the ability to perceive the spatial position and category of objects in the real environment, and has a wide range of uses in the fields of autonomous driving and augmented reality. However, objects in the real environment have complex changes in size, shape, material, texture, geometric details, etc., which poses a huge challenge to object pose detection and tracking technology.

[0003] At present, many objects have rotational symmetry. The images taken of objects with rotational symmetry at different observation positions may be the same, so the position and posture of objects with rotational symmetry are uncertain. In the process of position detection and tracking, if the objects with rotational symmetry are not processed specifically, it will be difficult to effectively identify the position and posture of the objects. Summary of the invention

[0004] The present application provides a method, device and storage medium for obtaining posture information and determining the symmetry of an object, which can realize adaptive adjustment of the posture information of the virtual object. The technical solution is as follows:

[0005] In a first aspect, a method for obtaining posture information is provided. In the method, the rotational symmetric degrees of freedom of a target object are obtained, and the posture information of the target object is obtained. Then, the posture information of the target object is adjusted according to the rotational symmetric degrees of freedom to obtain the adjusted posture information, and the adjusted posture information is used for displaying a virtual object, which is an object associated with the target object. Furthermore, the virtual object can also be displayed according to the adjusted posture information.

[0006] It should be noted that the rotational symmetry degree of freedom can be represented by N, where N is a natural number. A rotational symmetry degree of freedom of N means that the object can completely overlap with the object before rotation after rotating 360 / N degrees around the rotation axis. When the rotational symmetry degree of freedom is 0, it means that the object can completely overlap with the object before rotation after rotating any angle around the rotation axis.

[0007] In an embodiment of the present application, the position information of the virtual object can be adaptively adjusted according to the rotational symmetry degrees of freedom of the target object, so that the display result of the virtual fusion can be the optimal display angle relative to the camera, thereby improving the user's viewing experience.

[0008] In a first possible manner, the operation of obtaining the rotational symmetric degrees of freedom of the target object may be: obtaining an identifier of the target object, and using the rotational symmetric degrees of freedom corresponding to the identifier of the target object as the rotational symmetric degrees of freedom of the target object.

[0009] When obtaining the rotational symmetric degrees of freedom corresponding to the identification of the target object, the rotational symmetric degrees of freedom corresponding to the identification of the target object can be obtained from the corresponding relationship between the stored object identification and the rotational symmetric degrees of freedom according to the identification of the target object.

[0010] In a second possible manner, the operation of obtaining the rotational symmetric degrees of freedom of the target object may be: obtaining the rotational symmetric degrees of freedom of the target object input by a user.

[0011] In a third possible manner, the operation of obtaining the rotational symmetric degrees of freedom of the target object may be: obtaining an identifier of the target object, and then obtaining a three-dimensional (3D) model corresponding to the identifier of the target object. Multiple two-dimensional (2D) images of the target object are obtained by rendering the 3D model. Based on similarity calculation and cluster analysis of the multiple 2D images, the rotational symmetric degrees of freedom of the target object are obtained.

[0012] It should be noted that the multiple 2D images may be 2D views obtained by rendering the 3D model at different viewing angles. The rendering viewing angles of the multiple 2D images may be different. The rendering viewing angle of each 2D image may include a first rendering angle and a second rendering angle. In one possible scenario, the first rendering angles of the multiple 2D images are different, and the second rendering angles of the multiple 2D images are the same.

[0013] In the embodiments of the present application, it is possible to accurately identify whether the target object has rotational symmetry and obtain the rotational symmetric degrees of freedom of the target object when it has rotational symmetry, which helps to reduce the difficulty of object posture recognition during posture detection and tracking and improve the accuracy of object posture recognition.

[0014] Wherein, based on the similarity calculation and cluster analysis of the multiple 2D images, the operation of obtaining the rotational symmetric degrees of freedom of the target object can be: sampling the multiple 2D images according to the multiple rotational symmetric degrees of freedom to obtain multiple image pairs. Afterwards, the similarity of each image pair in the multiple image pairs is first calculated, and then the multiple image pairs are clustered according to the similarity and the rotational symmetric degrees of freedom to obtain multiple image sets. The image set with the most image pairs in the multiple image sets is determined as the target image set. Finally, when the minimum similarity of all image pairs in the target image set is greater than the similarity threshold, the rotational symmetric degrees of freedom corresponding to the target image set are determined as the rotational symmetric degrees of freedom of the target object.

[0015] It should be noted that each rotationally symmetric degree of freedom corresponds to at least one image pair. That is, for any one of the multiple rotationally symmetric degrees of freedom, multiple 2D images are sampled according to the rotationally symmetric degree of freedom to obtain at least one image pair corresponding to the rotationally symmetric degree of freedom. Optionally, each of the image pairs includes two 2D images, and when the rotationally symmetric degree of freedom is N, the first rendering angles of the two 2D images included in the image pair differ by 360 / N degrees.

[0016] In addition, the difference between the maximum similarity and the minimum similarity of all image pairs in an image set is less than or equal to a preset value, and all image pairs in the image set have the same rotational symmetric degrees of freedom. That is, the similarities of all image pairs in the image set are within a similarity interval, and the interval length of the similarity interval is a preset value.

[0017] Furthermore, when the preset value is 0, it indicates that all image pairs in the image set have the same similarity. At this time, for any image set in the multiple image sets, all image pairs in this image set have the same similarity. That is, this image set corresponds to one similarity and one rotational symmetric degree of freedom.

[0018] In an embodiment of the present application, since the target image set is the image set with the largest number of image pairs included in the multiple image sets, the image pairs included in the target image set can better reflect the symmetry of the target object. Therefore, the symmetry of the target object can be determined based on the target image set. When the minimum similarity of all image pairs in the target image set is greater than the similarity threshold, it indicates that the similarity of each image pair in the target image set is relatively high. In other words, it means that after the target object is rotated by the rotation angle indicated by the rotational symmetry degree of freedom corresponding to the target image set, it is very likely to completely overlap with the rotation before the rotation. Therefore, it can be determined that the target object has rotational symmetry at this time, and the rotational symmetry degree of freedom corresponding to the target image set can be determined as the rotational symmetry degree of freedom of the target object.

[0019] Among them, when the second rendering angles of the multiple 2D images are the same, the operation of determining the rotational symmetric degrees of freedom corresponding to the target image set as the rotational symmetric degrees of freedom of the target object can be: rendering to obtain 2D images of the 3D model at multiple different second rendering angles, and then sampling the 2D images at the multiple different second rendering angles according to the rotational symmetric degrees of freedom corresponding to the target image set to obtain multiple image pairs at different second rendering angles. Afterwards, when the similarities of the image pairs at the multiple different second rendering angles are all greater than the similarity threshold, it is determined that the target object has rotational symmetry, and the rotational symmetric degrees of freedom corresponding to the target image set are determined as the rotational symmetric degrees of freedom of the target object.

[0020] In the embodiment of the present application, a multi-view verification can be performed on the target object, that is, a 2D image under a plurality of different second rendering angles can be reacquired, and the symmetry of the target object can be verified accordingly. After the multi-view verification is successful, it can be determined that the target object has rotational symmetry, and the rotational symmetry degrees of freedom corresponding to the target image set can be determined as the rotational symmetry degrees of freedom of the target object, thereby improving the accuracy of determining the symmetry of the object.

[0021] Among them, the operation of rendering multiple two-dimensional 2D images of the target object according to the 3D model can be: determine multiple first coordinate points from the coordinate system, the first angle of each first coordinate point is 0 degrees, and the second angle of each first coordinate point is greater than 0 degrees and less than or equal to a preset angle. Afterwards, render to obtain the first reference image of the 3D model with each first coordinate point among the multiple first coordinate points as the observation position, and then determine the second angle of the first coordinate point corresponding to the first reference image with the largest area among the multiple first reference images obtained as the second rendering angle. Finally, render to obtain the 2D image of the 3D model with each second coordinate point among the multiple second coordinate points in the coordinate system as the observation position, the first angle of each second coordinate point is the first rendering angle, and the first rendering angle of each second coordinate point is greater than 0 degrees and less than or equal to 360 degrees, and the second angle of each second coordinate point is the determined second rendering angle.

[0022] It should be noted that the first angle is the angle between the projection of the line between the coordinate point and the origin on the target plane and the positive semi-axis of the second coordinate axis, and the second angle is the angle between the line between the coordinate point and the origin and the positive semi-axis of the first coordinate axis. The target plane is the plane where the second coordinate axis and the third coordinate axis are located.

[0023] In addition, the origin of the coordinate system is the center point of the 3D model, the first coordinate axis of the coordinate system is the rotation axis of the 3D model, the second coordinate axis of the coordinate system is perpendicular to the first coordinate axis, and the third coordinate axis of the coordinate system is perpendicular to both the first coordinate axis and the second coordinate axis.

[0024] In the embodiment of the present application, for the first coordinate point corresponding to the first reference image with the largest area among the multiple first reference images obtained, since the first angle of this first coordinate point is 0 degrees, it can be known that the second angle of this first coordinate point is the main observation angle, and the area of ​​the 2D image rendered is the largest. Therefore, at this time, the second coordinate point whose second angle in the coordinate system is the second angle of this first coordinate point and whose first angle is any angle within [0 degrees, 306 degrees] can be used as the observation position, so that a 2D image with relatively complete and comprehensive shape and texture information can be obtained later based on this observation position.

[0025] Among them, the operation of obtaining the posture information of the target object can be: inputting the image data obtained by shooting the target object into a posture detection classifier, and then analyzing the image data based on the posture detection classifier to obtain the posture information of the target object, and the posture detection classifier is obtained by training the image of the target object according to the rotational symmetry degree of freedom.

[0026] In a possible implementation, before adjusting the posture information of the target object according to the rotationally symmetric degree of freedom to obtain the adjusted posture information, the method also includes: determining whether it is necessary to adjust the posture information of the target object; adjusting the posture information of the target object according to the rotationally symmetric degree of freedom to obtain the adjusted posture information, specifically including: when it is necessary to adjust the posture information of the target object, adjusting the posture information of the target object according to the rotationally symmetric degree of freedom to obtain the adjusted posture information.

[0027] In a possible implementation, determining whether the position and posture information of the target object needs to be adjusted specifically includes: determining whether the position and posture information of the target object needs to be adjusted according to the rotational symmetric degrees of freedom of the target object and the yaw angle in the position and posture information of the target object.

[0028] In a possible implementation, determining whether the position and posture information of the target object needs to be adjusted specifically includes: when the rotational symmetric degree of freedom of the target object is 0 degrees and the yaw angle is 0, determining that the position and posture information of the target object does not need to be adjusted; or, when the rotational symmetric degree of freedom of the target object is not 0 degrees and the yaw angle satisfies When , it is determined that there is no need to adjust the position and posture information of the target object; wherein yaw1 is the yaw angle in the position and posture information of the target object, and N is the rotational symmetric degree of freedom of the target object.

[0029] In a possible implementation, when the target object does not have rotationally symmetric degrees of freedom, the virtual object is displayed according to the position information of the target object; or when the position information of the target object does not need to be adjusted, the virtual object is displayed according to the position information of the target object.

[0030] It should be noted that after the image data is input into the pose detection classifier, the pose detection classifier can output the identification of the target object contained in the image data and the position information of the projection points corresponding to the center point and vertices of the target object, or the pose detection classifier can output the identification of the target object contained in the image data and the pose information of the target object.

[0031] In addition, when the image data is analyzed based on the posture detection classifier to obtain the posture information of the target object, if the posture detection classifier outputs the identification of the target object contained in the image data and the position information of the projection points corresponding to the center point and vertices of the target object, then the 3D model corresponding to the identification of the target object can be obtained; based on the 3D model and the position information, the posture information of the target object is determined.

[0032] The posture information includes 6 degrees of freedom (DoF) posture information (x, y, z, yaw, pitch, roll), wherein x is the front-to-back displacement, y is the left-to-right displacement, z is the up-and-down displacement, yaw is the yaw angle, pitch is the pitch angle, and roll is the roll angle. The posture information of the target object is adjusted according to the rotationally symmetric degree of freedom to obtain the adjusted posture information. The operation may be: determining other posture information except the yaw angle in the posture information of the target object as the adjusted posture information, and the yaw angle is the angle at which the target object rotates around the rotation axis. Afterwards, when the rotationally symmetric degree of freedom of the target object is 0, the yaw angle in the adjusted posture information is determined to be 0 degrees; when the rotationally symmetric degree of freedom of the target object is not 0, the yaw angle in the adjusted posture information is determined according to the following formula;

[0033]

[0034] Among them, yaw2 is the yaw angle in the adjusted posture information, yaw1 is the yaw angle in the posture information of the target object, and N is the rotational symmetric degree of freedom of the target object.

[0035] In this case, the position information of the target object is (x, y, z, yaw1, pitch, roll), and the position information of the virtual object is (x, y, z, yaw2, pitch, roll). In addition, when the rotational symmetric degree of freedom of the target object is 0, yaw2 is 0 degrees; when the rotational symmetric degree of freedom of the target object is not 0, yaw2 is determined by the above formula.

[0036] In a second aspect, a method for determining the symmetry of an object is provided. In this method, after obtaining a 3D model of a target object, multiple 2D images of the target object are rendered according to the 3D model. Then, based on similarity calculation and cluster analysis of the multiple 2D images, the rotational symmetry degrees of freedom of the target object are obtained.

[0037] It should be noted that the multiple 2D images may be 2D views obtained by rendering the 3D model at different viewing angles. The rendering viewing angles of the multiple 2D images may be different. The rendering viewing angle of each 2D image may include a first rendering angle and a second rendering angle. In one possible scenario, the first rendering angles of the multiple 2D images are different, and the second rendering angles of the multiple 2D images are the same.

[0038] In addition, the rotational symmetry degree of freedom can be represented by N, where N is a natural number. A rotational symmetry degree of freedom of N means that the object can completely overlap with the object before rotation after rotating 360 / N degrees around the rotation axis. When the rotational symmetry degree of freedom is 0, it means that the object can completely overlap with the object before rotation after rotating any angle around the rotation axis.

[0039] In the embodiments of the present application, it is possible to accurately identify whether the target object has rotational symmetry and obtain the rotational symmetric degrees of freedom of the target object when it has rotational symmetry, which helps to reduce the difficulty of object posture recognition during posture detection and tracking and improve the accuracy of object posture recognition.

[0040] Wherein, based on the similarity calculation and cluster analysis of the multiple 2D images, the operation of obtaining the rotational symmetric degrees of freedom of the target object can be: sampling the multiple 2D images according to the multiple rotational symmetric degrees of freedom to obtain multiple image pairs. Afterwards, the similarity of each image pair in the multiple image pairs is first calculated, and then the multiple image pairs are clustered according to the similarity and the rotational symmetric degrees of freedom to obtain multiple image sets. The image set with the most image pairs in the multiple image sets is determined as the target image set. Finally, when the minimum similarity of all image pairs in the target image set is greater than the similarity threshold, the rotational symmetric degrees of freedom corresponding to the target image set are determined as the rotational symmetric degrees of freedom of the target object.

[0041] It should be noted that each rotationally symmetric degree of freedom corresponds to at least one image pair. That is, for any one of the multiple rotationally symmetric degrees of freedom, multiple 2D images are sampled according to the rotationally symmetric degree of freedom to obtain at least one image pair corresponding to the rotationally symmetric degree of freedom. Optionally, each of the image pairs includes two 2D images, and when the rotationally symmetric degree of freedom is N, the first rendering angles of the two 2D images included in the image pair differ by 360 / N degrees.

[0042] In addition, the difference between the maximum similarity and the minimum similarity of all image pairs in an image set is less than or equal to a preset value, and all image pairs in the image set have the same rotational symmetric degrees of freedom. That is, the similarities of all image pairs in the image set are within a similarity interval, and the interval length of the similarity interval is a preset value.

[0043] Furthermore, when the preset value is 0, it indicates that all image pairs in the image set have the same similarity. At this time, for any image set in the multiple image sets, all image pairs in this image set have the same similarity. That is, this image set corresponds to one similarity and one rotational symmetric degree of freedom.

[0044] In an embodiment of the present application, since the target image set is the image set with the largest number of image pairs included in the multiple image sets, the image pairs included in the target image set can better reflect the symmetry of the target object. Therefore, the symmetry of the target object can be determined based on the target image set. When the minimum similarity of all image pairs in the target image set is greater than the similarity threshold, it indicates that the similarity of each image pair in the target image set is relatively high. In other words, it means that after the target object is rotated by the rotation angle indicated by the rotational symmetry degree of freedom corresponding to the target image set, it is very likely to completely overlap with the rotation before the rotation. Therefore, it can be determined that the target object has rotational symmetry at this time, and the rotational symmetry degree of freedom corresponding to the target image set can be determined as the rotational symmetry degree of freedom of the target object.

[0045] Among them, when the second rendering perspectives of the multiple 2D images are the same, the operation of determining the rotational symmetric degrees of freedom corresponding to the target image set as the rotational symmetric degrees of freedom of the target object can be: rendering to obtain 2D images of the 3D model at multiple different second rendering perspectives, and then sampling the 2D images at the multiple different second rendering perspectives according to the rotational symmetric degrees of freedom corresponding to the target image set to obtain multiple image pairs at different second rendering perspectives. Afterwards, when the similarities of the image pairs at the multiple different second rendering perspectives are all greater than the similarity threshold, it is determined that the target object has rotational symmetry, and the rotational symmetric degrees of freedom corresponding to the target image set are determined as the rotational symmetric degrees of freedom of the target object.

[0046] In the embodiment of the present application, a multi-view verification can be performed on the target object, that is, a 2D image under a plurality of different second rendering angles can be reacquired, and the symmetry of the target object can be verified accordingly. After the multi-view verification is successful, it can be determined that the target object has rotational symmetry, and the rotational symmetry degrees of freedom corresponding to the target image set can be determined as the rotational symmetry degrees of freedom of the target object, thereby improving the accuracy of determining the symmetry of the object.

[0047] Among them, the operation of rendering multiple two-dimensional 2D images of the target object according to the 3D model can be: determine multiple first coordinate points from the coordinate system, the first angle of each first coordinate point is 0 degrees, and the second angle of each first coordinate point is greater than 0 degrees and less than or equal to a preset angle. Afterwards, render to obtain the first reference image of the 3D model with each first coordinate point among the multiple first coordinate points as the observation position, and then determine the second angle of the first coordinate point corresponding to the first reference image with the largest area among the multiple first reference images obtained as the second rendering angle. Finally, render to obtain the 2D image of the 3D model with each second coordinate point among the multiple second coordinate points in the coordinate system as the observation position, the first angle of each second coordinate point is the first rendering angle, and the first rendering angle of each second coordinate point is greater than 0 degrees and less than or equal to 360 degrees, and the second angle of each second coordinate point is the determined second rendering angle.

[0048] It should be noted that the first angle is the angle between the projection of the line between the coordinate point and the origin on the target plane and the positive semi-axis of the second coordinate axis, and the second angle is the angle between the line between the coordinate point and the origin and the positive semi-axis of the first coordinate axis. The target plane is the plane where the second coordinate axis and the third coordinate axis are located.

[0049] In addition, the origin of the coordinate system is the center point of the 3D model, the first coordinate axis of the coordinate system is the rotation axis of the 3D model, the second coordinate axis of the coordinate system is perpendicular to the first coordinate axis, and the third coordinate axis of the coordinate system is perpendicular to both the first coordinate axis and the second coordinate axis.

[0050] In the embodiment of the present application, for the first coordinate point corresponding to the first reference image with the largest area among the multiple first reference images obtained, since the first angle of this first coordinate point is 0 degrees, it can be known that the second angle of this first coordinate point is the main observation angle, and the area of ​​the 2D image rendered is the largest. Therefore, at this time, the second coordinate point whose second angle in the coordinate system is the second angle of this first coordinate point and whose first angle is any angle within [0 degrees, 306 degrees] can be used as the observation position, so that a 2D image with relatively complete and comprehensive shape and texture information can be obtained later based on this observation position.

[0051] Furthermore, after obtaining the rotational symmetric degrees of freedom of the target object, the position and posture information of the target object can also be obtained. Then, the position and posture information of the target object is adjusted according to the rotational symmetric degrees of freedom to obtain the adjusted position and posture information, and the adjusted position and posture information is used for displaying a virtual object, which is an object associated with the target object. Then, the virtual object can also be displayed according to the adjusted position and posture information.

[0052] In an embodiment of the present application, the position information of the virtual object can be adaptively adjusted according to the rotational symmetry degrees of freedom of the target object, so that the display result of the virtual fusion can be the optimal display angle relative to the camera, thereby improving the user's viewing experience.

[0053] Among them, the operation of obtaining the posture information of the target object can be: inputting the image data obtained by shooting the target object into a posture detection classifier, and then analyzing the image data based on the posture detection classifier to obtain the posture information of the target object, and the posture detection classifier is obtained by training the image of the target object according to the rotational symmetry degree of freedom.

[0054] In a possible implementation, before adjusting the posture information of the target object according to the rotationally symmetric degree of freedom to obtain the adjusted posture information, the method also includes: determining whether it is necessary to adjust the posture information of the target object; adjusting the posture information of the target object according to the rotationally symmetric degree of freedom to obtain the adjusted posture information, specifically including: when it is necessary to adjust the posture information of the target object, adjusting the posture information of the target object according to the rotationally symmetric degree of freedom to obtain the adjusted posture information.

[0055] In a possible implementation, when the posture information of the target object needs to be adjusted, it specifically includes: when the rotational symmetric degree of freedom of the target object is 0 degrees and the yaw angle is not 0, the posture information of the target object needs to be adjusted; or, when the rotational symmetric degree of freedom of the target object is not 0 degrees and the yaw angle does not belong to [0, 360 / N), the posture information of the target object needs to be adjusted; wherein N is the rotational symmetric degree of freedom of the target object. It should be noted that after the image data is input into the posture detection classifier, the posture detection classifier can output the identification of the target object contained in the image data and the position information of the projection points corresponding to the center point and vertices of the target object, or the posture detection classifier can output the identification of the target object contained in the image data and the posture information of the target object.

[0056] In addition, when the image data is analyzed based on the posture detection classifier to obtain the posture information of the target object, if the posture detection classifier outputs the identification of the target object contained in the image data and the position information of the projection points corresponding to the center point and vertices of the target object, then the 3D model corresponding to the identification of the target object can be obtained; based on the 3D model and the position information, the posture information of the target object is determined.

[0057] Furthermore, after obtaining the rotational symmetric degrees of freedom of the target object, multiple images of the target object and the pose information of the target object in each image can also be obtained, and then images with a yaw angle greater than 0 degrees and less than a rotation angle in the pose information are selected from the multiple images as training images, and the rotation angle is the ratio of 360 to the rotational symmetric degrees of freedom of the target object. Afterwards, the selected multiple training images are used as input samples, and the pose information of the target object in each training image in the multiple training images and the identification of the target object are used as sample labels, and the detection model to be trained is trained to obtain the pose detection classifier.

[0058] In an embodiment of the present application, since the images in the multiple images whose yaw angle is greater than the rotation angle are the same as the images in the multiple images whose yaw angle is greater than 0 degrees and less than the rotation angle, the images in the multiple images whose yaw angle is greater than the rotation angle can be filtered out, and only the images in the multiple images whose yaw angle is greater than 0 degrees and less than the rotation angle are selected as training images to train the detection model to be trained, thereby improving the performance of the trained posture detection classifier.

[0059] Among them, the posture information includes 6DoF posture information (x, y, z, yaw, pitch, roll), wherein the x is the front-to-back displacement, the v is the left-to-right displacement, the z is the up-and-down displacement, the yaw is the yaw angle, the pitch is the pitch angle, and the roll is the roll angle. The posture information of the target object is adjusted according to the rotationally symmetric degree of freedom, and the operation of obtaining the adjusted posture information can be: the other posture information except the yaw angle in the posture information of the target object is determined as the adjusted posture information, and the yaw angle is the angle when the target object rotates around the rotation axis. Afterwards, when the rotationally symmetric degree of freedom of the target object is 0, the yaw angle in the adjusted posture information is determined to be 0 degrees; when the rotationally symmetric degree of freedom of the target object is not 0, the yaw angle in the adjusted posture information is determined according to the following formula;

[0060]

[0061] Among them, yaw2 is the yaw angle in the adjusted posture information, yaw1 is the yaw angle in the posture information of the target object, and N is the rotational symmetric degree of freedom of the target object.

[0062] In this case, the position information of the target object is (x, y, z, yaw1, pitch, roll), and the position information of the virtual object is (x, y, z, yaw2, pitch, roll). In addition, when the rotational symmetric degree of freedom of the target object is 0, yaw2 is 0 degrees; when the rotational symmetric degree of freedom of the target object is not 0, yaw2 is determined by the above formula.

[0063] In a third aspect, a device for acquiring posture information is provided, the device comprising: a first acquisition module, a second acquisition module and an adjustment module. The first acquisition module is used to acquire the rotational symmetric degrees of freedom of the target object; the second acquisition module is used to acquire the posture information of the target object; the adjustment module is used to adjust the posture information of the target object according to the rotational symmetric degrees of freedom to obtain the adjusted posture information, and the adjusted posture information is used for displaying a virtual object, which is an object associated with the target object.

[0064] Optionally, the first acquisition module is specifically used to: acquire an identification of the target object; and use the rotational symmetric degrees of freedom corresponding to the identification of the target object as the rotational symmetric degrees of freedom of the target object.

[0065] Optionally, the first acquisition module is specifically used to: acquire the rotational symmetric degrees of freedom of the target object input by the user.

[0066] Optionally, the first acquisition module is specifically used to: obtain the identification of the target object; obtain a 3D model corresponding to the identification of the target object; render multiple 2D images of the target object according to the 3D model; and obtain the rotational symmetry degrees of freedom of the target object based on similarity calculation and cluster analysis of the multiple 2D images.

[0067] Optionally, the first acquisition module is specifically used to: sample the multiple 2D images according to multiple rotational symmetric degrees of freedom to obtain multiple image pairs; wherein each rotational symmetric degree of freedom corresponds to at least one image pair; calculate the similarity of each image pair in the multiple image pairs; perform cluster analysis on the multiple image pairs according to the similarity and the rotational symmetric degrees of freedom to obtain multiple image sets; wherein the difference between the maximum similarity and the minimum similarity of all image pairs in an image set is less than or equal to a preset value, and all image pairs in an image set have the same rotational symmetric degrees of freedom; determine the image set with the most image pairs in the multiple image sets as the target image set; when the minimum similarity of all image pairs in the target image set is greater than a similarity threshold, determine the rotational symmetric degrees of freedom corresponding to the target image set as the rotational symmetric degrees of freedom of the target object.

[0068] Optionally, each image pair includes two 2D images, and when the rotational symmetry degree of freedom is N, the first rendering angles of the two 2D images included in the image pair differ by 360 / N degrees.

[0069] Optionally, the second rendering angles of the multiple 2D images are the same, and the first acquisition module is specifically used to: render and obtain 2D images of the 3D model at multiple different second rendering angles; sample the 2D images at the multiple different second rendering angles according to the rotational symmetry degrees of freedom corresponding to the target image set to obtain multiple image pairs at the second rendering angles; when the similarities of the image pairs at the multiple different second rendering angles are all greater than a similarity threshold, determine that the target object has rotational symmetry, and determine the rotational symmetry degrees of freedom corresponding to the target image set as the rotational symmetry degrees of freedom of the target object.

[0070] Optionally, the second acquisition module is specifically used to: input the image data obtained by shooting the target object into a posture detection classifier; analyze the image data based on the posture detection classifier to obtain the posture information of the target object, and the posture detection classifier is obtained by training the image of the target object according to the rotational symmetry degrees of freedom of the target object.

[0071] Optionally, the posture information includes 6DoF posture information (x, y, z, yaw, pitch, roll), wherein x is the front-to-back displacement, y is the left-to-right displacement, z is the up-and-down displacement, yaw is the yaw angle, pitch is the pitch angle, and roll is the roll angle; the adjustment module is specifically used to: determine other posture information except the yaw angle in the posture information of the target object as the adjusted posture information, and the yaw angle is the angle at which the target object rotates around the rotation axis; when the rotational symmetric degree of freedom of the target object is 0, determine the yaw angle in the adjusted posture information to be 0 degrees; when the rotational symmetric degree of freedom of the target object is not 0, determine the yaw angle in the adjusted posture information according to the following formula;

[0072]

[0073] Among them, yaw2 is the yaw angle in the adjusted posture information, yaw1 is the yaw angle in the posture information of the target object, and N is the rotational symmetric degree of freedom of the target object.

[0074] Optionally, the device further includes: a display module, configured to display the virtual object according to the adjusted posture information.

[0075] Optionally, the first acquisition module is specifically used to: determine multiple first coordinate points from the coordinate system, the first angle of each first coordinate point is 0 degrees, the second angle of each first coordinate point is greater than 0 degrees and less than or equal to a preset angle, the first angle is the angle between the projection of the line between the coordinate point and the origin on the target plane and the positive semi-axis of the second coordinate axis, the second angle is the angle between the line between the coordinate point and the origin and the positive semi-axis of the first coordinate axis, the target plane is the plane where the second coordinate axis and the third coordinate axis are located; wherein the origin of the coordinate system is the center point of the 3D model, the first coordinate axis of the coordinate system is the rotation axis of the 3D model, and the second coordinate axis of the coordinate system is perpendicular to the first coordinate axis The third coordinate axis of the coordinate system is perpendicular to both the first coordinate axis and the second coordinate axis; a first reference image of the 3D model is obtained by rendering with each of the multiple first coordinate points as the observation position; the second angle of the first coordinate point corresponding to the first reference image with the largest area among the multiple first reference images obtained is determined as the second rendering angle; a 2D image of the 3D model is obtained by rendering with each of the multiple second coordinate points in the coordinate system as the observation position, the first angle of each second coordinate point is the first rendering angle, and the first rendering angle of each second coordinate point is greater than 0 degree and less than or equal to 360 degrees, and the second angle of each second coordinate point is the second rendering angle.

[0076] In a fourth aspect, a device for obtaining posture information is provided, the device for obtaining posture information comprising a processor and a communication interface, the processor receiving or sending information through the communication interface, the processor being used to call program instructions stored in a memory to implement the steps of the method for obtaining posture information provided in the first aspect above.

[0077] Exemplarily, the information includes data information or analog information, and the data information includes image data information, video data information or audio data information.

[0078] In a possible implementation, the device further includes a memory and a communication bus, and the processor, the memory, and the communication interface are all coupled to the communication bus.

[0079] In a fifth aspect, a computer-readable storage medium is provided, wherein instructions are stored in the computer-readable storage medium. When the instructions are executed on a computer or a processor, the computer or the processor executes the method for obtaining posture information described in the first aspect.

[0080] In a sixth aspect, a computer program product comprising instructions is provided, which, when executed on a computer or a processor, enables the computer or the processor to execute the method for obtaining posture information as described in the first aspect above.

[0081] The technical effects obtained by the third, fourth, fifth and sixth aspects are similar to those obtained by the corresponding technical means in the first aspect, and will not be repeated here.

[0082] In a seventh aspect, a device for determining the symmetry of an object is provided, the device comprising: a first acquisition module, a rendering module and an analysis module.

[0083] A first acquisition module is used to acquire a 3D model of a target object;

[0084] A rendering module, used for rendering a plurality of two-dimensional 2D images of the target object according to the 3D model;

[0085] The analysis module is used to obtain the rotational symmetric degrees of freedom of the target object based on similarity calculation and cluster analysis of the multiple 2D images.

[0086] Optionally, the analysis module is specifically used for:

[0087] Sampling a plurality of 2D images according to a plurality of rotationally symmetric degrees of freedom to obtain a plurality of image pairs, wherein each rotationally symmetric degree of freedom corresponds to at least one image pair;

[0088] Calculating the similarity of each image pair in the multiple image pairs;

[0089] Performing cluster analysis on the multiple image pairs according to the similarity and the rotational symmetry degrees of freedom to obtain multiple image sets; wherein the difference between the maximum similarity and the minimum similarity of all image pairs in one image set is less than or equal to a preset value, and all image pairs in one image set have the same rotational symmetry degrees of freedom;

[0090] Determine an image set having the most image pairs among the multiple image sets as a target image set;

[0091] When the minimum similarity among all image pairs in the target image set is greater than a similarity threshold, the rotational symmetric degrees of freedom corresponding to the target image set are determined as the rotational symmetric degrees of freedom of the target object.

[0092] Optionally, each image pair includes two 2D images, and when the rotational symmetry degree of freedom is N, the first rendering angles of the two 2D images included in the image pair differ by 360 / N degrees.

[0093] Optionally, the second rendering perspectives of the multiple 2D images are the same, and the analysis module is specifically configured to:

[0094] Rendering to obtain 2D images of the 3D model at multiple different second rendering angles;

[0095] Sampling the 2D images at the plurality of different second rendering angles according to the rotationally symmetric degrees of freedom corresponding to the target image set to obtain a plurality of image pairs at the second rendering angles;

[0096] When the similarities of the image pairs at the multiple different second rendering angles are all greater than the similarity threshold, it is determined that the target object has rotational symmetry, and the rotational symmetry degrees of freedom corresponding to the target image set are determined as the rotational symmetry degrees of freedom of the target object.

[0097] Optionally, the rendering module is specifically used for:

[0098] Determine a plurality of first coordinate points from a coordinate system, wherein a first angle of each first coordinate point is 0 degrees, a second angle of each first coordinate point is greater than 0 degrees and less than or equal to a preset angle, the first angle is the angle between a projection of a line between the coordinate point and the origin on a target plane and the positive semi-axis of the second coordinate axis, the second angle is the angle between a line between the coordinate point and the origin and the positive semi-axis of the first coordinate axis, and the target plane is the plane where the second coordinate axis and the third coordinate axis are located; wherein the origin of the coordinate system is the center point of the 3D model, the first coordinate axis of the coordinate system is the rotation axis of the 3D model, the second coordinate axis of the coordinate system is perpendicular to the first coordinate axis, and the third coordinate axis of the coordinate system is perpendicular to both the first coordinate axis and the second coordinate axis;

[0099] Rendering to obtain a first reference image of the 3D model with each first coordinate point among the plurality of first coordinate points as an observation position;

[0100] Determine a second angle of a first coordinate point corresponding to a first reference image with the largest area among the acquired multiple first reference images as a second rendering angle;

[0101] The 3D model is rendered to obtain a 2D image with each of the multiple second coordinate points in the coordinate system as the observation position, the first angle of each second coordinate point is the first rendering angle, and the first rendering angle of each second coordinate point is greater than 0 degree and less than or equal to 360 degrees, and the second angle of each second coordinate point is the second rendering angle.

[0102] Optionally, the device further comprises:

[0103] The second acquisition module is used to acquire the position and posture information of the target object;

[0104] The adjustment module is used to adjust the position and posture information of the target object according to the rotational symmetric degree of freedom to obtain the adjusted position and posture information, and the adjusted position and posture information is used for displaying a virtual object, which is an object associated with the target object.

[0105] Optionally, the second acquisition module is specifically used for:

[0106] Input the image data obtained by shooting the target object into the pose detection classifier;

[0107] The image data is analyzed based on a posture detection classifier to obtain the posture information of the target object. The posture detection classifier is obtained by training the image of the target object according to the rotationally symmetric degrees of freedom.

[0108] Optionally, the device further comprises:

[0109] A third acquisition module is used to acquire multiple images of the target object and the position and posture information of the target object in each image;

[0110] A selection module is used to select an image whose yaw angle in the posture information is greater than 0 degree and less than the rotation angle from multiple images as a training image, and the rotation angle is a ratio of 360 to the rotational symmetric degree of freedom of the target object;

[0111] The training module is used to use the selected multiple training images as input samples, and use the pose information and identification of the target object in each training image in the multiple training images as sample labels to train the detection model to be trained to obtain a pose detection classifier.

[0112] Optionally, the posture information includes 6DoF posture information (x, y, z, yaw, pitch, roll), where x is the front-to-back displacement, y is the left-to-right displacement, z is the up-and-down displacement, yaw is the yaw angle, pitch is the pitch angle, and roll is the roll angle;

[0113] The adjustment module is specifically used for:

[0114] Determine other posture information of the target object except the yaw angle in the posture information as adjusted posture information, where the yaw angle is the angle at which the target object rotates around the rotation axis;

[0115] When the rotational symmetric degree of freedom of the target object is 0, the yaw angle in the adjusted posture information is determined to be 0 degrees;

[0116] When the rotational symmetric degree of freedom of the target object is not 0, the yaw angle in the adjusted posture information is determined according to the following formula;

[0117]

[0118] Among them, yaw2 is the yaw angle in the adjusted posture information, yaw1 is the yaw angle in the posture information of the target object, and N is the rotational symmetric degree of freedom of the target object.

[0119] Optionally, the device further comprises:

[0120] The display module is used to display the virtual object according to the adjusted posture information.

[0121] In an eighth aspect, a device for determining the symmetry of an object is provided, the device comprising a processor and a communication interface, the processor receiving or sending information through the communication interface, the processor being used to call program instructions stored in a memory to implement the steps of the method for determining the symmetry of an object provided in the second aspect above.

[0122] Exemplarily, the information includes data information or analog information, and the data information includes image data information, video data information or audio data information.

[0123] In a possible implementation, the device further includes a memory and a communication bus, and the processor, the memory, and the communication interface are all coupled to the communication bus.

[0124] In a ninth aspect, a computer-readable storage medium is provided, wherein instructions are stored in the computer-readable storage medium. When the instructions are executed on a computer or a processor, the computer or the processor executes the method for determining the symmetry of an object as described in the second aspect.

[0125] In a tenth aspect, a computer program product comprising instructions is provided, which, when executed on a computer or a processor, enables the computer or the processor to execute the method for determining the symmetry of an object as described in the second aspect above.

[0126] The technical effects obtained in the seventh, eighth, ninth and tenth aspects are similar to the technical effects obtained by the corresponding technical means in the second aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0127] Figure 1 is a schematic diagram of a posture recognition system provided in an embodiment of the present application;

[0128] Figure 2 It is a structural schematic diagram of a device for obtaining posture information provided by an embodiment of the present application;

[0129] Figure 3 is a schematic diagram of another posture recognition system provided in an embodiment of the present application;

[0130] Figure 4 is a schematic diagram of the structure of a device provided in an embodiment of the present application;

[0131] Figure 5 is a flow chart of a method for determining the symmetry of an object provided in an embodiment of the present application;

[0132] Figure 6 is a schematic diagram of a rotationally symmetric degree of freedom provided in an embodiment of the present application;

[0133] Figure 7 is a schematic diagram of a first angle and a second angle provided in an embodiment of the present application;

[0134] Figure 8 is a schematic diagram of posture information provided by an embodiment of the present application;

[0135] Fig. 9 is a flow chart of a method for obtaining posture information provided by an embodiment of the present application;

[0136] Fig.10 is a structural schematic diagram of another device for obtaining posture information provided in an embodiment of the present application;

[0137] Fig.11 It is a structural schematic diagram of a device for determining the symmetry of an object provided in an embodiment of the present application. DETAILED DESCRIPTION

[0138] The terms "first", "second", etc. in the specification embodiments and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or precedence. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, including a series of steps or units. The method, system, product or device is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0139] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0140] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below in conjunction with the accompanying drawings.

[0141] Before explaining the embodiments of the present application in detail, the application scenarios of the embodiments of the present application are first described.

[0142] This application can be applied to the field of environmental understanding such as autonomous driving and augmented reality. For example, in the process of shopping in a mall, visiting a museum or exhibition hall, it is an important requirement to present the explanation information of commodities and exhibits to users in an intuitive and vivid way. This application can identify the posture of an object, and accurately superimpose the corresponding explanation information at different positions of the object in three-dimensional space according to the posture of the object, so that the user can see the best virtual-real fusion effect of the explanation information and the real object through the terminal.

[0143] Next, the system architecture involved in the embodiments of the present application is described.

[0144] Figure 1 is a schematic diagram of a posture recognition system provided in an embodiment of the present application. Figure 1 The system may include: a symmetry detection module 101, a model training module 102, a posture recognition module 103 and a display result adjustment module 104.

[0145] Specifically, the symmetry detection module 101 can detect whether the target object has rotational symmetry based on the 3D model of the target object and obtain the rotational symmetry degrees of freedom of the target object when it has rotational symmetry. The model training module 102 can select at least a part of the images from multiple images of the target object as training images based on the rotational symmetry degrees of freedom of the target object, and then use the selected multiple training images to train the detection model to be trained to obtain a posture detection classifier. In an optional case, the symmetry detection module 101 detects whether the target object has rotational symmetry during an offline training stage, and the symmetry detection module 101 obtains the rotational symmetry degrees of freedom of the target object during an offline training stage or obtains it in advance, and the posture detection classifier is also obtained by training during an offline training stage or by training in advance. In an optional case, the symmetry detection module 101 and the model training module 102 are located in a server.

[0146] The posture recognition module 103 can perform posture recognition on the input image to obtain the posture information of the target object in the image. Exemplarily, the input image is an image obtained by the user taking a photo of the target object, which can be a color (RedGreenBlue, RGB) image or a color depth (RedGreenBlueDepth, RGBD) image. The posture recognition module 103 can be a posture detection classifier trained in the offline training phase, and the posture detection classifier can be a neural network model or a deep learning model. The display result adjustment model 104 can determine the posture information of the virtual object according to the rotational symmetry degree of freedom of the target object and the posture information of the target object, and display the virtual object in the image according to the posture information of the virtual object to achieve a virtual-real fusion effect. In an optional case, the posture recognition module 103 and the display result adjustment model 104 are located on the terminal side, and are used to obtain the posture information of the virtual object associated with the target object according to user needs, and display the virtual object in the three-dimensional space where the target object is located according to the posture information.

[0147] Taking the museum scene as an example, the server performs symmetry detection on each object in the museum's library of objects to be exhibited in advance during the offline training phase to obtain a symmetry detection result, which includes the correspondence between the object identification and the rotational symmetry degree of freedom, that is, the symmetry detection result records the identification of each object, whether each object has rotational symmetry, and the rotational symmetry degree of freedom of the object with rotational symmetry. As shown in Table 1 below, an exemplary symmetry detection result is shown, and the symmetry detection result includes: object identification, rotational symmetry, and rotational symmetry degree of freedom. Among them, the object identification is used to identify the object. The rotational symmetry corresponding to each object identification is used to indicate whether the object identified by this object identification has rotational symmetry, and whether the object has rotational symmetry can be indicated by an indication value, such as when the rotational symmetry indication value corresponding to this object identification is 0, it means that the object does not have rotational symmetry, and when the rotational symmetry indication value corresponding to this object identification is 1, it means that the object has rotational symmetry. The rotational symmetry degree of freedom corresponding to each object identification is the rotational symmetry degree of freedom of this object when the object identified by this object identification has rotational symmetry degree of freedom.

[0148] Table 1

[0149] Object identification Rotational symmetry Rotational symmetric degrees of freedom 1 symmetry N1 2 Asymmetry 3 symmetry N2 4 symmetry N3

[0150] It should be noted that the embodiment of the present application only uses Table 1 above as an example to illustrate the symmetry detection result, and Table 1 above does not constitute a limitation on the embodiment of the present application.

[0151] In addition, the symmetry detection result may be stored in a memory of the server, or may be sent to a terminal and stored in a memory of the terminal.

[0152] Further, the server can train a pose detection classifier based on the symmetry detection result, and the pose detection classifier can be used to output the identification and pose information of the three-dimensional object. In an optional case, the pose detection classifier trained by the server is integrated in the terminal. When visiting a museum, the user can use the terminal to obtain an image of any object of interest in the museum in real time, and the terminal can input the image into the pose detection classifier to obtain the identification and pose information of the object. In an optional case, the terminal obtains the rotational symmetric degrees of freedom corresponding to the object identification from the server based on the obtained object identification, or obtains the rotational symmetric degrees of freedom corresponding to the object identification from the local memory based on the object identification. Further, the terminal adjusts the pose information obtained based on the pose detection classifier according to the obtained rotational symmetric degrees of freedom to obtain the adjusted pose information, and displays a virtual object associated with the object according to the adjusted pose information. Exemplarily, the virtual object is the introduction information of the object.

[0153] In an optional solution, after obtaining the rotational symmetric degrees of freedom of the target object and the position and posture information of the target object, it is determined whether the position and posture information of the target object needs to be adjusted. When it is determined that the position and posture information of the target object needs to be adjusted, the position and posture information of the target object is adjusted according to the rotational symmetric degrees of freedom to obtain the adjusted position and posture information. Exemplarily, it can be determined whether the position and posture information of the target object needs to be adjusted according to the rotational symmetric degrees of freedom of the target object and the yaw angle in the position and posture information of the target object.

[0154] Specifically, when the rotational symmetric degree of freedom of the target object is 0 degrees and the yaw angle in the position and posture information of the target object is 0, it is determined that the position and posture information does not need to be adjusted; or,

[0155] When the rotational symmetric degree of freedom of the target object is not 0 degrees, and the yaw angle in the target object's posture information satisfies , it is determined that there is no need to adjust the pose information of the target object; where yaw1 is the yaw angle in the pose information of the target object, and N is the rotational symmetric degree of freedom of the target object. When the rotational symmetric degree of freedom of the target object is 0 degrees and the yaw angle is not 0, the pose information of the target object needs to be adjusted; or when the rotational symmetric degree of freedom of the target object is not 0 degrees and the yaw angle does not belong to [0, 360 / N), the pose information of the target object needs to be adjusted; where N is the rotational symmetric degree of freedom of the target object.

[0156] In an optional solution, when the target object does not have rotationally symmetric degrees of freedom, the virtual object is displayed according to the position and posture information of the target object; or when the position and posture information of the target object does not need to be adjusted, the virtual object is displayed according to the position and posture information of the target object.

[0157] Figure 2 is a schematic diagram of the structure of a device for obtaining posture information provided by an embodiment of the present application. Figure 2 The device includes: a symmetry detection module 201, a posture recognition module 202 and a display result adjustment module 203.

[0158] The symmetry detection module 201 can obtain the rotational symmetric degrees of freedom of the target object. The posture recognition module 202 can obtain the posture information of the target object. Exemplarily, the posture recognition module 202 can be a posture detection classifier obtained by pre-training. The display result adjustment module 203 can adjust the posture information of the target object according to the rotational symmetric degrees of freedom of the target object to obtain the adjusted posture information, and the adjusted posture information can be used for the display of the virtual object associated with the target object. In an optional scheme, the symmetry detection module 201 can obtain the rotational symmetric degrees of freedom corresponding to the object identifier from the pre-stored object rotational symmetric degrees of freedom relationship table based on the object identifier, and the pre-stored object rotational symmetric degrees of freedom relationship table contains a one-to-one correspondence between the object identifier and the rotational symmetric degrees of freedom. Exemplarily, the pre-stored object rotational symmetric degrees of freedom relationship table can be stored in a local memory, or stored in the cloud or server side. In this case, the input of the symmetry detection module 201 is the object identifier, which can be output by the posture recognition module 202, or input by the user.

[0159] In an optional case, the symmetry detection module 201 can obtain the symmetry detection result of the target object by analyzing the acquired image information of the target object. In this case, the input of the symmetry detection module 201 is a 2D image of the target object, and the 2D image can be image data of the target object acquired by a camera. It should be understood that the symmetry detection result obtained by the symmetry detection module 201 can also be used for training a pose detection classifier. The training of the pose detection classifier can be offline training, or it can be online real-time training and updating.

[0160] In an optional case, the input of the posture recognition module 202 is 2D image data of the target object, which may be image data of the target object acquired by a camera. The posture recognition module 202 analyzes the input 2D image data to obtain the identification and posture information of the target object.

[0161] Figure 3is a schematic diagram of an exemplary posture recognition system provided in an embodiment of the present application. Figure 3 The system may include: a server 301 and a terminal 302.

[0162] The server 301 may include a symmetry detection module 311 and a model training module 312. The symmetry detection module 311 may detect whether the target object has rotational symmetry based on the 3D model of the target object and obtain the rotational symmetry degree of freedom of the target object when it has rotational symmetry. The model training module 312 may select at least a portion of images from multiple images of the target object as training images based on the rotational symmetry degree of freedom of the target object, and then use the selected multiple training images to train the detection model to be trained to obtain a posture detection classifier.

[0163] The terminal 302 may include a posture recognition module 321 and a display result adjustment module 322. The posture recognition module 321 may obtain the posture information of the target object, and the posture recognition module 321 may be a posture detection classifier trained by the server 301 and integrated in the terminal 302. The display result adjustment module 322 may adjust the posture information of the target object according to the rotational symmetric degrees of freedom of the target object to obtain the adjusted posture information, and the adjusted posture information may be used for displaying a virtual object associated with the target object.

[0164] Although Figure 3 Not shown, the terminal 302 may further include a degree of freedom acquisition module, which is used to acquire the rotational symmetry degrees of freedom of the target object. The rotational symmetry degrees of freedom of the target object may be acquired from the server 301. In an optional solution, the server 301 stores the symmetry detection result in the terminal 302, and the rotational symmetry degrees of freedom of the target object may also be acquired locally from the terminal 302.

[0165] Figure 4 is a schematic diagram of the structure of a device provided in an embodiment of the present application, Figure 1 The posture recognition system shown in can be realized by this device. Figure 4 The device includes at least one processor 401, a connector 402, a memory 403 and at least one communication interface 404.

[0166] Processor 401 may be a general-purpose central processing unit (CPU), a microprocessor, a system on chip (SOC), a processor integrated on SOC, a separate processor chip or controller, etc. Processor 401 may also include a dedicated processing device, such as an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or a digital signal processor (DSP), a dedicated video or graphics processor, a graphics processing unit (GPU) and a neural network processing unit (NPU). Processor 401 may be a processor group composed of multiple processors, and multiple processors are coupled to each other through one or more buses. Processor 401 may include an analog-to-digital converter (ADC) and a digital-to-analog converter (DAC) to realize the connection of signals between different components of the device. Processor 401 is used to realize the processing of media signals such as images, audio and video.

[0167] The above-mentioned parts are coupled through a connector 402. Exemplarily, the connector 402 includes various interfaces, transmission lines or communication buses, etc. These interfaces are usually electrical communication interfaces, but may also be mechanical interfaces or other forms of interfaces. The embodiments of the present application are not limited to this.

[0168] The memory 403 may be a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, or a random access memory (RAM) or other types of dynamic storage devices that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of an instruction or data structure and can be accessed by a computer, but is not limited thereto. The memory 403 may exist independently and be connected to the processor 401 through the connector 402. The memory 403 may also be integrated with the processor 401.

[0169] The communication interface 404 uses any transceiver-like device for communicating with other devices or communication networks, such as Ethernet, Radio Access Network (RAN), Wireless Local Area Networks (WLAN), etc. The communication interface 404 can be an interface for data input or output of the processor chip. In an optional case, the communication interface 404 can include a receiving interface and a transmitting interface. The communication interface 404 can be a High Definition Multimedia Interface (HDMI), a V-By-One interface, an Embedded Display Port (eDP), a Mobile Industry Processor Interface (MIPI), or a Display Port (DP), etc.

[0170] In a specific implementation, as an embodiment, the processor 401 may include one or more CPUs, such as Figure 4 CPU0 and CPU1 are shown in the figure.

[0171] In a specific implementation, as an embodiment, the device may include multiple processors, such as Figure 4401 and processor 405 are shown in FIG. Each of these processors may be a single-core processor (single-CPU) or a multi-core processor (multi-CPU). A processor herein may refer to one or more devices, circuits, and / or processing cores for processing data (such as computer program instructions).

[0172] In a specific implementation, as an embodiment, the apparatus may further include an output device 406 and an input device 407. The output device 406 communicates with the processor 401 and may display information in a variety of ways. For example, the output device 406 may be a liquid crystal display (LCD), a light emitting diode (LED) display device, a cathode ray tube (CRT) display device, or a projector. The input device 407 communicates with the processor 401 and may receive user input in a variety of ways. For example, the input device 407 may be a mouse, a keyboard, a touch screen device, or a sensor device.

[0173] The above-mentioned device can be a general-purpose computer device or a special-purpose computer device. In a specific implementation, the device can be a desktop computer, a portable computer, a network server, a PDA (Personal Digital Assistant), a mobile phone, a tablet computer, a wireless terminal device, a communication device or an embedded device. The embodiment of the present application does not limit the type of the device.

[0174] The memory 403 is used to store the program code 410 for executing the solution of the present application, and the processor 401 is used to execute the program code 410 stored in the memory 403. The device can implement the following by using the processor 401 and the program code 410 in the memory 403: Figure 5 The method for determining the symmetry of an object provided in the embodiment, or implementing the following Fig. 9 The embodiment provides a method for obtaining posture information.

[0175] Next, the method for determining the symmetry of an object is explained in detail. In one possible case, the method for determining the symmetry of an object may be performed by a server during an offline training phase.

[0176] Figure 5 is a flow chart of a method for determining the symmetry of an object provided in an embodiment of the present application. Figure 5 , the method comprising:

[0177] Step 501: Obtain a 3D model of a target object.

[0178] It should be noted that the target object is an object that needs to be detected for rotational symmetry, and the target object can usually be a three-dimensional object that can be seen by the user. For example, the target object can be a speaker, a car, a doll, etc.

[0179] In addition, the 3D model of the target object represents the representation of a target object in three-dimensional space, and is a data set composed of various geometric entities (such as triangles, lines, surfaces, etc.) and information such as color and material. For example, the 3D model of the target object can be a polygonal mesh model, which is composed of vertices, polygonal patches, and the connection relationship between vertices, and can also include color information such as material and texture. Polygonal patches are generally composed of simple polygons such as triangles and quadrilaterals. For another example, the 3D model of the target object can be an implicit surface. The implicit surface is represented by a zero set function about three-dimensional coordinates in Euclidean space, that is, F(x, y, z) = 0.

[0180] Specifically, there are multiple ways to obtain the 3D model of the target object. In one possible way, the 3D model of the target object can be directly input by the user. In another possible way, the 3D model of the target object can be generated based on multiple images obtained by shooting the target object. Of course, the 3D model of the target object can also be obtained by other ways, which is not limited in the embodiments of the present application.

[0181] Step 502: Rendering a plurality of 2D images of the target object according to the 3D model of the target object.

[0182] It should be noted that the multiple 2D images may be 2D views obtained by rendering the 3D model at different viewing angles. In practical applications, a rendering engine (renderer) may be used to render multiple 2D images of the 3D model at different viewing angles. That is, the rendering viewing angles of the multiple 2D images may be different. The rendering viewing angle of each 2D image may include a first rendering angle and a second rendering angle. In one possible scenario, the first rendering angles of the multiple 2D images are different, and the second rendering angles of the multiple 2D images are the same.

[0183] Specifically, the operation of step 502 may be: determining a plurality of reference viewpoints from the coordinate system, and then rendering to obtain a 2D image of the 3D model with each of the plurality of reference viewpoints as an observation position.

[0184] It should be noted that the origin of the coordinate system is the center point of the 3D model, the first coordinate axis of the coordinate system is the rotation axis of the 3D model, the second coordinate axis of the coordinate system is perpendicular to the first coordinate axis, and the third coordinate axis of the coordinate system is perpendicular to both the first coordinate axis and the second coordinate axis. That is, the coordinate system can be constructed based on the center point and the rotation axis of the 3D model.

[0185] In addition, the multiple reference viewpoints are all coordinate points in the coordinate system. The 3D model can be observed with the reference viewpoint as the observation position, and then the 2D view seen at the observation position is rendered to obtain a 2D image of the 3D model.

[0186] There are many ways to determine the center point and rotation axis of the 3D model. In one possible way, a preset center point can be used as the center point of the 3D model, and a straight line where a preset gravity direction is located can be used as the rotation axis of the 3D model. In another possible way, the center point of the bounding box of the 3D model can be used as the center point of the 3D model; based on the center point of the 3D model, the rotation axis of the 3D model is determined by a principal component analysis method. Of course, the center point and rotation axis of the 3D model can also be determined by other methods, which are not limited in the embodiments of the present application.

[0187] It should be noted that the preset center point and the preset gravity direction can be set in advance. For example, the user can input the preset center point and the preset gravity direction while inputting the 3D model.

[0188] In addition, the bounding box of the 3D model is a closed space that completely encompasses the 3D model. The 3D model is encapsulated in a simple bounding box, so that the shape of the 3D model is approximately replaced by the simple bounding box shape. Therefore, the center point of the bounding box of the 3D model can be directly used as the center point of the 3D model.

[0189] Furthermore, when determining the rotation axis of the 3D model by the principal component analysis method based on the center point of the 3D model, the positions of the k vertices of the 3D model can be respectively subtracted from the position of the center point of the 3D model to obtain k position differences, each of the k position differences including the difference in three main directions (horizontal, longitudinal and vertical); using each of the k position differences as each column element in the first matrix to obtain a first matrix, the first matrix being a matrix of 3 rows and k columns; multiplying the first matrix by the transpose of the first matrix to obtain a second matrix, the second matrix being a matrix of 3 rows and 3 columns; determining the eigenvalues ​​and eigenvectors of the second matrix; and using the straight line where the eigenvector corresponding to the largest eigenvalue among the eigenvalues ​​of the second matrix is ​​located as the rotation axis of the 3D model.

[0190] For example, the 3D model has 4 vertices, and the positions of the 4 vertices of the 3D model are (1, 1, 3), (5, 1, 3), (1, 7, 3), (5, 7, 3), and the position of the center point of the 3D model is (3, 4, 3). The positions of the 4 vertices of the 3D model are respectively subtracted from the position of the center point of the 3D model to obtain 4 position differences, which are (-2, -3, 0), (2, -3, 0), (-2, 3, 0), and (2, 4, 0). Each of the 4 position differences is used as an element of each column in the first matrix to obtain the first matrix. Exemplarily, the first matrix can be a matrix of 3 rows and 4 columns as shown below.

[0191] First matrix:

[0192] There are multiple ways to determine multiple reference viewpoints from the coordinate system. In one possible way, multiple coordinate points can be randomly selected from the coordinate system as multiple reference viewpoints. In another possible way, multiple first coordinate points can be determined from the coordinate system, the first angle of each first coordinate point is 0 degrees, and the second angle of each first coordinate point is greater than 0 degrees and less than or equal to a preset angle; render the 3D model to obtain a first reference image with each of the multiple first coordinate points as the observation position; determine the second angle of the first coordinate point corresponding to the first reference image with the largest area among the multiple first reference images obtained as the second rendering angle; determine each of the multiple second coordinate points in the coordinate system as a reference viewpoint, the first angle of each second coordinate point is the first rendering angle, and the first rendering angle of each coordinate point is greater than 0 degrees and less than or equal to 360 degrees, and the second angle of each second coordinate point is the determined second rendering angle.

[0193] It should be noted that the preset angle can be set in advance, such as the preset angle can be 180 degrees, 360 degrees, etc., and the embodiment of the present application is not limited to this.

[0194] In addition, the first angles of the multiple first coordinate points are the same, all of which are 0 degrees. The second angles of the multiple first coordinate points may be different, that is, the second angles of the multiple first coordinate points may be different angles within [0 degrees, preset angle]. The first angles of the multiple second coordinate points may be different, that is, the first angles of the multiple second coordinate points may be different angles within [0 degrees, 360 degrees]. The second angles of the multiple second coordinate points are the same, all of which are the second angles of the first coordinate points corresponding to the first reference image with the largest area among the multiple first reference images.

[0195] Furthermore, the first angle is the angle between the projection of the line between the coordinate point and the origin on the target plane and the positive semi-axis of the second coordinate axis, and the second angle is the angle between the line between the coordinate point and the origin and the positive semi-axis of the first coordinate axis. The target plane is the plane where the second and third coordinate axes are located. For example, Figure 6 As shown, the origin of the coordinate system is point O, the first coordinate axis is the z-axis, the second coordinate axis is the x-axis, the third coordinate axis is the y-axis, and the plane where the x-axis and the y-axis are located is the target plane. Assuming that the first coordinate point is coordinate point A, the first angle of coordinate point A is the angle between the projection of line segment OA on the target plane and the positive semi-axis of the x-axis, that is, angle θ; the second angle of coordinate point A is the angle between line segment OA and the positive semi-axis of the z-axis, that is, angle

[0196] Finally, for the 3D model, when a first reference image is obtained by rendering with a first coordinate point as the observation position, the first coordinate point corresponds to the first reference image. Similarly, for the 3D model, when a 2D image is obtained by rendering with a reference viewpoint as the observation position, the reference viewpoint corresponds to the 2D image, and the rendering perspective of the 2D image includes the first angle and the second angle of the reference viewpoint. Exemplarily, the first rendering angle is the first angle, and the second rendering angle is the second angle. When the second rendering angle is fixed to an angle When the second rendering angle is obtained In this way, when performing multi-view verification later, the second rendering angle can be used as the angle of division. An angle outside Get the second rendering angle The plurality of 2D images corresponding to the plurality of different first rendering angles can be obtained; further, the second rendering angle can be taken as an angle Get the second rendering angle The present invention further provides a plurality of 2D images corresponding to the plurality of different first rendering angles.

[0197] It is worth noting that, for the first coordinate point corresponding to the first reference image with the largest area among the multiple first reference images obtained, since the first angle of this first coordinate point is 0 degrees, it can be known that the area of ​​the 2D image rendered by using the second angle of this first coordinate point as the main observation angle is the largest. Therefore, at this time, the second coordinate point whose second angle in the coordinate system is the second angle of this first coordinate point and whose first angle is any angle within [0 degrees, 306 degrees] can be used as a reference viewpoint, so that a 2D image with relatively complete and comprehensive shape and texture information can be obtained based on this reference viewpoint later.

[0198] Step 503: Based on similarity calculation and cluster analysis of the multiple 2D images, the rotational symmetric degrees of freedom of the target object are obtained.

[0199] It should be noted that the rotational symmetry degree of freedom can be represented by N, where N is a natural number. A rotational symmetry degree of freedom of N means that the object can completely overlap with the object before rotation after rotating 360 / N degrees around the rotation axis. When the rotational symmetry degree of freedom is 0, it means that the object can completely overlap with the object before rotation after rotating any angle around the rotation axis. For example, Figure 7 The rotation axes of the six objects shown are all straight lines passing through the center points of the six objects and perpendicular to the planes the six objects are located in. The first object does not have rotational symmetry, while the other five objects have rotational symmetry, and the rotational symmetry degrees of freedom of the other five objects are 2, 3, 4, 5, and 0, respectively.

[0200] In addition, since the multiple 2D images are rendered based on the 3D model of the target object, the symmetry of the target object can be determined after similarity calculation and cluster analysis are performed on the multiple 2D images, and the rotational symmetry degrees of freedom of the target object can be obtained accordingly.

[0201] Specifically, the operation of step 503 may include the following steps (1) to (5):

[0202] (1) The multiple 2D images are sampled according to multiple rotationally symmetric degrees of freedom to obtain multiple image pairs.

[0203] It should be noted that each rotationally symmetric degree of freedom corresponds to at least one image pair. That is, for any one of the multiple rotationally symmetric degrees of freedom, by sampling multiple 2D images according to this rotationally symmetric degree of freedom, at least one image pair corresponding to this rotationally symmetric degree of freedom can be obtained.

[0204] In addition, for any one of the multiple image pairs, the image pair may include two 2D images, and the first rendering angles of the two 2D images included in the image pair differ by 360 / N degrees, where N is the rotational symmetry degree of freedom corresponding to the image pair. For example, if the rotational symmetry degree of freedom corresponding to the image pair is 2, the first rendering angles of the two 2D images included in the image pair differ by 180 degrees. Exemplarily, the image pair includes a first 2D image and a second 2D image, and the second rendering angles of the first 2D image and the second 2D image are both angles. The first rendering angle of the first 2D image is angle θ1, and the first rendering angle of the second 2D image is angle θ2, and θ1 differs from θ2 by 180 degrees.

[0205] Specifically, the operation of step (1) may be: for any one of the multiple rotationally symmetric degrees of freedom, divide 360 ​​degrees by the one rotationally symmetric degree of freedom to obtain a rotation angle; based on the rotation angle, obtain at least one image pair corresponding to the one rotationally symmetric degree of freedom from the multiple 2D images, and the angle difference between the first rendering angles of the two 2D images included in each image pair (that is, the first angle of the reference viewpoints corresponding to the two 2D images) is the rotation angle.

[0206] It should be noted that the rotation angle obtained by dividing 360 degrees by this rotationally symmetric degree of freedom can be called the rotation angle indicated by this rotationally symmetric degree of freedom.

[0207] In addition, when obtaining at least one image pair corresponding to this rotationally symmetric degree of freedom from the multiple 2D images based on the rotation angle, for any one of the multiple 2D images, the first angle of the reference viewpoint corresponding to this 2D image can be added to the rotation angle to obtain a specified angle, and then a reference viewpoint whose first angle is the specified angle is selected from the multiple reference viewpoints, and then this 2D image and the 2D image corresponding to the selected reference viewpoint are combined to form an image pair.

[0208] (2) Calculate the similarity of each image pair in the multiple image pairs.

[0209] It should be noted that the similarity of each image pair is the similarity between the two 2D images included in each image pair. That is, the greater the similarity of a certain image pair, the more similar the two 2D images included in this image pair are, and the smaller the similarity of a certain image pair, the less similar the two 2D images included in this image pair are.

[0210] Specifically, for any one of the multiple image pairs, there are multiple ways to calculate the similarity of the image pair. In one possible way, the similarity of the two 2D images can be determined based on the difference between the two 2D images. The smaller the difference, the greater the similarity, and the more similar the two 2D images are. For example, the Euclidean distance between the image features of the two 2D images included in the image pair can be calculated, and the value obtained by subtracting the calculated Euclidean distance from 1 is determined as the similarity of the image pair. The smaller the Euclidean distance, the greater the similarity, and the more similar the two 2D images are. In another possible way, the cosine distance between the image features of the two 2D images included in the image pair can be calculated, and the calculated cosine distance is determined as the similarity of the image pair. Of course, the similarity of the image pair can also be calculated in other ways, and the embodiments of the present application are not limited to this.

[0211] It should be noted that the image features of a 2D image can characterize the image content of the 2D image. The operation of obtaining the image features of a 2D image is similar to the operation of obtaining the image features of a certain image in the related art, and the embodiments of the present application will not be elaborated on this. For example, the 2D image can be input into a neural network model, and the image features of the 2D image can be output by the neural network model. The neural network model can be pre-set, such as the neural network model can be a convolutional neural network (CNN) model, etc., in which case the image features of the 2D image can be a 1*M-dimensional vector, where M is a positive integer.

[0212] (3) Cluster analysis is performed on the multiple image pairs according to similarity and rotational symmetry degrees of freedom to obtain multiple image sets.

[0213] It should be noted that, for any image set among the multiple image sets obtained by cluster analysis, the difference between the maximum similarity and the minimum similarity of all image pairs in this image set is less than or equal to a preset value, and at this time, the similarities of all image pairs in this image set are close. The similarities of all image pairs in this image set are within a similarity interval, and the interval length of this similarity interval is a preset value. Among them, the preset value can be set in advance, and the preset value can be set smaller. In addition, all image pairs in this image set have the same rotational symmetric degrees of freedom. That is, this image set corresponds to a similarity interval and a rotational symmetric degree of freedom.

[0214] Exemplarily, when the preset value is 0, it indicates that all pairs of images in the image set have the same similarity. At this time, for any image set in the multiple image sets, all pairs of images in this image set have the same similarity. That is, this image set corresponds to one similarity and one rotational symmetry degree of freedom.

[0215] (4) Determine the image set with the most image pairs among the multiple image sets as the target image set.

[0216] It should be noted that, since the target image set is the image set including the largest number of image pairs among the multiple image sets, the image pairs included in the target image set can better reflect the symmetry of the target object. Therefore, the symmetry of the target object can be determined based on the target image set.

[0217] (5) When the minimum similarity of all image pairs in the target image set is greater than a similarity threshold, the rotational symmetric degrees of freedom corresponding to the target image set are determined as the rotational symmetric degrees of freedom of the target object.

[0218] It should be understood that when determining the similarity of two 2D images according to the difference between the two 2D images, if the similarity of the image pair is greater than the similarity threshold, it means that the difference value of the image pair is less than a certain threshold.

[0219] It should be noted that if the minimum similarity of all image pairs in the target image set is less than or equal to the similarity threshold, it is determined that the target object does not have rotational symmetry.

[0220] In addition, the similarity threshold can be set in advance, and the similarity threshold can be set relatively large, such as the similarity threshold can be set to 80%, 90%, etc., which is not limited in the embodiment of the present application.

[0221] Furthermore, when the minimum similarity of all image pairs in the target image set is greater than the similarity threshold, it indicates that the similarity of each image pair in the target image set is relatively high. This means that the target object is likely to completely overlap with the rotation angle indicated by the rotational symmetry degree of freedom corresponding to the target image set after being rotated by the rotation angle indicated by the rotational symmetry degree of freedom corresponding to the target image set, so it can be determined that the target object has rotational symmetry, and the rotational symmetry degree of freedom corresponding to the target image set can be determined as the rotational symmetry degree of freedom of the target object.

[0222] When the minimum similarity of all image pairs in the target image set is less than or equal to the similarity threshold, it indicates that the similarity of some image pairs in the target image set is relatively low. This means that the target object is likely to not overlap with the object before the rotation after being rotated by the rotation angle indicated by the rotational symmetry degree of freedom corresponding to the target image set, and therefore it can be determined that the target object does not have rotational symmetry.

[0223] It is worth noting that when the second rendering angles of the multiple 2D images obtained in step 502 are the same, in order to improve the accuracy of determining the symmetry of the target object, the operation of determining the rotational symmetry degrees of freedom corresponding to the target image set as the rotational symmetry degrees of freedom of the target object in the above step (5) can be: rendering to obtain 2D images of the 3D model at multiple different second rendering angles; sampling the 2D images at the multiple different second rendering angles according to the rotational symmetry degrees of freedom corresponding to the target image set to obtain multiple image pairs at different second rendering angles; when the similarities of the image pairs at the multiple different second rendering angles are all greater than the similarity threshold, determining that the target object has rotational symmetry, and determining the rotational symmetry degrees of freedom corresponding to the target image set as the rotational symmetry degrees of freedom of the target object.

[0224] It should be understood that the second rendering angle is a second angle of the coordinate point as the observation position, and by changing the second rendering angle, it can be verified whether the target object has rotational symmetry under different rendering viewing angles.

[0225] It should be noted that the second rendering angles of the multiple 2D images obtained in step 502 are the same, and the first rendering angles are multiple different angles in [0 degrees, 360 degrees]. The multiple different second rendering angles indicate that the second angles of the coordinate points as the observation positions take different values, and the multiple different second rendering angles can be different from the second rendering angles of the multiple 2D images. For example, the second rendering angles of the multiple 2D images are all 30 degrees, and the multiple different second rendering angles can be 60 degrees, 120 degrees, 180 degrees, etc. The multiple different second rendering angles can also be any angle in [0 degrees, 360 degrees] except the second rendering angles of the multiple 2D images. In addition, when rendering to obtain the 2D images of the 3D model at multiple different second rendering angles, the 2D images of the 3D model at multiple different second rendering angles and multiple different first rendering angles can be rendered.

[0226] In addition, in the embodiment of the present application, a multi-view verification can be performed on the target object, that is, a 2D image under a plurality of different second rendering angles can be re-acquired, and the symmetry of the target object can be verified accordingly. After the multi-view verification is successful, it can be determined that the target object has rotational symmetry, and the rotational symmetry degrees of freedom corresponding to the target image set can be determined as the rotational symmetry degrees of freedom of the target object, thereby improving the accuracy of determining the symmetry of the object.

[0227] Furthermore, after the rotational symmetric degrees of freedom of the target object are obtained in step 503, the position and posture detection and tracking of the object can be performed accordingly. The specific process is as follows:

[0228] In the first step, the image of the target object can be trained according to the rotational symmetric freedom of the target object to obtain a pose detection classifier. Specifically, multiple images of the target object and the pose information of the target object in each image can be obtained; images with a yaw angle greater than 0 degrees and less than a rotation angle in the pose information are selected from the multiple images as training images, and the rotation angle is the ratio of 360 to the rotational symmetric freedom of the target object; the selected multiple training images are used as input samples, and the pose information of the target object in each training image in the multiple training images and the identification of the target object are used as sample labels, and the detection model to be trained is trained to obtain a pose detection classifier.

[0229] It should be noted that multiple images of the target object can be generated based on the 3D model of the target object, and the pose information of the target object in the image of the target object can be directly determined by the rendering engine. Of course, the multiple images of the target object can also be obtained by directly photographing the target object, and the pose information of the target object in the image of the target object can be obtained by manual annotation by the user. Of course, multiple images of the target object and the pose information of the target object in each image can also be obtained by other means, and the embodiments of the present application are not limited to this.

[0230] In addition, the pose information of the object refers to the pose of the object relative to the camera. The pose information can be 6DoF pose information (x, y, z, yaw, pitch, roll). Figure 8 As shown in the figure, x refers to the forward and backward translation of the object relative to the camera, that is, x is the forward and backward displacement; y refers to the left and right translation of the object relative to the camera, that is, y is the left and right displacement; z refers to the up and down translation of the object relative to the camera, that is, z is the up and down displacement; yaw refers to the yaw angle, which is the angle when the object rotates around the z axis; pitch refers to the pitch angle, which is the angle when the object rotates around the y axis; roll refers to the roll angle, which is the angle when the object rotates around the z axis. In general, the yaw angle is the angle when the object rotates around the rotation axis.

[0231] Furthermore, the object identifier is used to identify the object. For example, the object identifier may be the name, category, etc. of the object. Optionally, the object identifier may also be the number of the object, such as a digital number or an alphabetical number. This embodiment of the present application is not limited to this.

[0232] It is worth noting that, since the images in the multiple images whose yaw angle is greater than the rotation angle are the same as the images in the multiple images whose yaw angle is greater than 0 degrees and less than the rotation angle, the images in the multiple images whose yaw angle is greater than the rotation angle can be filtered out, and only the images in the multiple images whose yaw angle is greater than 0 degrees and less than the rotation angle are selected as training images to train the detection model to be trained, thereby improving the performance of the trained posture detection classifier.

[0233] Among them, multiple selected training images are used as input samples, and the posture information of the target object and the identification of the target object in each training image of the multiple training images are used as sample labels. The detection model to be trained is trained to obtain the operation of the posture detection classifier, which is similar to the model training process in the related technology, and the embodiment of the present application will not elaborate on this.

[0234] For example, each of the multiple training images can be input into the detection model to be trained. After obtaining the output of the detection model, the parameters in the detection model are adjusted using a preset loss function based on the gap between the output of the detection model and the sample labels of each of the multiple training images. The detection model after the parameter adjustment is completed is a pose detection classifier.

[0235] It should be noted that the preset loss function can be set in advance, such as the preset loss function can be λ pose L pose +λ conf L conf +λ cls L cls Among them, L pose Represents the loss function of pose prediction, L conf Represents the confidence loss function, L cls represents the loss function for object classification, λ pose YesL pose The weight coefficient, λ conf YesL conf The weight coefficient, λ cls YesL conf The weight coefficient of .

[0236] In addition, for the loss function L of pose prediction pose , using the true value y i and predicted values The distance between them is expressed as For the confidence loss function L conf , using the true value c i and predicted values The distance between them is expressed as The loss function L for object classification cls , using the true value p i and predicted values The cross entropy between is expressed as

[0237] In the second step, a posture detection classifier can be used to identify the posture of the object. Specifically, the image data obtained by shooting the target object can be input into the posture detection classifier, and the image data can be analyzed based on the posture detection classifier to obtain the posture information of the target object.

[0238] It should be noted that the image data may be an RGB image or an RGBD image, which is not limited in the present embodiment. After the image data is input into the pose detection classifier, the pose detection classifier may output the identification of the target object contained in the image data and the position information of the projection points corresponding to the center point and the vertex of the target object, or the pose detection classifier may output the identification of the target object contained in the image data and the pose information of the target object.

[0239] Among them, when the image data is analyzed based on the posture detection classifier to obtain the posture information of the target object, if the posture detection classifier outputs the identification of the target object contained in the image data and the position information of the projection points corresponding to the center point and vertices of the target object, then the 3D model corresponding to the identification of the target object can be obtained; based on the 3D model and the position information, the posture information of the target object is determined.

[0240] It should be noted that the operation of determining the pose information of the target object according to the 3D model and the position information can refer to the relevant technology, and the embodiment of the present application is not limited to this. For example, the pose information of the target object can be determined according to the 3D model and the position information using a pespective-n-point (PnP) algorithm.

[0241] Furthermore, in practical applications, it is often necessary to display virtual objects in images, such as displaying explanations of goods, exhibits, etc. in images taken of goods, exhibits, etc. during shopping in a mall, visiting a museum or exhibition hall, etc., so as to achieve a display effect of virtual-real integration. In this case, when the target object has rotational symmetry, the rotational symmetric degrees of freedom of the target object and the position and posture information of the target object can be obtained; the position and posture information of the target object can be adjusted according to the rotational symmetric degrees of freedom of the target object to obtain the adjusted position and posture information, which is used for the display of the virtual object. Afterwards, the virtual object can be displayed according to the adjusted position and posture information.

[0242] It should be noted that the pose information of the target object is usually used to indicate the pose of the target object relative to the camera. In this case, when obtaining the pose information of the target object, the image data obtained by shooting the target object can be input into the pose detection classifier, and the image data can be analyzed based on the pose detection classifier to obtain the pose information of the target object.

[0243] In addition, the virtual object is usually an object related to the target object, such as the virtual object can be the explanation information, marking information, etc. of the target object. The virtual object can be obtained according to the identification of the target object, that is, after obtaining the identification of the target object, the virtual object corresponding to the identification of the target object can be obtained from the correspondence between the object identification and the virtual object.

[0244] Among them, the operation of adjusting the pose information of the target object according to the rotational symmetric degree of freedom of the target object to obtain the adjusted pose information may be: determining other pose information except the yaw angle in the pose information of the target object as the adjusted pose information; when the rotational symmetric degree of freedom of the target object is 0, determining the yaw angle in the adjusted pose information to be 0 degrees; when the rotational symmetric degree of freedom of the target object is 0 and the yaw angle in the pose information is 0 degrees, the pose information of the target object is correct and does not need to be adjusted;

[0245] When the rotational symmetric degree of freedom of the target object is not 0, the yaw angle in the adjusted posture information is determined according to the following formula;

[0246]

[0247] It should be noted that yaw2 is the yaw angle in the adjusted pose information, yaw1 is the yaw angle in the pose information of the target object, and N is the rotational symmetric degree of freedom of the target object. According to the above formula, it can be deduced that when the rotational symmetric degree of freedom of the target object is not 0 and the yaw angle belongs to [0, 360 / N), yaw2 = yaw1. In this case, the pose information of the target object is correct and there is no need to adjust the pose information of the target object.

[0248] In this case, the pose information of the target object is (x, y, z, yaw1, pitch, roll), and the pose information of the virtual object is (x, y, z, yaw2, pitch, roll). Moreover, when the rotational symmetric freedom of the target object is 0, yaw2 is 0 degrees; when the rotational symmetric freedom of the target object is not 0, yaw2 is determined by the above formula. In this way, the pose information of the virtual object can be adaptively adjusted according to the rotational symmetric freedom of the target object, so that the display result of the virtual fusion can be the optimal display angle relative to the camera, thereby improving the user's viewing experience.

[0249] When the virtual object is displayed according to the adjusted posture information, the virtual object can be displayed in a preset area in the image data obtained by photographing the target object according to the posture of the virtual object.

[0250] It should be noted that the preset area can be set in advance, such as the preset area can be an area near the location of the target object. Of course, the preset area can also be other areas, and the embodiments of the present application are not limited to this.

[0251] In an embodiment of the present application, after obtaining a 3D model of the target object, multiple 2D images of the target object are rendered according to the 3D model. Afterwards, based on similarity calculation and cluster analysis of the multiple 2D images, the rotational symmetric degrees of freedom of the target object are obtained. In this way, it is possible to accurately identify whether the target object has rotational symmetry and obtain the rotational symmetric degrees of freedom of the target object when it has rotational symmetry, which helps to reduce the difficulty of object posture recognition in the posture detection and tracking process and improve the accuracy of object posture recognition.

[0252] Next, a method for obtaining posture information provided by an embodiment of the present application is explained in detail. In a possible situation, the method for obtaining posture information may be executed by a terminal during an offline training phase.

[0253] Fig. 9 is a flow chart of a method for obtaining posture information provided by an embodiment of the present application. Fig. 9 , the method comprises the following steps:

[0254] Step 901: Obtain the rotational symmetric degrees of freedom of the target object.

[0255] Specifically, step 901 may be implemented in any one of the following possible ways.

[0256] A first possible method is to obtain an identification of the target object; and use the rotational symmetric degree of freedom corresponding to the identification of the target object as the rotational symmetric degree of freedom of the target object.

[0257] When obtaining the identification of the target object, the identification of the target object may be obtained by obtaining the identification of the target object input by the user; or, the image data obtained by shooting the target object may be input into the posture detection classifier, and the posture detection classifier may output the identification of the target object. Of course, the identification of the target object may also be obtained by other means, which is not limited in the embodiments of the present application.

[0258] Among them, when obtaining the rotational symmetric degrees of freedom corresponding to the identification of the target object, the rotational symmetric degrees of freedom corresponding to the identification of the target object can be obtained from the corresponding relationship between the stored object identification and the rotational symmetric degrees of freedom according to the identification of the target object; or, the identification of the target object can be sent to the server, and the server returns the rotational symmetric degrees of freedom corresponding to the identification.

[0259] It should be noted that the pre-stored correspondence between the object identification and the rotational symmetry degrees of freedom may be input by a user, or obtained from a server, or preset when the device leaves the factory, etc., and this embodiment of the present application does not limit this.

[0260] The second possible way is to obtain the rotational symmetric degrees of freedom of the target object input by the user.

[0261] A third possible method is to obtain an identifier of the target object; obtain a 3D model corresponding to the identifier of the target object; render multiple 2D images of the target object based on the 3D model; and obtain the rotational symmetric degrees of freedom of the target object based on similarity calculation and cluster analysis of the multiple 2D images.

[0262] When obtaining the identification of the target object, the identification of the target object may be obtained by obtaining the identification of the target object input by the user; or, the image data obtained by shooting the target object may be input into the posture detection classifier, and the posture detection classifier may output the identification of the target object. Of course, the identification of the target object may also be obtained by other means, which is not limited in the embodiments of the present application.

[0263] When obtaining the 3D model corresponding to the identification of the target object, the corresponding 3D model can be obtained from the correspondence between the stored object identification and the 3D model according to the identification of the target object. The 3D model corresponding to the identification of the target object is the 3D model of the target object. The pre-stored correspondence between the object identification and the 3D model can be input by the user, or obtained from the server, or preset when the device leaves the factory, etc., and the embodiments of the present application do not limit this.

[0264] Among them, multiple 2D images of the target object are obtained by rendering the 3D model; based on the similarity calculation and cluster analysis of the multiple 2D images, the operation of obtaining the rotational symmetric degrees of freedom of the target object can be referred to Figure 5 The relevant steps in the method for determining the symmetry of an object provided in the embodiment will not be described in detail in the embodiment of the present application.

[0265] Step 902: Obtain the position and posture information of the target object.

[0266] Specifically, the image data obtained by shooting the target object can be input into a posture detection classifier, and the image data is analyzed based on the posture detection classifier to obtain the posture information of the target object.

[0267] It should be noted that the specific operation of step 902 can be referred to Figure 5 The relevant steps in the method for determining the symmetry of an object provided in the embodiment will not be described in detail in the embodiment of the present application.

[0268] Step 903: adjusting the position and posture information of the target object according to the rotational symmetric degrees of freedom of the target object to obtain adjusted position and posture information.

[0269] It should be noted that the adjusted position and posture information is used for displaying a virtual object, which is an object associated with the target object.

[0270] In addition, the specific operation of step 903 can refer to Figure 5 The relevant steps in the method for determining the symmetry of an object provided in the embodiment will not be described in detail in the embodiment of the present application.

[0271] In an embodiment of the present application, the rotational symmetric degrees of freedom of the target object are obtained, and the position and posture information of the target object is obtained. Afterwards, the position and posture information of the target object is adjusted according to the rotational symmetric degrees of freedom to obtain the adjusted position and posture information, and the adjusted position and posture information is used for displaying a virtual object, which is an object associated with the target object. In this way, the position and posture information of the virtual object can be adaptively adjusted according to the rotational symmetric degrees of freedom of the target object, so that the display result of the virtual fusion can be the optimal display angle relative to the camera, thereby improving the user's viewing experience.

[0272] Fig.10 is a schematic diagram of a structure of a device for obtaining posture information provided by an embodiment of the present application. The device can be implemented by software, hardware, or a combination of both to become part or all of a computer device. The computer device can be Figure 4 See the device shown. Fig.10 The device includes: a first acquisition module 1001, a second acquisition module 1002 and an adjustment module 1003.

[0273] The first acquisition module 1001 is used to execute Fig. 9 Step 901 in the embodiment;

[0274] The second acquisition module 1002 is used to execute Fig. 9 Step 902 in the embodiment;

[0275] Adjustment module 1003, used to execute Fig. 9 Step 903 in the embodiment.

[0276] Optionally, the first acquisition module 1001 is specifically used for:

[0277] Get the target object's ID;

[0278] The rotational symmetric degrees of freedom corresponding to the identification of the target object are taken as the rotational symmetric degrees of freedom of the target object.

[0279] Optionally, the first acquisition module 1001 is specifically used for:

[0280] Get the rotational symmetric degrees of freedom of the target object input by the user.

[0281] Optionally, the first acquisition module 1001 specifically includes:

[0282] A first acquisition subunit, used to acquire an identification of a target object;

[0283] A second acquisition subunit is used to acquire a 3D model corresponding to the identifier of the target object;

[0284] A rendering unit, used for rendering a plurality of 2D images of a target object according to the 3D model;

[0285] The analysis unit is used to obtain the rotational symmetric degrees of freedom of the target object based on similarity calculation and cluster analysis of multiple 2D images.

[0286] Optionally, the analysis unit is specifically used for:

[0287] Sampling a plurality of 2D images according to a plurality of rotationally symmetric degrees of freedom to obtain a plurality of image pairs, wherein each rotationally symmetric degree of freedom corresponds to at least one image pair;

[0288] Calculate the similarity of each image pair in multiple image pairs;

[0289] Performing cluster analysis on multiple image pairs according to similarity and rotational symmetry degrees of freedom to obtain multiple image sets; wherein the difference between the maximum similarity and the minimum similarity of all image pairs in an image set is less than or equal to a preset value, and all image pairs in an image set have the same rotational symmetry degrees of freedom;

[0290] Determine an image set having the most image pairs among the multiple image sets as a target image set;

[0291] When the minimum similarity among all image pairs in the target image set is greater than a similarity threshold, the rotational symmetric degrees of freedom corresponding to the target image set are determined as the rotational symmetric degrees of freedom of the target object.

[0292] Optionally, each image pair includes two 2D images, and when the rotational symmetry degree of freedom is N, the first rendering angles of the two 2D images included in the image pair differ by 360 / N degrees.

[0293] Optionally, the second rendering angles of the multiple 2D images are the same, and the analyzing unit is specifically configured to:

[0294] Rendering to obtain 2D images of the 3D model at multiple different second rendering angles;

[0295] Sampling a plurality of 2D images at different second rendering angles according to the rotationally symmetric degrees of freedom corresponding to the target image set to obtain a plurality of image pairs at different second rendering angles;

[0296] When the similarities of the image pairs at multiple different second rendering angles are all greater than the similarity threshold, it is determined that the target object has rotational symmetry, and the rotational symmetry degrees of freedom corresponding to the target image set are determined as the rotational symmetry degrees of freedom of the target object.

[0297] Optionally, the second acquisition module 1002 is specifically configured to:

[0298] Input the image data obtained by shooting the target object into the pose detection classifier;

[0299] The image data is analyzed based on a posture detection classifier to obtain the posture information of the target object. The posture detection classifier is obtained by training the image of the target object according to the rotationally symmetric degrees of freedom.

[0300] Optionally, the posture information includes 6DoF posture information (x, y, z, yaw, pitch, roll), where x is the front-to-back displacement, y is the left-to-right displacement, z is the up-and-down displacement, yaw is the yaw angle, pitch is the pitch angle, and roll is the roll angle; the adjustment module 1003 is specifically used for:

[0301] The other posture information of the target object except the yaw angle is determined as the adjusted posture information, and the yaw angle is the angle when the target object rotates around the rotation axis:

[0302] When the rotational symmetric degree of freedom of the target object is 0, the yaw angle in the adjusted posture information is determined to be 0 degrees;

[0303] When the rotational symmetric degree of freedom of the target object is not 0, the yaw angle in the adjusted posture information is determined according to the following formula;

[0304]

[0305] Among them, yaw2 is the yaw angle in the adjusted posture information, yaw1 is the yaw angle in the posture information of the target object, and N is the rotational symmetric degree of freedom of the target object.

[0306] Optionally, the device further comprises:

[0307] The display module is used to display the virtual object according to the adjusted posture information.

[0308] Optionally, the rendering unit is specifically used for:

[0309] Determine a plurality of first coordinate points from a coordinate system, wherein a first angle of each first coordinate point is 0 degrees, a second angle of each first coordinate point is greater than 0 degrees and less than or equal to a preset angle, the first angle is the angle between a projection of a line between the coordinate point and the origin on a target plane and the positive semi-axis of the second coordinate axis, the second angle is the angle between a line between the coordinate point and the origin and the positive semi-axis of the first coordinate axis, and the target plane is the plane where the second coordinate axis and the third coordinate axis are located; wherein the origin of the coordinate system is the center point of the 3D model, the first coordinate axis of the coordinate system is the rotation axis of the 3D model, the second coordinate axis of the coordinate system is perpendicular to the first coordinate axis, and the third coordinate axis of the coordinate system is perpendicular to both the first coordinate axis and the second coordinate axis;

[0310] Rendering to obtain a first reference image of the 3D model with each first coordinate point among the plurality of first coordinate points as an observation position;

[0311] Determine a second angle of a first coordinate point corresponding to a first reference image with the largest area among the acquired multiple first reference images as a second rendering angle;

[0312] The 3D model is rendered to obtain a 2D image with each of the multiple second coordinate points in the coordinate system as the observation position, the first angle of each second coordinate point is the first rendering angle, and the first rendering angle of each second coordinate point is greater than 0 degree and less than or equal to 360 degrees, and the second angle of each second coordinate point is the second rendering angle.

[0313] In an embodiment of the present application, the rotational symmetric degrees of freedom of the target object are obtained, and the position and posture information of the target object is obtained. Afterwards, the position and posture information of the target object is adjusted according to the rotational symmetric degrees of freedom to obtain the adjusted position and posture information, and the adjusted position and posture information is used for displaying a virtual object, which is an object associated with the target object. In this way, the position and posture information of the virtual object can be adaptively adjusted according to the rotational symmetric degrees of freedom of the target object, so that the display result of the virtual fusion can be the optimal display angle relative to the camera, thereby improving the user's viewing experience.

[0314] It should be noted that: the device for obtaining posture information provided in the above embodiment only uses the division of the above functional modules as an example when obtaining posture information. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the device for obtaining posture information provided in the above embodiment and the method embodiment for obtaining posture information belong to the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0315] Fig.11is a schematic diagram of a device for determining the symmetry of an object provided in an embodiment of the present application. The device can be implemented as part or all of a computer device by software, hardware, or a combination of both. The computer device can be Figure 4 See the device shown. Fig.11 The device includes: a first acquisition module 1101, a rendering module 1102 and an analysis module 1103.

[0316] The first acquisition module 1101 is used to execute Figure 5 Step 501 in the embodiment;

[0317] Rendering module 1102, used to execute Figure 5 Step 502 in the embodiment;

[0318] Analysis module 1103, used to perform Figure 5 Step 503 in the embodiment.

[0319] Optionally, the analysis module 1103 specifically includes:

[0320] A sampling unit, used for sampling a plurality of 2D images according to a plurality of rotationally symmetric degrees of freedom to obtain a plurality of image pairs; wherein each rotationally symmetric degree of freedom corresponds to at least one image pair;

[0321] A calculation unit, used for calculating the similarity of each image pair in the plurality of image pairs;

[0322] A clustering unit, used for performing cluster analysis on a plurality of image pairs according to similarity and rotational symmetry degrees of freedom to obtain a plurality of image sets; wherein the difference between the maximum similarity and the minimum similarity of all image pairs in an image set is less than or equal to a preset value, and all image pairs in an image set have the same rotational symmetry degrees of freedom;

[0323] A first determining unit, configured to determine an image set having the most image pairs among the multiple image sets as a target image set;

[0324] The second determining unit is configured to determine the rotational symmetric degrees of freedom corresponding to the target image set as the rotational symmetric degrees of freedom of the target object when the minimum similarity among all image pairs in the target image set is greater than a similarity threshold.

[0325] Optionally, each image pair includes two 2D images, and when the rotational symmetry degree of freedom is N, the first rendering angles of the two 2D images included in the image pair differ by 360 / N degrees.

[0326] Optionally, the second rendering viewing angles of the multiple 2D images are the same, and the second determining unit is specifically configured to:

[0327] Rendering to obtain 2D images of the 3D model at multiple different second rendering angles;

[0328] Sampling a plurality of 2D images at different second rendering angles according to the rotationally symmetric degrees of freedom corresponding to the target image set to obtain a plurality of image pairs at different second rendering angles;

[0329] When the similarities of the image pairs at multiple different second rendering angles are all greater than the similarity threshold, it is determined that the target object has rotational symmetry, and the rotational symmetry degrees of freedom corresponding to the target image set are determined as the rotational symmetry degrees of freedom of the target object.

[0330] Optionally, the rendering module 1102 is specifically used for:

[0331] Determine a plurality of first coordinate points from a coordinate system, wherein a first angle of each first coordinate point is 0 degrees, a second angle of each first coordinate point is greater than 0 degrees and less than or equal to a preset angle, the first angle is the angle between a projection of a line between the coordinate point and the origin on a target plane and the positive semi-axis of the second coordinate axis, the second angle is the angle between a line between the coordinate point and the origin and the positive semi-axis of the first coordinate axis, and the target plane is the plane where the second coordinate axis and the third coordinate axis are located; wherein the origin of the coordinate system is the center point of the 3D model, the first coordinate axis of the coordinate system is the rotation axis of the 3D model, the second coordinate axis of the coordinate system is perpendicular to the first coordinate axis, and the third coordinate axis of the coordinate system is perpendicular to both the first coordinate axis and the second coordinate axis;

[0332] Rendering to obtain a first reference image of the 3D model with each first coordinate point among the plurality of first coordinate points as an observation position;

[0333] Determine a second angle of a first coordinate point corresponding to a first reference image with the largest area among the acquired multiple first reference images as a second rendering angle;

[0334] The 3D model is rendered to obtain a 2D image with each of the multiple second coordinate points in the coordinate system as the observation position, the first angle of each second coordinate point is the first rendering angle, and the first rendering angle of each second coordinate point is greater than 0 degree and less than or equal to 360 degrees, and the second angle of each second coordinate point is the second rendering angle.

[0335] Optionally, the device further comprises:

[0336] The second acquisition module is used to acquire the position and posture information of the target object;

[0337] The adjustment module is used to adjust the position and posture information of the target object according to the rotational symmetric degree of freedom to obtain the adjusted position and posture information, and the adjusted position and posture information is used for displaying a virtual object, which is an object associated with the target object.

[0338] Optionally, the second acquisition module is specifically used for:

[0339] Input the image data obtained by shooting the target object into the pose detection classifier;

[0340] The image data is analyzed based on a posture detection classifier to obtain the posture information of the target object. The posture detection classifier is obtained by training the image of the target object according to the rotationally symmetric degrees of freedom.

[0341] Optionally, the device further comprises:

[0342] A third acquisition module is used to acquire multiple images of the target object and the position and posture information of the target object in each image;

[0343] A selection module is used to select an image whose yaw angle in the posture information is greater than 0 degree and less than the rotation angle from multiple images as a training image, and the rotation angle is a ratio of 360 to the rotational symmetric degree of freedom of the target object;

[0344] The training module is used to use the selected multiple training images as input samples, and use the pose information and identification of the target object in each training image in the multiple training images as sample labels to train the detection model to be trained to obtain a pose detection classifier.

[0345] Optionally, the posture information includes 6DoF posture information (x, y, z, yaw, pitch, roll), where x is the front-to-back displacement, y is the left-to-right displacement, z is the up-and-down displacement, yaw is the yaw angle, pitch is the pitch angle, and roll is the roll angle;

[0346] The adjustment module is specifically used for:

[0347] The other posture information of the target object except the yaw angle is determined as the adjusted posture information, and the yaw angle is the angle when the target object rotates around the rotation axis:

[0348] When the rotational symmetric degree of freedom of the target object is 0, the yaw angle in the adjusted posture information is determined to be 0 degrees;

[0349] When the rotational symmetric degree of freedom of the target object is not 0, the yaw angle in the adjusted pose information is determined according to the following formula:

[0350]

[0351] Among them, yaw2 is the yaw angle in the adjusted posture information, yaw1 is the yaw angle in the posture information of the target object, and N is the rotational symmetric degree of freedom of the target object.

[0352] Optionally, the device further comprises:

[0353] The display module is used to display the virtual object according to the adjusted posture information.

[0354] In an embodiment of the present application, after obtaining a 3D model of the target object, multiple 2D images of the target object are rendered according to the 3D model. Afterwards, based on similarity calculation and cluster analysis of the multiple 2D images, the rotational symmetric degrees of freedom of the target object are obtained. In this way, it is possible to accurately identify whether the target object has rotational symmetry and obtain the rotational symmetric degrees of freedom of the target object when it has rotational symmetry, which helps to reduce the difficulty of object posture recognition in the posture detection and tracking process and improve the accuracy of object posture recognition.

[0355] It should be noted that: the device for determining the symmetry of an object provided in the above embodiment only uses the division of the above functional modules as an example when determining the symmetry of an object. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the device for determining the symmetry of an object provided in the above embodiment and the method embodiment for determining the symmetry of an object belong to the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0356] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, data subscriber line (Digital Subscriber Line, DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium may be any available medium that a computer can access, or a data storage device such as a server or data center that includes one or more available media integrations. The available medium may be a magnetic medium (eg, a floppy disk, a hard disk, a magnetic tape), an optical medium (eg, a digital versatile disc (DVD)), or a semiconductor medium (eg, a solid state disk (SSD)).

[0357] The above-mentioned embodiments are provided for the present application and are not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A method for obtaining posture information, characterized in that: The method comprises: Obtain the rotational symmetric degrees of freedom of the target object; Acquire the position information of the target object, wherein the position information includes a yaw angle, and the yaw angle is an angle at which the target object rotates around a rotation axis; Determining whether it is necessary to adjust the position and posture information of the target object according to the rotationally symmetric degrees of freedom and the yaw angle; When the posture information of the target object needs to be adjusted, the posture information of the target object is adjusted according to the rotational symmetry degree of freedom to obtain adjusted posture information, wherein the adjusted posture information is the posture information of the virtual object, and the virtual object is an object associated with the target object, and the virtual object includes explanation information and / or marking information of the target object.

2. The method according to claim 1, characterized in that The obtaining of the rotational symmetric degrees of freedom of the target object specifically includes: Obtaining an identification of the target object; The rotational symmetric degree of freedom corresponding to the identifier of the target object is used as the rotational symmetric degree of freedom of the target object.

3. The method according to claim 1, characterized in that The obtaining of the rotational symmetric degrees of freedom of the target object specifically includes: The rotational symmetric degrees of freedom of the target object input by the user are obtained.

4. The method according to claim 1, characterized in that The obtaining of the rotational symmetric degrees of freedom of the target object specifically includes: Obtaining an identification of the target object; Obtaining a three-dimensional 3D model corresponding to the identifier of the target object; Rendering the 3D model to obtain a plurality of two-dimensional 2D images of the target object; Based on similarity calculation and cluster analysis of the multiple 2D images, the rotational symmetric degrees of freedom of the target object are obtained.

5. The method according to claim 4, characterized in that The obtaining of the rotational symmetric degrees of freedom of the target object based on similarity calculation and cluster analysis of the multiple 2D images specifically includes: Sampling the plurality of 2D images according to a plurality of rotationally symmetric degrees of freedom to obtain a plurality of image pairs, wherein each rotationally symmetric degree of freedom corresponds to at least one image pair; Calculating the similarity of each image pair in the plurality of image pairs; Performing cluster analysis on the multiple image pairs according to similarity and rotational symmetry degrees of freedom to obtain multiple image sets; wherein the difference between the maximum similarity and the minimum similarity of all image pairs in one image set is less than or equal to a preset value, and all image pairs in one image set have the same rotational symmetry degrees of freedom; Determine an image set having the most image pairs among the multiple image sets as a target image set; When the minimum similarity among all the image pairs in the target image set is greater than a similarity threshold, the rotational symmetric degrees of freedom corresponding to the target image set are determined as the rotational symmetric degrees of freedom of the target object.

6. The method according to claim 5, characterized in that Each of the image pairs includes two 2D images. When the rotational symmetry degree of freedom is N, the first rendering angles of the two 2D images included in the image pair differ by 360 / N degrees.

7. The method according to claim 6, characterized in that The second rendering angles of the two 2D images included in the image pair are the same, and determining the rotational symmetric degrees of freedom corresponding to the target image set as the rotational symmetric degrees of freedom of the target object specifically includes: Rendering to obtain 2D images of the 3D model at multiple different second rendering angles; Sampling the 2D images at the plurality of different second rendering angles according to the rotationally symmetric degrees of freedom corresponding to the target image set to obtain a plurality of image pairs at the second rendering angles; When the similarities of the image pairs at the multiple different second rendering angles are all greater than the similarity threshold, it is determined that the target object has rotational symmetry, and the rotational symmetry degrees of freedom corresponding to the target image set are determined as the rotational symmetry degrees of freedom of the target object.

8. The method according to any one of claims 1 to 7, characterized in that The obtaining of the position information of the target object specifically includes: Inputting the image data obtained by photographing the target object into a posture detection classifier; The image data is analyzed based on the posture detection classifier to obtain the posture information of the target object, and the posture detection classifier is obtained by training the image of the target object according to the rotationally symmetric degrees of freedom.

9. The method according to any one of claims 1 to 7, characterized in that: The determining whether the position and posture information of the target object needs to be adjusted specifically includes: When the rotational symmetric degree of freedom of the target object is 0 degrees and the yaw angle is 0 degrees, it is determined that there is no need to adjust the position and posture information of the target object; or, When the rotational symmetric degree of freedom of the target object is not 0 degrees and the yaw angle satisfies When , it is determined that there is no need to adjust the posture information of the target object; wherein, yaw1 is the yaw angle in the posture information of the target object, and N is the rotational symmetric degree of freedom of the target object.

10. The method according to any one of claims 1 to 7, characterized in that: The posture information includes 6-DOF posture information (x, y, z, yaw, pitch, roll), wherein x is the front-to-back displacement, y is the left-to-right displacement, z is the up-and-down displacement, yaw is the yaw angle, pitch is the pitch angle, and roll is the roll angle; The step of adjusting the position and posture information of the target object according to the rotationally symmetric degree of freedom to obtain the adjusted position and posture information specifically includes: Determining other posture information of the target object except the yaw angle as the adjusted posture information; When the rotational symmetric degree of freedom of the target object is 0 degrees, determining that the yaw angle in the adjusted posture information is 0 degrees; When the rotational symmetric degree of freedom of the target object is not 0 degrees, the yaw angle in the adjusted posture information is determined according to the following formula; Among them, the yaw2 is the yaw angle in the adjusted posture information, the yaw1 is the yaw angle in the posture information of the target object, and N is the rotational symmetric degree of freedom of the target object.

11. The method according to any one of claims 1 to 7, characterized in that: The method further comprises: The virtual object is displayed according to the adjusted posture information.

12. The method according to claim 11, characterized in that The method further comprises: When the target object does not have rotationally symmetric degrees of freedom, displaying the virtual object according to the position information of the target object; Or when there is no need to adjust the position and posture information of the target object, the virtual object is displayed according to the position and posture information of the target object.

13. The method according to any one of claims 4 to 7, characterized in that: The step of rendering a plurality of two-dimensional 2D images of the target object according to the 3D model specifically includes: Determine a plurality of first coordinate points from a coordinate system, wherein a first angle of each first coordinate point is 0 degrees, a second angle of each first coordinate point is greater than 0 degrees and less than or equal to a preset angle, the first angle is an angle between a projection of a line between the coordinate point and the origin on a target plane and a positive semi-axis of a second coordinate axis, the second angle is an angle between a line between the coordinate point and the origin and the positive semi-axis of the first coordinate axis, and the target plane is a plane where the second coordinate axis and the third coordinate axis are located; wherein the origin of the coordinate system is a center point of the 3D model, the first coordinate axis of the coordinate system is a rotation axis of the 3D model, the second coordinate axis of the coordinate system is perpendicular to the first coordinate axis, and the third coordinate axis of the coordinate system is perpendicular to both the first coordinate axis and the second coordinate axis; Rendering to obtain a first reference image of the 3D model with each first coordinate point among the plurality of first coordinate points as an observation position; Determine a second angle of a first coordinate point corresponding to a first reference image with the largest area among the acquired multiple first reference images as a second rendering angle; A 2D image of the 3D model is rendered with each of the multiple second coordinate points in the coordinate system as an observation position, the first angle of each second coordinate point is a first rendering angle, and the first rendering angle of each second coordinate point is greater than 0 degree and less than or equal to 360 degrees, and the second angle of each second coordinate point is the second rendering angle.

14. A device for obtaining posture information, characterized in that: The apparatus comprises a processor and a communication interface, the processor receives or sends information through the communication interface, and the processor is configured to call program instructions stored in a memory to implement the following steps: Obtain the rotational symmetric degrees of freedom of the target object; Acquire the position information of the target object, wherein the position information includes a yaw angle, and the yaw angle is an angle at which the target object rotates around a rotation axis; Determining whether it is necessary to adjust the position and posture information of the target object according to the rotationally symmetric degrees of freedom and the yaw angle; When the posture information of the target object needs to be adjusted, the posture information of the target object is adjusted according to the rotational symmetry degree of freedom to obtain adjusted posture information, wherein the adjusted posture information is the posture information of the virtual object, and the virtual object is an object associated with the target object, and the virtual object includes explanation information and / or marking information of the target object.

15. The device according to claim 14, characterized in that The processor is specifically used for: Obtaining an identification of the target object; The rotational symmetric degree of freedom corresponding to the identifier of the target object is used as the rotational symmetric degree of freedom of the target object.

16. The device according to claim 14, characterized in that The processor is specifically used for: Obtaining an identification of the target object; Obtaining a three-dimensional 3D model corresponding to the identifier of the target object; Rendering the 3D model to obtain a plurality of two-dimensional 2D images of the target object; Based on similarity calculation and cluster analysis of the multiple 2D images, the rotational symmetric degrees of freedom of the target object are obtained.

17. The device according to claim 16, characterized in that The processor is specifically used for: Sampling the plurality of 2D images according to a plurality of rotationally symmetric degrees of freedom to obtain a plurality of image pairs, wherein each rotationally symmetric degree of freedom corresponds to at least one image pair; Calculating the similarity of each image pair in the plurality of image pairs; Performing cluster analysis on the multiple image pairs according to similarity and rotational symmetry degrees of freedom to obtain multiple image sets; wherein the difference between the maximum similarity and the minimum similarity of all image pairs in one image set is less than or equal to a preset value, and all image pairs in one image set have the same rotational symmetry degrees of freedom; Determine an image set having the most image pairs among the multiple image sets as a target image set; When the minimum similarity among all the image pairs in the target image set is greater than a similarity threshold, the rotational symmetric degrees of freedom corresponding to the target image set are determined as the rotational symmetric degrees of freedom of the target object.

18. The device according to claim 17, characterized in that Each of the image pairs includes two 2D images. When the rotational symmetry degree of freedom is N, the first rendering angles of the two 2D images included in the image pair differ by 360 / N degrees.

19. The device according to claim 18, characterized in that The second rendering angles of the two 2D images included in the image pair are the same, and the processor is specifically configured to: Rendering to obtain 2D images of the 3D model at multiple different second rendering angles; Sampling the 2D images at the plurality of different second rendering angles according to the rotationally symmetric degrees of freedom corresponding to the target image set to obtain a plurality of image pairs at the second rendering angles; When the similarities of the image pairs at the multiple different second rendering angles are all greater than the similarity threshold, it is determined that the target object has rotational symmetry, and the rotational symmetry degrees of freedom corresponding to the target image set are determined as the rotational symmetry degrees of freedom of the target object.

20. The device according to any one of claims 14 to 19, characterized in that The processor is specifically used for: Inputting the image data obtained by photographing the target object into a posture detection classifier; The image data is analyzed based on the posture detection classifier to obtain the posture information of the target object, and the posture detection classifier is obtained by training the image of the target object according to the rotationally symmetric degrees of freedom.

21. The device according to any one of claims 14 to 19, characterized in that The processor is specifically used for: When the rotational symmetric degree of freedom of the target object is 0 degrees and the yaw angle is not 0 degrees, it is determined that the position and posture information of the target object needs to be adjusted; or, When the rotational symmetric degree of freedom of the target object is not 0 degrees and the yaw angle does not belong to [0, 360 / N), it is determined that the position information of the target object needs to be adjusted; wherein N is the rotational symmetric degree of freedom of the target object.

22. The device according to any one of claims 14 to 19, characterized in that The posture information includes 6 degrees of freedom (DoF) posture information (x, y, z, yaw, pitch, roll), wherein the x is the front-to-back displacement, y is the left-to-right displacement, z is the up-down displacement, yaw is the yaw angle, pitch is the pitch angle, and roll is the roll angle; The processor is specifically used for: Determining other posture information of the target object except the yaw angle as the adjusted posture information; When the rotational symmetric degree of freedom of the target object is 0 degrees, determining that the yaw angle in the adjusted posture information is 0 degrees; When the rotational symmetric degree of freedom of the target object is not 0 degrees, the yaw angle in the adjusted posture information is determined according to the following formula; Among them, the yaw2 is the yaw angle in the adjusted posture information, the yaw1 is the yaw angle in the posture information of the target object, and N is the rotational symmetric degree of freedom of the target object.

23. The device according to any one of claims 14 to 19, characterized in that The processor is further configured to: The virtual object is displayed according to the adjusted posture information.

24. The device according to claim 23, characterized in that The processor is further configured to: When the target object does not have rotationally symmetric degrees of freedom, displaying the virtual object according to the position information of the target object; or, When there is no need to adjust the position and posture information of the target object, the virtual object is displayed according to the position and posture information of the target object.

25. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores instructions, and when the instructions are executed on a computer or a processor, the computer or the processor executes the method according to any one of claims 1 to 13.

Images