Method for generating virtual object, method for processing three-dimensional pose, and electronic device

By performing two-dimensional pose estimation and three-dimensional pose analysis on physical objects in real scenes, virtual objects are generated, and the problem of low replicate accuracy of single images in the prior art is solved, and a higher virtual object shape and pose accuracy is achieved. sex.

WO2024228663A1PCT designated stage expired Publication Date: 2024-11-07ALIBABA INNOVATION PRIVATE LIMITED

Patent Information

Application Number
PCT/SG2024/050205
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-05-04
Filing Date
2024-03-29
Publication Date
2024-11-07

AI Technical Summary

Technical Problem

In Extended Reality (XR) and Virtual Reality (AR) projects, the accuracy is low when replicating physical objects based on a single image, resulting in the acquired virtual objects inaccurate enough shapes and poses. The existing technology has not effectively solved it. This question.

Method used

By capturing the two-dimensional images captured by the solid objects in the real scene, two-dimensional pose estimation is performed, and two-dimensional pose features are obtained from multiple different perspectives. Based on these features, three-dimensional pose estimation is performed to generate associations with the solid objects in the virtual scene in a virtual scene. virtual object.

Benefits of technology

The accuracy of replicating physical objects based on a single image is improved, and more accurate virtual objects are generated, solving the problem of low accuracy in the prior art.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure SG2024050205_07112024_PF_FP_ABST
    Figure SG2024050205_07112024_PF_FP_ABST
Patent Text Reader

Abstract

Disclosed in the embodiments of the present disclosure are a method for generating a virtual object, a method for processing a three-dimensional pose, and an electronic device. The method for generating a virtual object is applied in the field of extended reality, and the method comprises: capturing a two-dimensional image, which is obtained by means of photographing an entity object in a real scene; performing two-dimensional pose estimation on the entity object in the two-dimensional image, so as to obtain two-dimensional pose features at a plurality of different angles of view; performing three-dimensional pose estimation on the entity object on the basis of the two-dimensional pose features at the plurality of different angles of view, so as to obtain a three-dimensional pose result of the entity object; and on the basis of the three-dimensional pose result, generating a virtual object associated with the entity object in a virtual scene.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] Cross-Reference This disclosure claims priority to Chinese patent application No. 202310506988.8, filed with the China Patent Office on May 4, 2023, entitled "Method for Generating a Virtual Object, Method for Processing a Three-Dimensional Poses, and Electronic Device," the entire contents of which are incorporated herein by reference. Technical Field This disclosure relates to the field of extended reality data processing, and more specifically, to a method for generating a virtual object, a method for processing a three-dimensional pose, and an electronic device. Background: In business intelligence (BI) projects in extended reality (XR) and fitting projects in augmented reality (AR), it is necessary to accurately capture the shape and pose of the human body from videos. Related technologies for replicating physical objects from a single image suffer from low accuracy. Reproduction refers to constructing a virtual object corresponding to the physical object based on a single image, resulting in inaccurate shapes and poses of the acquired virtual object. Currently, no effective solution has been proposed to address the aforementioned issues. The present disclosure provides a method for generating a virtual object, a method for processing a three-dimensional pose, and an electronic device to at least address the technical issue of low accuracy in replicating a physical object based on a single image in related technologies. According to one aspect of the present disclosure, a method for generating a virtual object is provided, comprising: capturing a two-dimensional image of a physical object in a real scene; estimating the two-dimensional pose of the physical object in the two-dimensional image to obtain two-dimensional pose features from multiple different perspectives; estimating the three-dimensional pose of the physical object based on the two-dimensional pose features from the multiple different perspectives to obtain a three-dimensional pose result for the physical object; and generating a virtual object associated with the physical object in a virtual scene based on the three-dimensional pose result. According to another aspect of an embodiment of the present disclosure, another method for generating a virtual object is provided, comprising: displaying a two-dimensional image on the operation interface in response to an input instruction applied to an operation interface, wherein the two-dimensional image is an image obtained by photographing a physical object in a real scene; and displaying a virtual object associated with the physical object in a virtual scene on the operation interface in response to a generation instruction applied to the operation interface, wherein the virtual object is generated based on a three-dimensional pose result of the physical object, the three-dimensional pose result being obtained by estimating the three-dimensional pose of the physical object based on two-dimensional pose features at multiple different perspectives, and the two-dimensional pose features at multiple different perspectives being obtained by estimating the two-dimensional pose of the physical object in the two-dimensional image.According to another aspect of an embodiment of the present disclosure, a method for generating a virtual object in a virtual reality scene, such as a virtual reality (VR) device or an augmented reality (AR) device, is provided. The method includes: displaying a two-dimensional image on a presentation screen of the virtual reality (VR) device or the augmented reality (AR) device, wherein the two-dimensional image is an image obtained by photographing a physical object in a real scene; performing two-dimensional pose estimation on the physical object in the two-dimensional image to obtain two-dimensional pose features under multiple different perspectives; performing three-dimensional pose estimation on the physical object based on the two-dimensional pose features under the multiple different perspectives to obtain a three-dimensional pose result of the physical object; generating a virtual object associated with the physical object in the virtual scene based on the three-dimensional pose result; and driving the VR device or the AR device to render and display the virtual object. According to another aspect of an embodiment of the present disclosure, a method for generating a virtual object is provided, comprising: acquiring a two-dimensional image by calling a first interface, wherein the first interface includes a first parameter, and a parameter value of the first parameter is a two-dimensional image, where the two-dimensional image is an image obtained by photographing a physical object in a real scene; performing two-dimensional pose estimation on the physical object in the two-dimensional image to obtain two-dimensional pose features under multiple different perspectives; performing three-dimensional pose estimation on the physical object based on the two-dimensional pose features under the multiple different perspectives to obtain a three-dimensional pose result of the physical object; generating a virtual object associated with the physical object in a virtual scene based on the three-dimensional pose result; and outputting the virtual object by calling a second interface, wherein the second interface includes a second parameter, and a parameter value of the second parameter is the virtual object. According to another aspect of an embodiment of the present disclosure, a method for processing a three-dimensional pose is provided, comprising: capturing a two-dimensional image of a person in a real scene; estimating the two-dimensional pose of the person in the two-dimensional image to obtain two-dimensional pose features at multiple different perspectives; estimating the three-dimensional pose of the person based on the two-dimensional pose features at multiple different perspectives to obtain a three-dimensional pose result of the person; and matching the three-dimensional pose result with a preset three-dimensional pose to obtain a processing result of the three-dimensional pose result, wherein the processing result is configured to indicate whether the three-dimensional pose result matches the preset three-dimensional pose. According to another aspect of an embodiment of the present disclosure, an electronic device is provided, comprising: a memory storing an executable program; and a processor configured to execute the program, wherein the program, when executed, executes any one of the aforementioned methods. According to another aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, wherein the computer-readable storage medium includes a stored executable program, wherein the execution of the executable program controls the device containing the computer-readable storage medium to execute any one of the aforementioned methods.In the embodiments of the present disclosure, a two-dimensional image is captured by photographing a physical object in a real scene; a two-dimensional pose estimation is performed on the physical object in the two-dimensional image to obtain two-dimensional pose features at multiple different perspectives; a three-dimensional pose estimation is performed on the physical object based on the two-dimensional pose features at multiple different perspectives to obtain a three-dimensional pose result of the physical object; and a virtual object associated with the physical object in a virtual scene is generated based on the three-dimensional pose result. It is readily apparent that by performing two-dimensional pose estimation on the physical object in the two-dimensional image from multiple perspectives, two-dimensional pose features at multiple different perspectives are obtained. By using the two-dimensional pose features at multiple different perspectives to perform three-dimensional pose estimation, a highly accurate three-dimensional pose can be estimated, thereby enabling the generation of a more precise virtual object. This improves the accuracy of replicating a physical object based on a single image, thereby resolving the technical problem of low accuracy in replicating a physical object based on a single image in related technologies. It is readily apparent that the general description above and the detailed description that follow are provided for illustrative purposes and explanation of the present disclosure and do not constitute limitations of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS The drawings described herein are used to provide a further understanding of the present disclosure and constitute a part of the present disclosure. The illustrative embodiments of the present disclosure and their descriptions are used to explain the present disclosure and do not constitute an improper limitation on the present disclosure. In the accompanying drawings: FIG1 is a schematic diagram of a hardware environment of a virtual reality device for a method for generating a virtual object according to an embodiment of the present disclosure; FIG2 is a structural block diagram of a computing environment for a method for generating a virtual object according to an embodiment of the present disclosure; FIG3 is a flow chart of the method for generating a virtual object according to embodiment 1 of the present disclosure; FIG4 is a schematic diagram of the method for generating a virtual object according to embodiment 1 of the present disclosure; FIG5 is a schematic diagram of the method for generating a virtual object according to embodiment 1 of the present disclosure; FIG6 is a flow chart of the method for generating a virtual object according to embodiment 2 of the present disclosure; FIG7 is a flow chart of the method for generating a virtual object according to embodiment 3 of the present disclosure; FIG8 is a flow chart of the method for generating a virtual object according to embodiment 4 of the present disclosure; FIG9 is a flow chart of a method for processing a three-dimensional posture according to embodiment 5 of the present disclosure; FIG10 is a schematic diagram of a device for generating a virtual object according to embodiment 1 of the present disclosure; FIG11 is a schematic diagram of a device for generating a virtual object according to embodiment 2 of the present disclosure; FIG12 is a schematic diagram of a device for generating a virtual object according to embodiment 3 of the present disclosure; FIG13 is a schematic diagram of a device for generating a virtual object according to embodiment 4 of the present disclosure; FIG14 is a schematic diagram of a device for processing a three-dimensional posture according to embodiment 5 of the present disclosure; and FIG15 is a structural block diagram of a computer terminal according to an embodiment of the present disclosure.DETAILED DESCRIPTION To help those skilled in the art better understand the disclosed solutions, the following will provide a clear and complete description of the technical solutions in the disclosed embodiments, in conjunction with the accompanying drawings. It should be noted that the described embodiments represent only a portion of the disclosed embodiments, and are not exhaustive. All other embodiments devised by persons of ordinary skill in the art based on the disclosed embodiments without inventive effort should fall within the scope of protection of the disclosed embodiments. It should be noted that the terms "first," "second," and so on, in the specification and claims of the disclosed embodiments, and in the accompanying drawings, are used to distinguish similar objects and are not necessarily used to describe a specific order or precedence. It should be understood that such terms are interchangeable where appropriate, so that the disclosed embodiments described herein can be implemented in an order other than that illustrated or described herein. Furthermore, the terms "including," "comprising," and "having," as well as any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus comprising a series of steps or elements is not necessarily limited to the steps or elements explicitly listed and may include other steps or elements not explicitly listed or inherent to the process, method, product, or apparatus. First, some nouns or terms appearing in the description of the embodiments of the present disclosure are subject to the following interpretations: Pose estimation: This is an important task in computer vision. It can be the prediction of key points, that is, predicting the position coordinates of key points and then determining the spatial relationship between key points to construct virtual objects. 2D pose estimation is the prediction of the 2D coordinates of key points, while 3D pose estimation is the prediction of the 3D coordinates of key points. Kinematic tree: This is a tree structure determined by kinematics. The nodes in the tree structure can represent the key points of a physical object.

[0002] CVF: Cross-View Fusion, also known as cross-view fusion.

[0003] CVA: Cross-View Attention, also known as cross-view attention.

[0004] MLP: Multilayer Perceptron, also known as multi-layer perceptron.

[0005] Softmax: Normalized exponential function.

[0006] FFN: Feed Forward Networks, also known as feedforward neural networks.

[0007] MHSA: Multi-Head Self Attention, also known as multi-head self-attention layer.

[0008] SMPL: Skinned Multi-Person Linear Model, also known as parameterized human body model, based on human body posture 24X3 and human shape (3 6 / ?« as input, and output vertices are 3D human meshes with V = R3BWX3. The human pose § consists of the global rotation of the heel joint (such as the pelvis) and the local rotation of the 23 joint points relative to the parent node in the kinematic tree. The K 3D joint points can be calculated by linear combination of vertices as follows: Where, M 6 / ? kx689, R is a set of real numbers. Example 1: According to an embodiment of the present disclosure, a method for generating a virtual object is provided. It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system, such as a set of computer-executable instructions. Furthermore, although the flowcharts illustrate a logical order, in some cases, the steps shown or described may be executed in a different order. Figure 1 is a schematic diagram of the hardware environment of a virtual reality device for generating a virtual object according to an embodiment of the present disclosure. As shown in Figure 1, a virtual reality device 104 is connected to a terminal 106, which is connected to a server 102 via a network. The virtual reality device 104 is not limited to a virtual reality helmet, virtual reality glasses, or an all-in-one virtual reality device. The terminal 104 is not limited to a PC, mobile phone, or tablet computer. The server 102 may be a server corresponding to a media file operator. The network includes, but is not limited to, a wide area network, a metropolitan area network, or a local area network. Optionally, the virtual reality device 104 of this embodiment includes a memory, a processor, and a transmission device. The memory is configured to store an application program, which can be configured to: capture a two-dimensional image of a physical object in a real scene; perform two-dimensional pose estimation on the physical object in the two-dimensional image to obtain two-dimensional pose features from multiple different perspectives; perform three-dimensional pose estimation on the physical object based on the two-dimensional pose features from the multiple different perspectives to obtain three-dimensional pose results for the physical object; and generate a virtual object associated with the physical object in a virtual scene based on the three-dimensional pose results. This solves the technical problem of low accuracy in replicating physical objects based on a single image in related technologies, thereby achieving the goal of generating more accurate virtual objects. The terminal of this embodiment can be used to display a two-dimensional image on a presentation screen of a virtual reality (VR) device or an augmented reality (AR) device. The two-dimensional image is obtained by photographing a physical object in a real scene; perform two-dimensional pose estimation on the physical object in the two-dimensional image to obtain two-dimensional pose features at multiple different perspectives; perform three-dimensional pose estimation on the physical object based on the two-dimensional pose features at multiple different perspectives to obtain three-dimensional pose results of the physical object; generate a virtual object associated with the physical object in a virtual scene based on the three-dimensional pose results; drive the VR or AR device to render and display the virtual object; and send the virtual object to the virtual reality device 104. After receiving the virtual object, the virtual reality device 104 displays it at a target placement location.Optionally, the virtual reality device 104 of this embodiment includes an eye-tracking HMD (Head Mount Display) and an eye-tracking module that function similarly to those in the aforementioned embodiment. Specifically, the plane within the HMD is configured to display real-time images, and the eye-tracking module within the HMD is configured to obtain the real-time trajectory of the user's eye movements. The terminal of this embodiment obtains the user's position and motion information in real three-dimensional space through a tracking system, and calculates the three-dimensional coordinates of the user's head in virtual three-dimensional space, as well as the user's field of view in virtual three-dimensional space. The hardware structure block diagram shown in FIG1 can serve not only as an exemplary block diagram of the aforementioned AR / VR device (or mobile device), but also as an exemplary block diagram of the aforementioned server. In an optional embodiment, FIG2 shows a block diagram of an embodiment using the AR / VR device (or mobile device) shown in FIG1 as a computing node in the computing environment 201. Figure 2 is a block diagram of a computing environment for a method for generating a virtual object according to an embodiment of the present disclosure. As shown in Figure 2 , computing environment 201 includes multiple computing nodes (e.g., servers) (illustrated as 210-1 and 210-2 in the figure) running on a distributed network. Each computing node contains local processing and memory resources. A terminal user 202 can remotely run applications or store data in computing environment 201. Applications can be provided as multiple services 220-1, 220-2, 220-3, and 220-4 in computing environment 201, representing services "A," "D," "E," and "H," respectively. Terminal users 202 can access and provision services through a web browser or other software application on a client. In some embodiments, terminal user 202's provisioning and / or requests can be provided to an ingress gateway 230. Ingress gateway 230 can include a corresponding agent to handle provisioning and / or requests for services (e.g., one or more services provided in computing environment 201). Services are provided or deployed based on various virtualization technologies supported by computing environment 201. In some embodiments, services may be provided based on virtual machines (VMs), container-based virtualization, and / or similar approaches. VM-based virtualization can simulate a real computer by initializing a virtual machine, allowing programs and applications to execute without directly accessing any actual hardware resources.While virtual machines virtualize machines, container-based virtualization allows you to launch containers to virtualize an entire operating system (OS) so that multiple workloads can run on a single operating system instance. In one embodiment based on container virtualization, several service containers can be assembled into a Pod (e.g., a Kubernetes Pod). For example, as shown in FIG2 , service 220-2 can be configured with one or more Pods 240-1, 240-2, …, 240-N (collectively referred to as Pods). A Pod can include a proxy 245 and one or more containers 242-1, 242-2, …, 242-M (collectively referred to as containers). One or more containers in a Pod handle requests related to one or more corresponding functions of the service. The proxy 245 typically controls network functions related to the service, such as routing and load balancing. Other services can also be configured with similar Pods. During operation, executing a user request from end user 202 may require invoking one or more services in computing environment 201. Executing one or more functions of one service may require invoking one or more functions of another service. As shown in FIG2 , service "A" 220-1 receives a user request from end user 202 from ingress gateway 230. Service 220-1 may invoke service "D" 220-2. Service "D" 220-2 can request service "E" 220-3 to perform one or more functions. The computing environment described above can be a cloud computing environment, where resource allocation is managed by the cloud service provider, allowing function development without having to consider implementing, adjusting, or scaling servers. This computing environment allows developers to execute code in response to events without building or maintaining complex infrastructure. Services can be partitioned to complete a set of functions that can be automatically and independently scaled, rather than expanding a single hardware device to handle potential load. Within the above operating environment, the present disclosure provides a method for generating a virtual object as shown in FIG3 . It should be noted that the method for generating a virtual object in this embodiment can be executed by the mobile terminal of the embodiment shown in FIG1 . FIG3 is a flowchart of the method for generating a virtual object according to Example 1 of the present disclosure. As shown in FIG3 , the method may include the following steps: Step S302: Capturing a two-dimensional image of a physical object in a real scene. The real scene described above can be set to represent a scene in which the physical object is located in reality, for example, a scene in which the physical object is running, a scene in which the physical object is swimming, a scene in which the physical object is sitting, a scene in which the physical object is standing, etc. The present disclosure does not impose any specific limitations on the real scene.The aforementioned physical objects may include human bodies, as well as other objects with or without life characteristics. The aforementioned two-dimensional image may be an image obtained by photographing the physical object using a camera at a preset perspective. The preset perspective may be a front view, a side view, or a top view. In the disclosed embodiments, the front view perspective is used as an example for illustration. Assuming the physical object is in a three-dimensional coordinate system O-xyz, where x can be set as the coordinate axis representing horizontal left-right position, y can be set as the coordinate axis representing horizontal front-back position, and z can be set as the coordinate axis representing vertical position, the two-dimensional image captured from the front view perspective may be an image in the xy plane, the two-dimensional image captured from the side view perspective may be an image in the xz plane, and the two-dimensional image captured from the top view perspective may be an image in the yz plane. In an optional embodiment, a monocular camera may be used to photograph the physical object in a real scene, and the captured image may be the aforementioned two-dimensional image. For illustration, using a human body standing upright as an example, the captured 2D images can be taken from the front, back, left, right, and overhead perspectives of the human body. Optionally, the captured 2D images can reveal the physical object's outline information, allowing the 3D pose of the physical object to be estimated based on the 2D images. Step S304: Estimate the 2D pose of the physical object in the 2D images to obtain 2D pose features from multiple different perspectives. These multiple perspectives can include any two or more of the following: front view, rear view, left view, right view, top view, and bottom view. These 2D pose features can be configured to represent the pose of the physical object in a 2D coordinate system, i.e., the positions of different joints of the physical object in the 2D coordinate system. In an optional embodiment, after capturing a 2D image of a physical object in a real scene, a 2D pose estimation method known in related art can be used to perform 2D pose estimation on the 2D image of the physical object in the real scene, thereby obtaining 2D pose features for the 2D image at the shooting perspective. Furthermore, 2D pose features for other perspectives can be determined through feature conversion, thereby obtaining 2D pose features for multiple perspectives. Alternatively, after capturing the 2D image, the 2D pose estimation solution can be used to determine the 2D coordinates of different key points on the physical object, thereby obtaining 2D joint point information. Feature extraction can then be used to further determine the 2D pose features for the shooting perspective.Step S306: Estimating the 3D pose of the physical object based on the 2D pose features from multiple different perspectives to obtain a 3D pose result for the physical object. The 3D pose result may be the 3D coordinates of different joint points on the physical object. In an optional embodiment, after obtaining the 2D pose features from multiple different perspectives, the 3D pose result of the physical object may be jointly estimated using these 2D pose features from multiple different perspectives, thereby ensuring a high degree of accuracy in the 3D pose result. Optionally, the 3D pose of the physical object may be estimated by fusing the 2D pose features from different perspectives during the estimation process. Step S308: Based on the 3D pose result, generating a virtual object associated with the physical object in a virtual scene. The virtual scene may be an AR scene, a VR scene, a game scene, or the like. The present disclosure does not impose any specific restrictions on the type of virtual scene. The virtual object associated with the physical object may be a virtual object in an AR scene, a virtual object in a VR scene, a game scene, or the like. The present disclosure does not impose any specific restrictions on the type of virtual object. For illustration, let's take the following example. Assuming the virtual scene is driving a virtual anchor, the physical object is the real anchor from the relevant program, and the virtual object is the virtual anchor, we can first obtain a 2D image of the real anchor captured during the program broadcast and perform 2D pose estimation on this 2D image to obtain 2D pose features from the shooting perspective of the 2D image. Furthermore, through feature conversion, we can determine 2D pose features from other perspectives of the real anchor contained in the 2D image, that is, obtain 2D pose features from multiple perspectives. Optionally, after obtaining the 2D pose features from multiple different perspectives, we can use these 2D pose features from multiple different perspectives to jointly estimate the 3D pose of the real anchor. This 3D pose result can then be used to generate a virtual anchor associated with the real anchor in the driving virtual anchor scene.Assuming that the aforementioned virtual scene is a virtual fitting scene, the physical object is a person fitting, and the virtual object is a virtual person in the virtual fitting scene, a two-dimensional image of the person fitting in the real scene can be first obtained, and two-dimensional pose estimation can be performed on the two-dimensional image to obtain two-dimensional pose features under the shooting perspective of the two-dimensional image. Optionally, two-dimensional pose features of the person fitting contained in the two-dimensional image under other perspectives can be determined through feature conversion, that is, two-dimensional pose features under multiple perspectives can be obtained. Furthermore, the two-dimensional pose features under multiple perspectives can be fused, so that the two-dimensional pose features under multiple perspectives can be used to jointly estimate the three-dimensional pose result of the person fitting, and the three-dimensional pose result can be used to generate a virtual person associated with the person fitting in the virtual fitting scene. Figure 4 is a schematic diagram of a method for generating a virtual object according to an embodiment of the present disclosure. As shown in Figure 4, after capturing a 2D image of a physical object in a real scene, feature extraction can be performed on the 2D image to determine 2D pose features at the viewing angle. Furthermore, cross-perspective fusion can be performed to fuse the 2D pose features at the viewing angle with 2D pose features at other viewing angles determined through feature transformation. A fused result is then generated, and 3D pose estimation is performed based on the fused result, thereby generating a virtual object corresponding to the physical object in the 2D image. In this embodiment of the present disclosure, a 2D image of a physical object in a real scene is captured; 2D pose estimation is performed on the physical object in the 2D image to obtain 2D pose features at multiple different viewing angles; 3D pose estimation is performed on the physical object based on the 2D pose features at multiple different viewing angles to obtain a 3D pose result for the physical object; and based on the 3D pose result, a virtual object associated with the physical object in the virtual scene is generated. It is easy to notice that by estimating the 2D pose of the physical object in the 2D image from multiple perspectives, 2D pose features from multiple different perspectives are obtained. By using the 2D pose features from multiple different perspectives to estimate the 3D pose, it is possible to estimate the 3D pose with high accuracy, thereby generating a more precise virtual object. This improves the accuracy of replicating the physical object based on a single image, thereby solving the technical problem of low accuracy when replicating the physical object based on a single image in the related art.In the above-described embodiments of the present disclosure, performing 2D pose estimation on a physical object in a 2D image to obtain 2D pose results at multiple different perspectives includes: extracting features from the 2D image to obtain image features; estimating the 2D pose of the physical object based on the image features to obtain 2D pose features at a first perspective among the multiple perspectives, wherein the first perspective is set to represent a perspective associated with the 2D image; and converting the 2D pose features at the first perspective based on the image features to obtain 2D pose features at a second perspective among the multiple perspectives, wherein the second perspective is set to represent a perspective other than the first perspective among the multiple perspectives. The first perspective can be set to capture a front view of the physical object, and optionally, the first perspective can include a front view or a back view of the physical object. The second perspective can be set to capture a side view or a top view of the physical object, and optionally, the second perspective can include a left side view or a right side view of the physical object, as well as the perspective in the above example. In an optional embodiment, a backbone network (also known as a backbone) can be used to extract features from a two-dimensional image, or a residual neural network (also known as a ResNet) series model can be used to extract features. Alternatively, a visual transformer (also known as a visual transformer) model can be used to extract features, thereby obtaining image features, which can be represented numerically. In this embodiment, the visual transformer model is used as a feature extractor for illustration. Optionally, the two-dimensional pose of a physical object can be estimated based on these image features. Assuming that the two-dimensional image is captured from a first perspective, feature extraction can be performed on this image to obtain image features. The image features are then used to perform two-dimensional pose estimation, obtaining two-dimensional pose features from the first perspective. Furthermore, the two-dimensional pose features from the first perspective can be transformed to obtain two-dimensional pose features from a second perspective among multiple different perspectives. That is, after estimating the pose features of the physical object in a frontal view using image features, the pose features from the frontal view can be further transformed to obtain pose features from top and top views. In the above-described embodiments of the present disclosure, performing 2D pose estimation on a physical object based on image features to obtain 2D pose features at a first perspective among multiple different perspectives includes: performing 2D pose estimation on the physical object based on the image features to obtain 2D key point information of the physical object, wherein the 2D key point information is set to represent information of different joints of the physical object in the 2D image; and generating 2D pose features at the first perspective based on the 2D key point information.The aforementioned two-dimensional key point information can be set to represent the two-dimensional coordinates of different joints on the body of the physical object in the two-dimensional image. In an optional embodiment, after obtaining the image features, two-dimensional pose estimation can be performed on the physical object based on the image features, thereby estimating the two-dimensional pose of the physical object in the first perspective. Optionally, by performing two-dimensional pose estimation on the physical object, the two-dimensional coordinates of different joints on the physical object can be obtained. A two-dimensional key point heat map can be constructed based on the coordinate positions of the two-dimensional coordinates of the different joints on the physical object, thereby obtaining the two-dimensional pose features of the physical object in the first perspective. The two-dimensional key point heat map can be composed of the two-dimensional key point information. Optionally, the two-dimensional key point heat map can be denoted as Jfmnt € RHWXK, where K is the number of joints. In the above-described embodiments of the present disclosure, converting 2D pose features from a first perspective based on image features to obtain 2D pose features from a second perspective from multiple different perspectives includes: performing convolution processing on the image features to obtain convolution features; and performing cross-attention processing on the convolution features and the 2D pose features from the first perspective to obtain 2D pose features from the second perspective. In an optional embodiment, after obtaining the 2D pose features from the first perspective, cross-view attention (CVA) can be used to transform the 2D pose features from the first perspective to obtain 2D pose features corresponding to the side view and 2D pose features corresponding to the top view. Optionally, convolution calculation can be performed on the image features using CVF to obtain convolution features. The convolution features and the 2D pose features from the first perspective can then be input into a cross-attention block for processing to obtain 2D pose features from the second perspective. The 2D pose features corresponding to the side view can be calculated using the following method: as follows:

[0009] , QK

[0010] CVA = % + MLPgn T Luodan =) Heart

[0011] ViiK • Wherein, Q = MLP(Ffr°nt), K = MLP(F;m), V = MLP(F"Q, χ represents the K dimension, MLP is a multilayer perceptron, and softmax is a normalized exponential function. Optionally, the corresponding two-dimensional posture features in the side view can be obtained from the kernel through multiple linear layers. In the above embodiment of the present disclosure, when the second perspective includes multiple sub-perspectives, cross-attention processing is performed on the convolution features and the two-dimensional posture features under the first perspective to obtain the two-dimensional posture features under the second perspective, including: performing multiple cross-attention processing on the convolution features and the two-dimensional posture features under the first perspective respectively to obtain the two-dimensional posture features under multiple sub-perspectives. The above-mentioned sub-perspectives can be a side view or a top view. In an optional embodiment, if the second perspective includes multiple sub-perspectives, the two-dimensional posture features under the multiple sub-perspectives can be obtained by performing multiple cross-attention processing on the convolution features and the two-dimensional posture features under the first perspective, which is explained by taking the following example. Assume that the two-dimensional image under the first perspective is a front view of the entity object, If the second perspective includes a side view and a top view of the entity object, then feature extraction can be performed on the two-dimensional image from the first perspective to obtain image features. The image features can then be used to estimate the two-dimensional posture features corresponding to the entity object in the front view. Furthermore, the two-dimensional posture features corresponding to the entity object in the front view can be converted into two-dimensional posture features corresponding to the side view and the top view, thereby obtaining two-dimensional posture features from multiple sub-perspectives. In the above-described embodiment of the present disclosure, feature extraction from the two-dimensional image to obtain image features includes: performing block processing on the two-dimensional image to obtain multiple image blocks; performing feature extraction on the multiple image blocks to obtain block features of the multiple image blocks; and performing self-attention and fitting on the block features of the multiple image blocks to obtain image features. The block processing can include segmenting the two-dimensional image to obtain different sub-images. In an optional embodiment, to achieve more accurate feature extraction from the two-dimensional image, the two-dimensional image can be segmented before feature extraction to obtain multiple image blocks. Furthermore, feature extraction can be performed on the multiple image blocks.Optionally, multiple image blocks can be processed by different Transformer layers. Any Transformer layer consists of a multi-head self-attention layer (MHSA) and a feed-forward network (FFN). The following formula can be used to calculate the output of any Transformer layer. Among them, LN represents layer normalization, ∂ represents the output of the i-th Transformer layer, and ∂=pa[chEmbed(X)] is the output feature of the block embedding layer. The output feature of the Transformer model can be recorded as In the above-described embodiments of the present disclosure, performing 3D pose estimation on a physical object based on 2D pose features from multiple different perspectives to obtain a 3D pose result of the physical object includes: superimposing the 2D pose features from multiple different perspectives to obtain a fused feature; and performing 3D pose estimation on the physical object based on the fused feature to obtain a 3D pose result. The 3D pose result may be an optimized pose of the physical object. In an optional embodiment, after obtaining the 2D pose features of the physical object from multiple different perspectives, the 2D pose features from multiple different perspectives may be superimposed to obtain a fused feature. Optionally, the fused feature may be used to perform 3D pose estimation on the physical object. Optionally, because the features of 2D images from different perspectives are inconsistent, a cross-view fusion (CVF) method may be employed to fuse the 2D pose features of 2D images from different perspectives to obtain the 3D features of the physical object. That is, the 2D pose features from multiple different perspectives are superimposed to obtain the fused feature. In the above-described embodiments of the present disclosure, generating a virtual object associated with a physical object in a virtual scene based on a 3D pose result includes: superimposing 2D pose features from multiple different perspectives to obtain fused features; decoding the fused features to obtain camera parameters and a virtual mesh corresponding to the physical object; and reconstructing the virtual object based on the camera parameters, the virtual mesh, and the 3D pose result. In an optional embodiment, a Reversible Kinematic Topology Decoder (RKTD) can be used to decode the fused features to obtain the camera parameters and a virtual mesh corresponding to the physical object. Reconstructing the virtual object based on the camera parameters, the virtual mesh, and the 3D pose result is then possible. Optionally, the fused features can be decoded using a regressor consisting of a fully connected layer consisting of pixel-set background modeling and a foreground detection algorithm (Visual Background Extraction, or VIBE). Optionally, the pose of the physical object can be input into the parameterized human body model SMPL. 24x3 , and the shape P of the entity object e, thus outputting a mesh of a 3D solid object with vertices V = . Furthermore, a virtual object can be generated by reconstructing it using the camera parameters, virtual mesh, and 3D pose results. The pose of the solid object consists of the global rotation between the solid object's joints and 23 local rotations of the response key relative to the parent joint along the kinematic tree. K three-dimensional key points can be calculated through a linear combination of vertices using the following formula: d = M", where M € Rkx6890. FIG5 is a schematic diagram of a method for generating a virtual object according to an embodiment of the present disclosure. As shown in FIG5, a two-dimensional image is taken as an example for description. After capturing the two-dimensional image, a convolution operation can be performed on the backbone features of the two-dimensional image to obtain the key points of the two-dimensional image. Optionally, a key point heat map 1 can be generated based on the key points of the two-dimensional image, that is, a key point heat map corresponding to the front view, to obtain the two-dimensional posture features under the first view. Furthermore, while performing a convolution operation on the backbone features extracted from the two-dimensional image, a query block can be used to query the key points of the two-dimensional image. The convolution result and the query result are subjected to cross-attention processing to convert them into two-dimensional posture features under the second view. Optionally, if the second view includes multiple sub-views, multiple cross-attention processes are required to obtain two-dimensional posture features corresponding to the side view and two-dimensional posture features corresponding to the top view. Optionally, The corresponding key point heat map 2 and key point heat map 3 can be generated by linear layer and sigmoid activation function, where key point heat map 2 and The steps illustrated in the flowchart of FIG6 can be executed in a computer system, such as a set of computer-executable instructions. Although the flowchart illustrates a logical order, in some cases, the steps illustrated or described may be executed in a different order. FIG6 is a flowchart of a method for generating a virtual object according to Embodiment 2 of the present disclosure. As shown in FIG6 , the method includes the following steps: Step S602: In response to an input instruction applied to an operating interface, displaying a two-dimensional image on the operating interface, wherein the two-dimensional image is an image obtained by photographing a physical object in a real scene. The operating interface can be a mobile phone interface or a tablet computer interface, and the present disclosure does not impose any specific restrictions on the type of operating interface. The instruction can be a display instruction, configured to instruct the operating interface to display a two-dimensional image obtained by photographing a physical object in a real scene. Optionally, the instruction can be expressed through gestures or voice input, and the present disclosure does not impose any specific restrictions on the form of the input instruction. Step S604: In response to a generation instruction applied to the operation interface, a virtual object associated with the physical object in the virtual scene is displayed on the operation interface. The virtual object is generated based on the 3D pose result of the physical object. The 3D pose result is obtained by estimating the 3D pose of the physical object based on 2D pose features from multiple different perspectives. The 2D pose features from multiple different perspectives are obtained by estimating the 2D pose of the physical object in a 2D image. In an optional embodiment, when a user inputs a display instruction on the operation interface, the operation interface can respond to the instruction by retrieving a 2D image of the physical object captured from a real scene from the system and displaying the 2D image on the operation interface. Alternatively, the user can indicate the generation instruction on the operation interface through gestures or voice input. After receiving the instruction, the operation interface can generate a virtual object associated with the physical object in the virtual scene based on the 3D pose result of the physical object and display the virtual object on the operation interface. Embodiment 3: According to an embodiment of the present disclosure, a method for generating a virtual object in a virtual reality scenario, such as a virtual reality (VR) device or an augmented reality (AR) device, is also provided. It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system, such as a set of computer-executable instructions. Furthermore, although the flowcharts illustrate a logical sequence, in some cases, the steps shown or described may be executed in a different order. Figure 7 is a flowchart of the method for generating a virtual object according to Embodiment 3 of the present disclosure.As shown in FIG3 , the method may include the following steps: Step S702: Displaying a two-dimensional image on a presentation screen of a virtual reality (VR) device or an augmented reality (AR) device, wherein the two-dimensional image is obtained by photographing a physical object in a real scene. Step S704: Estimating the two-dimensional pose of the physical object in the two-dimensional image to obtain two-dimensional pose features from multiple different perspectives. Step S706: Estimating the three-dimensional pose of the physical object based on the two-dimensional pose features from multiple different perspectives to obtain three-dimensional pose results for the physical object. Step S708: Generating a virtual object associated with the physical object in the virtual scene based on the three-dimensional pose results. Step S710: Driving the VR device or AR device to render and display the virtual object. In an optional embodiment, a physical object in a real scene can be photographed and displayed on a presentation screen of a virtual reality (VR) device or an augmented reality (AR) device. Feature extraction can be performed on the 2D image using the method in Example 1 to obtain image features. These image features are then used to perform 2D pose estimation, thereby obtaining 2D pose features at a first perspective. These 2D pose features at the first perspective can then be converted to 2D pose features at a second perspective. Based on the 2D pose features at multiple different perspectives, 3D pose estimation can be performed on the physical object to obtain a 3D pose result for the physical object. Furthermore, the 3D pose result for the physical object can be used to generate a virtual object associated with the physical object in a virtual scene. Optionally, in this embodiment, the aforementioned virtual object generation method can be applied to a hardware environment consisting of a server and a virtual reality device. The virtual object generation method is displayed on a presentation screen of a virtual reality (VR) device or an augmented reality (AR) device. The server can be a server corresponding to a media file operator. The aforementioned network includes, but is not limited to, a wide area network, a metropolitan area network, or a local area network. The aforementioned virtual reality device is not limited to a virtual reality helmet, virtual reality glasses, or an all-in-one virtual reality device. Optionally, a virtual reality device includes: a memory, a processor, and a transmission device. The memory is configured to store an application program, and the application program may be configured to: display a two-dimensional image on a presentation screen of a virtual reality (VR) device or an augmented reality (AR) device, wherein the two-dimensional image is a photograph of a physical object in a real scene; perform two-dimensional pose estimation on the physical object in the two-dimensional image to obtain two-dimensional pose features under multiple different perspectives; perform three-dimensional pose estimation on the physical object based on the two-dimensional pose features under the multiple different perspectives to obtain three-dimensional pose results of the physical object; generate a virtual object associated with the physical object in a virtual scene based on the three-dimensional pose results; and drive the VR device or AR device to render and display the virtual object.It should be noted that the method for generating a virtual object in a VR or AR device in this embodiment may include the method of the embodiment shown in FIG. 7 , to achieve the purpose of driving the VR or AR device to display a virtual object. Optionally, the processor in this embodiment may invoke an application stored in the memory via a transmission device to execute the above steps. The transmission device may receive media files sent by a server via a network and may also be used for data transmission between the processor and the memory. Optionally, in a virtual reality device, a head-mounted display with eye tracking may include a plane in the HMD configured to display a displayed video image, an eye tracking module in the HMD configured to obtain a real-time eye movement trajectory of a user, a tracking system configured to track the user's position and motion information in real three-dimensional space, and a computing processing unit configured to obtain the user's real-time position and motion information from the tracking system and calculate the three-dimensional coordinates of the user's head in the virtual three-dimensional space, as well as the user's field of view in the virtual three-dimensional space. In embodiments of the present disclosure, a virtual reality device can be connected to a terminal, and the terminal and server are connected via a network. The virtual reality device is not limited to a virtual reality helmet, virtual reality glasses, or all-in-one virtual reality device. The terminal is not limited to a PC, mobile phone, or tablet computer. The server can be a server corresponding to a media file provider. The network includes, but is not limited to, a wide area network, a metropolitan area network, or a local area network. Example 4: According to another aspect of the embodiments of the present disclosure, another method for generating a virtual object is provided. FIG8 is a flowchart of the method for generating a virtual object according to Example 4 of the present disclosure. As shown in FIG4 , the method may include the following steps: Step S802: Acquire a two-dimensional image by calling a first interface, wherein the first interface includes a first parameter, and the parameter value of the first parameter is a two-dimensional image, where the two-dimensional image is an image obtained by photographing a physical object in a real scene. Step S804: Perform two-dimensional pose estimation on the physical object in the two-dimensional image to obtain two-dimensional pose features at multiple different perspectives. Step S806: Perform three-dimensional pose estimation on the physical object based on the two-dimensional pose features at multiple different perspectives to obtain a three-dimensional pose result of the physical object. Step S808: Generate a virtual object associated with the physical object in the virtual scene based on the 3D pose result. Step S810: Output the virtual object by calling a second interface, wherein the second interface includes a second parameter whose parameter value is the virtual object. The first interface and the second interface may be network interfaces for data transmission.In an optional embodiment, a first interface can be used to obtain a two-dimensional image obtained by photographing a physical object in a real scene. Feature extraction can be performed on the two-dimensional image using the method described in Example 1 to obtain image features. These image features are then used to perform two-dimensional pose estimation, thereby obtaining two-dimensional pose features at a first perspective. The two-dimensional pose features at the first perspective can then be converted to two-dimensional pose features at a second perspective. Three-dimensional pose estimation can then be performed on the physical object based on the two-dimensional pose features at multiple different perspectives, resulting in a three-dimensional pose result for the physical object. Furthermore, the three-dimensional pose result for the physical object can be used to generate a virtual object associated with the physical object in a virtual scene. Optionally, after generating the virtual object associated with the physical object, the virtual object associated with the physical object can be output via the second interface. Example 5: According to another aspect of the present disclosure, a method for processing three-dimensional poses is provided. FIG9 is a flowchart of the method for processing three-dimensional poses according to Example 5 of the present disclosure. As shown in FIG9 , the method may include the following steps: Step S902: Capturing a two-dimensional image obtained by photographing a physical person in a real scene. Step S904: Perform 2D pose estimation on the physical person in the 2D image to obtain 2D pose features from multiple different perspectives. Step S906: Perform 3D pose estimation on the physical person based on the 2D pose features from multiple different perspectives to obtain a 3D pose result of the physical person. Step S908: Match the 3D pose result with a preset 3D pose to obtain a processing result of the 3D pose result, where the processing result is configured to indicate whether the 3D pose result matches the preset 3D pose. The preset 3D pose may be a standard pose for different scenarios, such as a standard pose for a running start or a standard pose for a dance movement. In an optional embodiment, after obtaining the 3D pose result of the physical person, the obtained 3D pose result of the physical person may be matched with the preset 3D pose to determine whether the 3D pose result of the physical person successfully matches the preset 3D pose. If the 3D pose result of the physical person successfully matches the preset 3D pose, the 3D pose result of the physical person may be considered accurate.Embodiment 6 According to an embodiment of the present disclosure, a device for generating a virtual object configured to implement the above-mentioned embodiment 1 is further provided. FIG10 is a schematic diagram of the device for generating a virtual object according to embodiment 1 of the present disclosure. As shown in FIG10 , the device includes: a capture module 1002, a first estimation module 1004, a second estimation module 1006, and a generation module 1008. The capture module 1002 is configured to capture a two-dimensional image obtained by photographing a physical object in a real scene. The first estimation module 1004 is configured to perform two-dimensional pose estimation on the physical object in the two-dimensional image to obtain two-dimensional pose features under multiple different perspectives. The second estimation module 1006 is configured to perform three-dimensional pose estimation on the physical object based on the two-dimensional pose features under multiple different perspectives to obtain a three-dimensional pose result of the physical object. The generation module 1008 is configured to generate a virtual object associated with the physical object in the virtual scene based on the three-dimensional pose result. It should be noted that the capture module 1002, the first estimation module 1004, the second estimation module 1006, Generation module 1008 corresponds to steps S302 to S308 in Example 1. The examples and application scenarios implemented by these four modules and corresponding steps are the same, but are not limited to the content disclosed in Example 1. In the above embodiment of the present disclosure, first estimation module 1004 includes: an extraction unit configured to extract features from a two-dimensional image to obtain image features; an estimation unit configured to estimate the two-dimensional pose of a physical object based on the image features to obtain two-dimensional pose features at a first perspective among multiple different perspectives, where the first perspective is configured to represent a perspective associated with the two-dimensional image; a conversion unit configured to convert the two-dimensional pose features at the first perspective based on the image features to obtain two-dimensional pose features at a second perspective among multiple different perspectives, where the second perspective is configured to represent a perspective other than the first perspective among the multiple different perspectives. In the above embodiment of the present disclosure, the estimation unit includes: an estimation subunit configured to estimate the two-dimensional pose of a physical object based on the image features to obtain two-dimensional key point information of the physical object, where the two-dimensional key point information is configured to represent information about different joints of the physical object in the two-dimensional image; The generation subunit is configured to generate two-dimensional posture features at a first perspective based on the two-dimensional keypoint information. In the above embodiment of the present disclosure, the conversion unit includes: a first processing subunit configured to perform convolution processing on the image features to obtain convolution features; and a second processing subunit configured to perform cross-attention processing on the convolution features and the two-dimensional posture features at the first perspective to obtain two-dimensional posture features at the second perspective.In the above embodiment of the present disclosure, the second processing sub-unit is further configured to perform multiple cross-attention processes on the convolution features and the two-dimensional pose features from the first view, respectively, to obtain two-dimensional pose features from multiple sub-views. In the above embodiment of the present disclosure, the extraction unit includes: a third processing sub-unit configured to perform block processing on the two-dimensional image to obtain multiple image blocks; an extraction sub-unit configured to perform feature extraction on the multiple image blocks to obtain block features of the multiple image blocks; and a fourth processing sub-unit configured to perform self-attention and fitting processes on the block features of the multiple image blocks to obtain image features. In the above embodiment of the present disclosure, the second estimation module 1006 includes: a first superposition unit configured to superimpose 2D posture features from multiple different perspectives to obtain a fused feature; a prediction unit configured to predict the contact state of the physical object based on the fused feature to obtain a part contact state corresponding to a first part of the physical object, wherein the part contact state is configured to indicate whether there is a gap between the first part and a preset plane in the real scene; a traversal unit configured to, when the contact state indicates that there is no gap between the first part and the preset plane, reversely traverse the kinematic tree based on the contact state to determine the postures of different key points between the first part and the second part, thereby obtaining a 3D posture result, wherein the second part is a part of the physical object whose distance from the first part is greater than a preset distance. In the above embodiment of the present disclosure, the prediction unit includes: a prediction subunit configured to predict the contact state of different vertices on the first part based on the fused feature to obtain vertex contact states of the different vertices; and a superposition subunit configured to superimpose the vertex contact states of the different vertices to obtain the part contact state corresponding to the first part. In the above embodiment of the present disclosure, the apparatus further includes: a first determination module configured to determine the number of target vertices on the multiple sub-parts based on the part contact states of the multiple sub-parts, wherein the vertex contact state of the target vertex is that there is no gap between the target vertex and a predetermined plane; a first determination module configured to determine a target sub-part among the multiple sub-parts based on the number of target vertices on the multiple sub-parts; and an acquisition module configured to acquire a kinematic tree corresponding to the target sub-part. In the above embodiment of the present disclosure, the generation module 1008 includes: a second superposition unit configured to superimpose two-dimensional posture features from multiple different perspectives to obtain a fused feature; and an estimation unit configured to perform three-dimensional posture estimation of the physical object based on the fused feature to obtain a three-dimensional posture result.Example 7 According to an embodiment of the present disclosure, a device for generating a virtual object configured to implement the aforementioned Example 2 is also provided. FIG11 is a schematic diagram of the device for generating a virtual object according to Example 2 of the present disclosure. As shown in FIG11 , the device includes a first display module 1102 and a second display module 1104. The first display module 1102 is configured to display a two-dimensional image on the operation interface in response to an input instruction applied to the operation interface. The two-dimensional image is an image obtained by photographing a physical object in a real scene. The second display module 1104 is configured to display a virtual object associated with the physical object in a virtual scene on the operation interface in response to a generation instruction applied to the operation interface. The virtual object is generated based on a three-dimensional pose result of the physical object. The three-dimensional pose result is obtained by performing three-dimensional pose estimation on the physical object based on two-dimensional pose features at multiple different perspectives. The two-dimensional pose features at multiple different perspectives are obtained by performing two-dimensional pose estimation on the physical object in the two-dimensional image. It should be noted that the first display module 1102 and the second display module 1104 correspond to steps S602 to S604 in Example 2. The examples and application scenarios implemented by these two modules and the corresponding steps are the same, but are not limited to the contents disclosed in Example 2. Example 8 According to an embodiment of the present disclosure, a device for generating a virtual object configured to implement Example 3 is also provided. FIG12 is a schematic diagram of the device for generating a virtual object according to Example 3 of the present disclosure. As shown in FIG12 , the device includes a display module 1202, a first estimation module 1204, a second estimation module 1206, a generation module 1208, and a driving module 1210. Among them, the display module 1202 is configured to display a two-dimensional image on a presentation screen of a virtual reality (VR) device or an augmented reality (AR) device, where the two-dimensional image is an image obtained by photographing a physical object in a real scene; the first estimation module 1204 is configured to perform two-dimensional pose estimation on the physical object in the two-dimensional image to obtain two-dimensional pose features under multiple different perspectives; the second estimation module 1206 is configured to perform three-dimensional pose estimation on the physical object based on the two-dimensional pose features under multiple different perspectives to obtain a three-dimensional pose result of the physical object; the generation module 1208 is configured to generate a virtual object associated with the physical object in the virtual scene based on the three-dimensional pose result; and the driving module 1210 is configured to drive the VR device or AR device to render and display the virtual object.It should be noted that the display module 1202, first estimation module 1204, second estimation module 1206, generation module 1208, and driver module 1210 described above correspond to steps S702 to S710 in Example 3. These five modules and the corresponding steps implement the same examples and application scenarios, but are not limited to those disclosed in Example 3. It should be noted that the above modules or units may be hardware or software components stored in a memory and processed by one or more processors. These modules may also be part of an apparatus that can run in the AR / VR device provided in Example 3. Example 9: According to an embodiment of the present disclosure, a virtual object generation device configured to implement the above-described Example 4 is also provided. Figure 13 is a schematic diagram of the virtual object generation device according to Example 4 of the present disclosure. As shown in Figure 13, the device includes a calling module 1302, a first estimation module 1304, a second estimation module 1306, a generation module 1308, and an output module 1310. The calling module 1302 is configured to acquire a two-dimensional image by calling a first interface, wherein the first interface includes a first parameter, the parameter value of the first parameter is a two-dimensional image, and the two-dimensional image is an image obtained by photographing a physical object in a real scene; the first estimation module 1304 is configured to perform two-dimensional pose estimation on the physical object in the two-dimensional image to obtain two-dimensional pose features under multiple different perspectives; the second estimation module 1306 is configured to perform three-dimensional pose estimation on the physical object based on the two-dimensional pose features under the multiple different perspectives to obtain a three-dimensional pose result of the physical object; the generation module 1308 is configured to generate a virtual object associated with the physical object in a virtual scene based on the three-dimensional pose result; and the output module 1310 is configured to output the virtual object by calling a second interface, wherein the second interface includes a second parameter, the parameter value of the second parameter is the virtual object. It should be noted that the aforementioned calling module 1302, first estimation module 1304, second estimation module 1306, generation module 1308, and output module 1310 correspond to steps S802 to S810 in Example 4. These five modules and the corresponding steps implement the same examples and application scenarios, but are not limited to those disclosed in Example 4. Example 10 According to an embodiment of the present disclosure, a 3D gesture processing device configured to implement the 5th embodiment is also provided. Figure 14 is a schematic diagram of the 3D gesture processing device according to Example 5 of the present disclosure. As shown in Figure 14 , the device includes a capture module 1402, a first estimation module 1404, a second estimation module 1406, and a matching module 1408.The capture module 1402 is configured to capture a 2D image of a real person in a real scene; the first estimation module 1404 is configured to perform 2D pose estimation on the real person in the 2D image, obtaining 2D pose features from multiple different perspectives; the second estimation module 1406 is configured to perform 3D pose estimation on the real person based on the 2D pose features from multiple different perspectives, obtaining a 3D pose result of the real person; and the matching module 1408 is configured to match the 3D pose result with a preset 3D pose, obtaining a processed result of the 3D pose result, wherein the processed result is configured to indicate whether the 3D pose result matches the preset 3D pose. It should be noted that the capture module 1402, the first estimation module 1404, the second estimation module 1406, and the matching module 1408 correspond to steps S902 to S908 in Example 5. The examples and application scenarios implemented by these four modules and the corresponding steps are the same, but are not limited to the content disclosed in Example 5. Example 11: An embodiment of the present disclosure may provide a system for generating a virtual object. The computer terminal may include an AR / VR device, a server, and a client. The AR / VR device may be any one of a group of AR / VR devices. Optionally, the computer terminal device includes: a processor; and a memory connected to the processor and configured to provide the processor with instructions for performing the following processing steps: capturing a two-dimensional image of a physical object in a real scene; performing two-dimensional pose estimation on the physical object in the two-dimensional image to obtain two-dimensional pose features from multiple different perspectives; performing three-dimensional pose estimation on the physical object based on the two-dimensional pose features from the multiple different perspectives to obtain three-dimensional pose results of the physical object; and generating a virtual object associated with the physical object in a virtual scene based on the three-dimensional pose results. Example 12: An embodiment of the present disclosure may provide an AR / VR device. The AR / VR device may be any one of a group of AR / VR devices. Optionally, in this embodiment, the AR / VR device may be replaced with a terminal device such as a mobile terminal. Optionally, in this embodiment, the AR / VR device may be located in at least one network device among multiple network devices of a computer network.In this embodiment, the AR / VR device can execute the program code for the following steps in the virtual object generation method: capturing a two-dimensional image obtained by photographing a physical object in a real scene; performing two-dimensional pose estimation on the physical object in the two-dimensional image to obtain two-dimensional pose features from multiple different perspectives; performing three-dimensional pose estimation on the physical object based on the two-dimensional pose features from multiple different perspectives to obtain three-dimensional pose results of the physical object; and generating a virtual object associated with the physical object in the virtual scene based on the three-dimensional pose results. Optionally, Figure 15 is a block diagram of a computer terminal according to an embodiment of the present disclosure. As shown in Figure 15, the computer terminal A may include one or more (one shown) processors 1502, a memory 1504, a storage controller, and a peripheral interface, wherein the peripheral interface is connected to a radio frequency module, an audio module, and a display. The memory may be configured to store software programs and modules, such as program instructions / modules corresponding to the virtual object generation and apparatus in the embodiments of the present disclosure. The processor executes the software programs and modules stored in the memory to perform various functional applications and data processing, thereby achieving the aforementioned virtual object generation. The memory may include high-speed random access memory (RAM) and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include memory located remotely from the processor, which may be connected to terminal A via a network. Examples of such networks include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof. The processor may access information and applications stored in the memory via a transmission device to perform the following steps: capturing a two-dimensional image of a physical object in a real scene; performing two-dimensional pose estimation on the physical object in the two-dimensional image to obtain two-dimensional pose features from multiple different perspectives; performing three-dimensional pose estimation on the physical object based on the two-dimensional pose features from multiple different perspectives to obtain a three-dimensional pose result of the physical object; and generating a virtual object associated with the physical object in a virtual scene based on the three-dimensional pose result.Optionally, the processor may further execute program code for the following steps: performing 2D pose estimation on the physical object in the 2D image to obtain 2D pose results at multiple different viewing angles, including: performing feature extraction on the 2D image to obtain image features; performing 2D pose estimation on the physical object based on the image features to obtain 2D pose features at a first viewing angle among the multiple different viewing angles, wherein the first viewing angle is set to represent a viewing angle associated with the 2D image; and converting the 2D pose features at the first viewing angle based on the image features to obtain 2D pose features at a second viewing angle among the multiple different viewing angles, wherein the second viewing angle is set to represent a viewing angle other than the first viewing angle among the multiple different viewing angles. Optionally, the processor may further execute program code for the following steps: performing 2D pose estimation on the physical object based on the image features to obtain 2D pose features at a first viewing angle among the multiple different viewing angles, including: performing 2D pose estimation on the physical object based on the image features to obtain 2D key point information of the physical object, wherein the 2D key point information is set to represent information of different joints of the physical object in the 2D image; and generating 2D pose features at the first viewing angle based on the 2D key point information. Optionally, the processor may further execute program code for the following steps: converting the two-dimensional gesture features at the first perspective based on the image features to obtain two-dimensional gesture features at a second perspective from multiple different perspectives, including: performing convolution processing on the image features to obtain convolution features; and performing cross-attention processing on the convolution features and the two-dimensional gesture features at the first perspective to obtain two-dimensional gesture features at the second perspective. Optionally, the processor may further execute program code for the following steps: performing cross-attention processing on the convolution features and the two-dimensional gesture features at the first perspective to obtain two-dimensional gesture features at the second perspective when the second perspective includes multiple sub-perspectives, including: performing cross-attention processing on the convolution features and the two-dimensional gesture features at the first perspective multiple times to obtain two-dimensional gesture features at the multiple sub-perspectives. Optionally, the processor may further execute program code for the following steps: extracting features from a two-dimensional image to obtain image features, including: dividing the two-dimensional image into blocks to obtain multiple image blocks; extracting features from the multiple image blocks to obtain block features of the multiple image blocks; and performing self-attention and fitting on the block features of the multiple image blocks to obtain image features. Optionally, the processor may further execute program code for the following steps: estimating the three-dimensional pose of a physical object based on two-dimensional pose features from multiple different perspectives to obtain a three-dimensional pose result of the physical object, including: superimposing the two-dimensional pose features from the multiple different perspectives to obtain a fused feature; and estimating the three-dimensional pose of the physical object based on the fused feature to obtain a three-dimensional pose result.In the disclosed embodiments, a 2D image of a physical object in a real scene is captured; 2D pose estimation is performed on the physical object in the 2D image to obtain 2D pose features from multiple different perspectives; 3D pose estimation is performed on the physical object based on the 2D pose features from multiple perspectives to obtain a 3D pose result for the physical object; and based on the 3D pose result, a virtual object associated with the physical object in a virtual scene is generated. It is readily apparent that by performing 2D pose estimation on the physical object in the 2D image from multiple perspectives, 2D pose features from multiple perspectives are obtained. By using these 2D pose features from multiple perspectives to perform 3D pose estimation, a highly accurate 3D pose can be estimated, thereby enabling the generation of a more precise virtual object. This improves the accuracy of replicating a physical object from a single image, thereby resolving the technical issue of low accuracy in replicating a physical object from a single image in related technologies. Those skilled in the art will appreciate that the structure shown in FIG15 is for illustration only, and the computer terminal may also be a smartphone (such as an Android phone, an iOS phone, etc.), a tablet computer, a PDA, a mobile internet device (MID), a PAD, or other terminal device. FIG15 does not limit the structure of the aforementioned electronic devices. For example, computer terminal A may include more or fewer components (such as a network interface, a display device, etc.), or have a configuration different from that shown in FIG15 . Those skilled in the art will appreciate that all or part of the steps in the various methods of the aforementioned embodiments can be completed by a program instructing the hardware associated with the terminal device. The program may be stored in a computer-readable storage medium, which may include a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk. Example 13 The present disclosure also provides a computer-readable storage medium. Optionally, in this embodiment, the computer-readable storage medium may be configured to store the program code executed by the virtual object generation method provided in Example 1. Optionally, in this embodiment, the computer-readable storage medium may be located in any computer terminal in an AR / VR device terminal group in an AR / VR device network, or in any mobile terminal in a mobile terminal group.Optionally, in this embodiment, the computer-readable storage medium is configured to store program code configured to execute the following steps: performing two-dimensional pose estimation on a physical object in a two-dimensional image to obtain two-dimensional pose results at multiple different perspectives, including: performing feature extraction on the two-dimensional image to obtain image features; performing two-dimensional pose estimation on the physical object based on the image features to obtain two-dimensional pose features at a first perspective among the multiple different perspectives, wherein the first perspective is configured to represent a perspective associated with the two-dimensional image; and converting the two-dimensional pose features at the first perspective based on the image features to obtain two-dimensional pose features at a second perspective among the multiple different perspectives, wherein the second perspective is configured to represent a perspective other than the first perspective among the multiple different perspectives. Optionally, the processor may further execute program code to perform the following steps: performing two-dimensional pose estimation on the physical object based on the image features to obtain two-dimensional pose features at a first perspective among the multiple different perspectives, including: performing two-dimensional pose estimation on the physical object based on the image features to obtain two-dimensional key point information of the physical object, wherein the two-dimensional key point information is configured to represent information of different joints of the physical object in the two-dimensional image; and generating two-dimensional pose features at the first perspective based on the two-dimensional key point information. Optionally, the processor may further execute program code for the following steps: converting the two-dimensional gesture features at the first viewing angle based on the image features to obtain two-dimensional gesture features at a second viewing angle from multiple different viewing angles, including: performing convolution processing on the image features to obtain convolution features; and performing cross-attention processing on the convolution features and the two-dimensional gesture features at the first viewing angle to obtain two-dimensional gesture features at the second viewing angle. Optionally, the processor may further execute program code for the following steps: performing cross-attention processing on the convolution features and the two-dimensional gesture features at the first viewing angle to obtain two-dimensional gesture features at the second viewing angle when the second viewing angle includes multiple sub-views, including: performing cross-attention processing on the convolution features and the two-dimensional gesture features at the first viewing angle multiple times to obtain two-dimensional gesture features at the multiple sub-views. Optionally, the processor may further execute program code for the following steps: performing feature extraction on a two-dimensional image to obtain image features, including: performing block processing on the two-dimensional image to obtain multiple image blocks; performing feature extraction on the multiple image blocks to obtain block features of the multiple image blocks; and performing self-attention processing and fitting processing on the block features of the multiple image blocks to obtain image features.Optionally, the processor may further execute program code for the following steps: performing three-dimensional pose estimation on a physical object based on two-dimensional pose features from multiple different perspectives to obtain a three-dimensional pose result for the physical object, including: superimposing the two-dimensional pose features from multiple different perspectives to obtain a fused feature; and performing three-dimensional pose estimation on the physical object based on the fused feature to obtain a three-dimensional pose result. The serial numbers of the embodiments of the present disclosure are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments. In the above embodiments of the present disclosure, the description of each embodiment has its own emphasis. For portions not described in detail in a particular embodiment, reference should be made to the relevant descriptions of other embodiments. It should be understood that the disclosed technical content can be implemented in other ways in the several embodiments provided in the present disclosure. The device embodiments described above are illustrative only. For example, the division of the units described is a logical functional division. In actual implementation, other divisions may be used. For example, multiple units or components may be combined or integrated into another system, or some features may be omitted or not implemented. Furthermore, the coupling, direct coupling, or communication connection shown or discussed may be through interfaces, indirect coupling, or communication connection between units or modules, and may be electrical or other forms. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the objectives of the present embodiment as needed. Furthermore, the functional units in the various embodiments of the present disclosure may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. These integrated units may be implemented in either hardware or software functional units. If these integrated units are implemented as software functional units and sold or used as independent products, they may be stored in a computer-readable storage medium. Based on this understanding, the essence of the technical solution of the present disclosure, or the part that contributes to the existing technology, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present disclosure.The aforementioned storage media include various media capable of storing program code, such as USB flash drives, read-only memories (ROMs), random access memories (RAMs), removable hard drives, magnetic disks, or optical disks. The above is a preferred embodiment of the present disclosure. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present disclosure, and such improvements and modifications should also be considered within the scope of protection of the present disclosure. Industrial Applicability: The embodiments of the present disclosure provide a method for generating a virtual object, a method for processing a three-dimensional pose, and an electronic device. These methods capture a two-dimensional image of a physical object in a real scene; perform two-dimensional pose estimation on the physical object in the two-dimensional image to obtain two-dimensional pose features from multiple different perspectives; perform three-dimensional pose estimation on the physical object based on the two-dimensional pose features from multiple different perspectives to obtain a three-dimensional pose result for the physical object; and generate a virtual object associated with the physical object in a virtual scene based on the three-dimensional pose result. It is easy to notice that by estimating the 2D pose of the physical object in the 2D image from multiple perspectives, 2D pose features from multiple different perspectives are obtained. By using the 2D pose features from multiple different perspectives to estimate the 3D pose, it is possible to estimate the 3D pose with high accuracy, thereby generating a more precise virtual object. This improves the accuracy of replicating the physical object based on a single image, thereby solving the technical problem of low accuracy when replicating the physical object based on a single image in the related art.

Claims

Claims 1. A method for generating a virtual object, comprising: Capturing two-dimensional images of physical objects in real scenes; Performing a two-dimensional posture estimation on the entity object in the two-dimensional image to obtain two-dimensional posture features under multiple different viewing angles; performing a three-dimensional posture estimation on the entity object based on the two-dimensional posture features under the multiple different viewing angles to obtain a three-dimensional posture result of the entity object; Based on the three-dimensional posture result, a virtual object associated with the physical object in the virtual scene is generated.

2. The method according to claim 1, wherein: Performing two-dimensional posture estimation on the entity object in the two-dimensional image to obtain two-dimensional posture results at multiple different viewing angles includes: performing feature extraction on the two-dimensional image to obtain image features; performing two-dimensional posture estimation on the entity object based on the image features to obtain two-dimensional posture features at a first viewing angle among the multiple different viewing angles, wherein the first viewing angle is used to characterize a viewing angle associated with the two-dimensional image; and converting the two-dimensional posture features at the first viewing angle based on the image features to obtain two-dimensional posture features at a second viewing angle among the multiple different viewing angles, wherein the second viewing angle is used to characterize a viewing angle other than the first viewing angle among the multiple different viewing angles.

3. The method according to claim 2, wherein: Performing two-dimensional posture estimation on the entity object based on the image features to obtain two-dimensional posture features at a first perspective among the multiple different perspectives, including: performing two-dimensional posture estimation on the entity object based on the image features to obtain two-dimensional key point information of the entity object, wherein the two-dimensional key point information is used to characterize information of different joints on the entity object in the two-dimensional image; and generating two-dimensional posture features at the first perspective based on the two-dimensional key point information.

4. The method according to claim 2, wherein: The two-dimensional posture feature at the first viewing angle is converted based on the image feature to obtain the two-dimensional posture feature at the second viewing angle among the multiple different viewing angles, including: performing convolution processing on the image feature to obtain convolution feature; and performing cross-attention processing on the convolution feature and the two-dimensional posture feature at the first viewing angle to obtain the two-dimensional posture feature at the second viewing angle.

5. The method according to claim 4, wherein: In the case where the second perspective includes multiple sub-perspectives, cross-attention processing is performed on the convolution feature and the two-dimensional posture feature under the first perspective to obtain the two-dimensional posture feature under the second perspective, including: performing multiple cross-attention processing on the convolution feature and the two-dimensional posture feature under the first perspective respectively to obtain the two-dimensional posture features under the multiple sub-perspectives.

6. The method according to claim 2, wherein: Extracting features from the two-dimensional image to obtain image features includes: performing block processing on the two-dimensional image to obtain multiple image blocks; extracting features from the multiple image blocks to obtain block features of the multiple image blocks; and performing self-attention processing and fitting processing on the block features of the multiple image blocks to obtain the image features.

7. The method according to claim 1, wherein: The three-dimensional posture estimation of the entity object is performed based on the two-dimensional posture features under the multiple different perspectives to obtain a three-dimensional posture result of the entity object, including: superimposing the two-dimensional posture features under the multiple different perspectives to obtain a fusion feature; and performing three-dimensional posture estimation of the entity object based on the fusion feature to obtain the three-dimensional posture result.

8. The method according to claim 1, wherein: Based on the three-dimensional posture result, generating a virtual object associated with the physical object in a virtual scene, including: superimposing the two-dimensional posture features under the multiple different perspectives to obtain a fusion feature; The fusion feature is decoded to obtain camera parameters and a virtual grid corresponding to the physical object; and reconstruction is performed based on the camera parameters, the virtual grid and the three-dimensional posture result to generate the virtual object.

9. A method for generating a virtual object, comprising: In response to an input instruction on an operation interface, a two-dimensional image is displayed on the operation interface, wherein the two-dimensional image is an image obtained by photographing a physical object in a real scene; in response to a generation instruction on the operation interface, a virtual object associated with the physical object in a virtual scene is displayed on the operation interface, wherein the virtual object is generated based on a three-dimensional posture result of the physical object, the three-dimensional posture result is obtained by estimating the three-dimensional posture of the physical object based on two-dimensional posture features under multiple different viewing angles, and the two-dimensional posture features under multiple different viewing angles are obtained by estimating the two-dimensional posture of the physical object in the two-dimensional image.

10. A method for generating a virtual object, comprising: A two-dimensional image is displayed on a presentation screen of a virtual reality (VR) device or an augmented reality (AR) device, wherein the two-dimensional image is an image obtained by photographing a physical object in a real scene; a two-dimensional posture estimation is performed on the physical object in the two-dimensional image to obtain two-dimensional posture features under multiple different viewing angles; a three-dimensional posture estimation is performed on the physical object based on the two-dimensional posture features under the multiple different viewing angles to obtain a three-dimensional posture result of the physical object; a virtual object associated with the physical object in a virtual scene is generated based on the three-dimensional posture result; and the VR device or the AR device is driven to render and display the virtual object.

11. A method for generating a virtual object, comprising: A two-dimensional image is acquired by calling a first interface, wherein the first interface includes a first parameter, a parameter value of the first parameter is the two-dimensional image, and the two-dimensional image is an image obtained by photographing a physical object in a real scene; a two-dimensional posture estimation is performed on the physical object in the two-dimensional image to obtain two-dimensional posture features under multiple different viewing angles; a three-dimensional posture estimation is performed on the physical object based on the two-dimensional posture features under the multiple different viewing angles to obtain a three-dimensional posture result of the physical object; a virtual object associated with the physical object in a virtual scene is generated based on the three-dimensional posture result; and the virtual object is output by calling a second interface, wherein the second interface includes a second parameter, and a parameter value of the second parameter is the virtual object.

12. A three-dimensional posture processing method, comprising: Capturing two-dimensional images of physical people in real scenes; Performing a two-dimensional posture estimation on the entity person in the two-dimensional image to obtain two-dimensional posture features under multiple different viewing angles; performing a three-dimensional posture estimation on the entity person based on the two-dimensional posture features under multiple different viewing angles to obtain a three-dimensional posture result of the entity person; The three-dimensional posture result is matched with a preset three-dimensional posture to obtain a processing result of the three-dimensional posture result, wherein the processing result is used to indicate whether the three-dimensional posture result matches the preset three-dimensional posture.

13. An electronic device, comprising: A memory storing an executable program; A processor is configured to run the program, wherein the program executes the method according to any one of claims 1 to 12 when running.

14. A computer-readable storage medium, wherein: The computer-readable storage medium includes a stored executable program, wherein when the executable program is executed, the device where the computer-readable storage medium is located is controlled to execute the method according to any one of claims 1 to 12.

Citation Information

Patent Citations

  • Three-dimensional model driving method and device, electronic equipment and storage medium

    CN111540055A

  • Computer-implemented method, data processing apparatus, and computer program for generating three-dimensional pose-estimation data

    US20220262036A1

  • Keypoint-based sampling for pose estimation

    US20220301304A1

Cited By

  • Three-dimensional efficient human body sitting posture recognition method based on deep learning

    CN119600688A

  • 3D human body posture estimation method and system, electronic equipment and storage medium

    CN120014713A