Pose estimation method, virtual object driving method, and electronic device

Abstract

The application discloses a pose estimation method, a driving method of a virtual object, and an electronic device. The method comprises the following steps: acquiring an image frame set, wherein the same biological object is displayed in the image frames in the image frame set, and a monitoring part in the biological object is a part that needs to be adjusted in three-dimensional pose; performing pose estimation on the biological object in the image frame to obtain initial pose features of the biological object, wherein the initial pose features at least comprise an initial three-dimensional pose of the biological object, an initial two-dimensional position, and a ground contact state of the monitoring part, and the ground contact state is used to represent that the monitoring part is in contact with a preset object in the image frame; and adjusting the three-dimensional pose of the monitoring part in the initial three-dimensional pose based on the initial two-dimensional position and the ground contact state to obtain a target pose of the biological object. The application solves the technical problem that it is difficult to adjust the monitoring part of the biological object in the related art.

Description

Technical Field

[0001] This application relates to the field of data processing, and more specifically, to a pose estimation method, a virtual object driving method, and an electronic device. Background Technology

[0002] Traditional optical or inertial motion capture technologies rely on expensive and cumbersome wearable acquisition equipment, which is the reason for the high cost of AI digital human services. With the popularization of deep learning, deep neural network models can be used to capture real human movements from a monocular camera and drive digital humans. On the one hand, this allows for the convenient and rapid generation of large amounts of realistic motion data; on the other hand, it can be used to directly drive digital human interaction, significantly reducing the cost of AI digital human services and representing an important technological means for the intelligence of AI digital humans.

[0003] Capturing real-person movements from a monocular camera and then replicating them onto a virtual avatar can easily and quickly generate a large amount of realistic motion material, and can also be used to directly drive virtual avatar interactions. However, current monocular driving methods only focus on the overall biological object, making it difficult to adjust specific monitoring parts of the biological object.

[0004] There is currently no effective solution to the above problems. Summary of the Invention

[0005] This application provides an attitude estimation method, a virtual object driving method, and an electronic device to at least solve the technical problem in the related art of making it difficult to adjust the monitoring parts of biological objects.

[0006] According to one aspect of the embodiments of this application, a pose estimation method is provided, comprising: acquiring a set of image frames, wherein the image frames in the set of image frames display the same biological object, and a monitoring part in the biological object is a part whose three-dimensional pose needs to be adjusted; performing pose estimation on the biological object in the image frames to obtain initial pose features of the biological object, wherein the initial pose features include at least: the initial three-dimensional pose and initial two-dimensional position of the biological object, and the ground contact state of the monitoring part, the ground contact state being used to characterize the contact between the monitoring part and a preset object in the image frame; adjusting the three-dimensional pose of the monitoring part in the initial three-dimensional pose based on the initial two-dimensional position and the ground contact state to obtain a target pose of the biological object.

[0007] According to one aspect of the embodiments of this application, a pose estimation method is provided, comprising: responding to an input command applied to an operation interface, displaying a set of image frames on the operation interface, wherein the image frames in the set of image frames display the same biological object, and a monitoring part in the biological object is a part whose three-dimensional pose needs to be adjusted; responding to a pose estimation command applied to the operation interface, displaying a target pose estimation of the biological object on the operation interface, wherein the target pose is obtained by adjusting the three-dimensional pose of the monitoring part in the initial three-dimensional pose of the biological object through the initial two-dimensional position of the biological object and the ground contact state of the monitoring part, wherein the initial three-dimensional pose, the initial two-dimensional position, and the ground contact state are predicted by pose estimation of the biological object in the image frames, and the ground contact state is used to characterize the contact between the monitoring part and a preset object in the image frame.

[0008] According to one aspect of the embodiments of this application, a pose estimation method is provided, comprising: displaying a set of image frames on a presentation screen of a virtual reality (VR) device or an augmented reality (AR) device, wherein the image frames in the set of image frames display the same biological object, and a monitoring part of the biological object is a part whose three-dimensional pose needs to be adjusted; performing pose estimation on the biological object in the image frames to predict the initial pose features of the biological object, wherein the initial pose features include at least: the initial three-dimensional pose and initial two-dimensional position of the biological object, and the ground contact state of the monitoring part, the ground contact state being used to characterize the contact between the monitoring part and a preset object in the image frames; adjusting the three-dimensional pose of the monitoring part in the initial three-dimensional pose based on the initial two-dimensional position and the ground contact state to obtain a target pose of the biological object; and driving the VR device or the AR device to render and display the target pose.

[0009] According to one aspect of the embodiments of this application, a pose estimation method is provided, comprising: acquiring a set of image frames by calling a first interface, wherein the image frames in the set of image frames display the same biological object, and a monitoring part in the biological object is a part whose three-dimensional pose needs to be adjusted; performing pose estimation on the biological object in the image frames to predict the initial pose features of the biological object, wherein the initial pose features include at least: the initial three-dimensional pose and initial two-dimensional position of the biological object, and the ground contact state of the monitoring part, the ground contact state being used to characterize the contact between the monitoring part and a preset object in the image frame; adjusting the three-dimensional pose of the monitoring part in the initial three-dimensional pose based on the initial two-dimensional position and the ground contact state to obtain a target pose of the biological object; and outputting the target pose by calling a second interface, wherein the second interface includes a second parameter, and the parameter value of the second parameter is the target pose.

[0010] According to one aspect of the embodiments of this application, a method for driving a virtual object is provided, comprising: obtaining a set of image frames by photographing a biological object, wherein a monitored part in the biological object is a part whose three-dimensional pose needs to be adjusted; performing pose estimation on the biological object in the image frames to predict the initial pose features of the biological object, wherein the initial pose features include at least: the initial three-dimensional pose and initial two-dimensional position of the biological object, and the ground contact state of the monitored part, the ground contact state being used to characterize the contact between the monitored part and a preset object in the image frames; adjusting the three-dimensional pose of the monitored part in the initial three-dimensional pose based on the initial two-dimensional position and the ground contact state to obtain a target pose of the biological object; and driving a virtual object corresponding to the biological object based on the target pose result.

[0011] According to one aspect of the embodiments of this application, a computer-readable storage medium is provided, the computer-readable storage medium including a stored program, wherein, when the program is executed, it controls the device where the computer-readable storage medium is located to perform the method described in any of the above-described embodiments.

[0012] According to one aspect of the embodiments of this application, an electronic device is provided, including: a processor; and a memory connected to the processor, configured to provide the processor with instructions to perform the following processing steps: acquiring a set of image frames, wherein the image frames in the set of image frames display the same biological object, and a monitoring part in the biological object is a part whose three-dimensional pose needs to be adjusted; performing pose estimation on the biological object in the image frames to predict the initial pose features of the biological object, wherein the initial pose features include at least: the initial three-dimensional pose and initial two-dimensional position of the biological object, and the ground contact state of the monitoring part, the ground contact state being used to characterize the contact between the monitoring part and a preset object in the image frame; adjusting the three-dimensional pose of the monitoring part in the initial three-dimensional pose based on the initial two-dimensional position and the ground contact state to obtain the target pose of the biological object.

[0013] In this embodiment, an image frame set is obtained, wherein the image frames in the image frame set display the same biological object. The monitoring part of the biological object is the part whose three-dimensional pose needs to be adjusted. The pose of the biological object in the image frame is estimated to predict the initial pose features of the biological object. The initial pose features include at least: the initial three-dimensional pose and initial two-dimensional position of the biological object, and the ground contact state of the monitoring part, so that the three-dimensional pose of the monitoring part in the initial three-dimensional pose can be adjusted based on the initial two-dimensional position and ground contact state, thereby achieving the purpose of obtaining the target pose of the biological object. Since the adjustment of the three-dimensional pose of the monitoring part is based on the initial two-dimensional position and ground contact state, the two-dimensional position and ground contact state can be used for supervision, and the adjustment is accurate. Moreover, the ground contact state is used to characterize the contact between the monitoring part and the preset object in the image frame. Therefore, the gap problem of the monitoring part can be better focused on, thereby solving the technical problem of difficulty in adjusting the monitoring part of the biological object in related technologies. Attached Figure Description

[0014] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0015] Figure 1 This is a schematic diagram of the hardware environment of a virtual reality device according to an embodiment of the present application, which describes a pose estimation method.

[0016] Figure 2 This is a structural block diagram of a computational environment for an attitude estimation method according to an embodiment of this application;

[0017] Figure 3 This is a flowchart of an attitude estimation method according to one embodiment of this application;

[0018] Figure 4 This is a flowchart of another attitude estimation method according to one embodiment of this application;

[0019] Figure 5 This is a flowchart of another attitude estimation method according to one embodiment of this application;

[0020] Figure 6 This is a flowchart of another attitude estimation method provided according to one embodiment of this application;

[0021] Figure 7 This is a flowchart of a virtual object driving method according to one embodiment of this application;

[0022] Figure 8 This is a flowchart of an attitude estimation method provided according to an optional embodiment of this application;

[0023] Figure 9 This is a structural block diagram of another attitude estimation device provided according to one embodiment of this application;

[0024] Figure 10 This is a structural block diagram of another attitude estimation device provided according to one embodiment of this application;

[0025] Figure 11 This is a structural block diagram of another attitude estimation device provided according to one embodiment of the present application;

[0026] Figure 12 This is a structural block diagram of another attitude estimation device provided according to one embodiment of the present application;

[0027] Figure 13 This is a structural block diagram of a virtual object driving device according to one embodiment of this application;

[0028] Figure 14 This is a structural block diagram of a computer terminal according to an embodiment of this application. Detailed Implementation

[0029] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0030] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0031] According to an embodiment of this application, an attitude estimation method is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0032] The disclosed method embodiments can be executed on a mobile terminal, computer terminal, or similar computing device. Taking the operation on a mobile terminal as an example, the mobile terminal can be a smartphone, tablet computer, PDA, mobile internet device, PAD, game console, or other terminal device. Figure 1 This is a schematic diagram of the hardware environment of a virtual reality device according to an embodiment of the pose estimation method of this application. Figure 1 As shown, the virtual reality device 104 is connected to the terminal 106, and the terminal 106 is connected to the server 102 via a network. The virtual reality device 104 is not limited to: virtual reality helmets, virtual reality glasses, virtual reality all-in-one machines, etc. The terminal 104 is not limited to PCs, mobile phones, tablets, etc. The server 102 can be a server corresponding to a media file operator. The network includes, but is not limited to: wide area network, metropolitan area network, or local area network.

[0033] Optionally, the virtual reality device 104 in this embodiment includes a memory, a processor, and a transmission device. The memory stores an application program that can be used to perform: acquiring a set of image frames, wherein the image frames in the set display the same biological object, and the monitoring part of the biological object is the part whose three-dimensional pose needs to be adjusted; estimating the pose of the biological object in the image frames to predict the initial pose features of the biological object, wherein the initial pose features include at least: the initial three-dimensional pose and initial two-dimensional position of the biological object, and the ground contact state of the monitoring part, the ground contact state being used to characterize the contact between the monitoring part and a preset object in the image frame; adjusting the three-dimensional pose of the monitoring part in the initial three-dimensional pose based on the initial two-dimensional position and the ground contact state to obtain the target pose of the biological object, thereby solving the technical problem in related technologies that it is difficult to adjust the monitoring part of the biological object.

[0034] The terminal in this embodiment can be used to display a target pose on the display screen of a Virtual Reality (VR) device or Augmented Reality (AR) device. The steps are as follows: A set of image frames is displayed on the display screen of the VR or AR device, wherein the image frames in the set display the same biological object, and the monitored part of the biological object is the part whose three-dimensional pose needs to be adjusted; pose estimation is performed on the biological object in the image frames to predict the initial pose features of the biological object, wherein the initial pose features include at least: the initial three-dimensional pose and initial two-dimensional position of the biological object, and the ground contact state of the monitored part, the ground contact state being used to characterize the contact between the monitored part and a preset object in the image frame; the three-dimensional pose of the monitored part in the initial three-dimensional pose is adjusted based on the initial two-dimensional position and the ground contact state to obtain the target pose of the biological object; the VR or AR device is driven to render and display the target pose.

[0035] In some optional embodiments primarily focused on gaming scenarios, the aforementioned device may also provide a human-computer interaction interface with a touch-sensitive surface. This interface can sense finger contact and / or gestures to interact with a graphical user interface (GUI). The human-computer interaction functions may include the following: creating web pages, drawing, word processing, creating electronic documents, playing games, video conferencing, instant messaging, sending and receiving emails, call interfaces, playing digital videos, playing digital music, and / or web browsing, etc. Executable instructions for performing the aforementioned human-computer interaction functions are configured / stored in a processor-executable computer program product or a readable storage medium.

[0036] Figure 1The hardware structure block diagram shown can serve not only as an exemplary block diagram of the aforementioned AR / VR device (or mobile device), but also as an exemplary block diagram of the aforementioned server. In one optional embodiment, Figure 2 The use of the above is illustrated in a block diagram. Figure 1 The AR / VR device (or mobile device) shown is an example of a computing node in computing environment 201. Figure 2 This is a structural block diagram of the computational environment for a pose estimation method according to an embodiment of this application, such as... Figure 2 As shown, computing environment 201 includes multiple computing nodes (such as servers) running on a distributed network (shown as 210-1, 210-2, ..., in the diagram). Each computing node contains local processing and memory resources, and end user 202 can remotely run applications or store data within computing environment 201. Applications can be provided as multiple services 220-1, 220-2, 220-3, and 220-4 within computing environment 201, representing services "A", "D", "E", and "H", respectively.

[0037] End user 202 can provide and access services through a web browser or other software application on a client. In some embodiments, the provisioning and / or requests of end user 202 can be provided to ingress gateway 230. Ingress gateway 230 may include a corresponding agent to handle the provisioning and / or requests for services (one or more services provided in computing environment 201).

[0038] The service is provided or deployed based on various virtualization technologies supported by the Computing Environment 201. In some embodiments, the service may be provided based on virtual machine (VM)-based virtualization, container-based virtualization, and / or similar methods. Virtual machine-based virtualization can simulate a real computer by initializing a virtual machine, executing programs and applications without directly accessing any actual hardware resources. While the machine is virtualized by a virtual machine, container-based virtualization can launch containers to virtualize an entire operating system (OS), allowing multiple workloads to run on a single OS instance.

[0039] In one embodiment based on container virtualization, several containers of a service can be assembled into a POD (e.g., a Kubernetes POD). For example, such as Figure 2As shown, service 220-2 can be equipped with one or more PODs 240-1, 240-2, ..., 240-N (collectively referred to as PODs). Each POD can include a proxy 245 and one or more containers 242-1, 242-2, ..., 242-M (collectively referred to as containers). One or more containers in a POD handle requests related to one or more corresponding functions of the service. The proxy 245 typically controls network functions related to the service, such as routing, load balancing, etc. Other services can also be PODs similar to PODs.

[0040] During operation, executing a user request from end user 202 may require calling one or more services in computing environment 201, and executing one or more functions of one service may require calling one or more functions of another service. For example... Figure 2 As shown, service "A" 220-1 receives user requests from terminal user 202 from ingress gateway 230. Service "A" 220-1 can call service "D" 220-2, and service "D" 220-2 can request service "E" 220-3 to perform one or more functions.

[0041] The aforementioned computing environment can be a cloud computing environment, where resource allocation is managed by cloud services, allowing functionality development without needing to consider implementation, adjustment, or server scaling. This computing environment allows developers to execute event-responsive code without building or maintaining complex infrastructure. Services can be partitioned into a set of functions that can automatically and independently scale, rather than scaling a single hardware device to handle potential loads.

[0042] Under the aforementioned operating environment, one embodiment of this application provides a method such as... Figure 3 The attitude estimation method shown is illustrated. It should be noted that the attitude estimation method in this embodiment can be derived from... Figure 1 The mobile terminal of the illustrated embodiment is executed. Figure 3 This is a flowchart of an attitude estimation method according to one embodiment of this application. Figure 3 As shown, the method may include the following steps:

[0043] Step S302: Obtain an image frame set, wherein the image frames in the image frame set display the same biological object, and the monitoring part in the biological object is the part whose three-dimensional pose needs to be adjusted;

[0044] In the technical solution provided in step S302 of the present invention, the image frame set can be obtained by sampling from a video of a biological object, or it can be a set of image frames obtained by directly capturing the biological object. The biological object can be a human, animal, or other living organism, and is not limited here; it can be customized according to the actual application and scenario. The monitoring part is a specific part of the biological object whose display result needs to be adjusted. This monitoring part is the focus of monitoring to ensure the accuracy of its position and the accurate display of the monitored part.

[0045] For example, when the biological object is a human and the monitoring part is the feet, the feet are a part of the human whose three-dimensional pose needs to be adjusted. The focus is on monitoring the feet to ensure the accurate determination of the feet's position and the accurate display of the feet.

[0046] Step S304: Perform pose estimation on the biological object in the image frame to obtain the initial pose features of the biological object. The initial pose features include at least: the initial three-dimensional pose and initial two-dimensional position of the biological object, and the ground contact state of the monitored part. The ground contact state is used to characterize the contact between the monitored part and the preset object in the image frame.

[0047] In the technical solution provided in step S304 of the present invention, the initial three-dimensional pose is the three-dimensional pose of the monitored part in space, and the initial two-dimensional position is the two-dimensional position of the monitored part in the image. The ground contact state can be represented by a value within a range. For example, taking the foot as the monitored part, the value in the range of 0-1 represents that the foot is not in contact with the ground, i.e., the foot is off the ground, and 1 represents complete contact, i.e., the foot is in contact with the ground, i.e., the foot is touching the ground. By estimating the pose of the biological object in the image frame, the initial pose features of the biological object are predicted, and the three-dimensional pose of the monitored part in the initial three-dimensional pose can be adjusted to obtain the final target pose.

[0048] For example, when the biological object is a human and the monitored part is the foot, the initial 3D pose is the 3D pose of the foot in space, the initial 2D position is the 2D position of the foot in the image frame, and the ground contact state of the monitored part is the ground contact state of the foot. When identifying the ground contact area in the image frame to obtain the ground contact state, the processing result can be determined by identifying the ground contact area. For example, if the probability of a human foot touching the ground is 50%, then a ground contact state value of 0.5 can be obtained.

[0049] Step S306: Based on the initial two-dimensional position and ground contact state, adjust the three-dimensional pose of the monitored part in the initial three-dimensional pose to obtain the target pose of the biological object.

[0050] In the technical solution provided by step S306 of the present invention, the three-dimensional pose of the monitored part in the initial three-dimensional pose is adjusted based on the initial two-dimensional position and the ground contact state to obtain the target pose of the biological object. This solves the technical problem in related technologies that it is difficult to adjust the monitored part of the biological object, and achieves the purpose of adjusting the monitored part to make the pose of the monitored part of the biological object more accurate. Moreover, since the adjustment is made through the initial two-dimensional position and the ground contact state, the two-dimensional position and the ground contact state can be used for supervision, making the pose of the monitored part of the biological object more accurate.

[0051] For example, when the biological object is a human and the monitoring part is the foot, the three-dimensional pose of the foot in space is supervised based on the two-dimensional position of the foot in the image frame and the ground contact state of 0.5, so as to obtain a more accurate target pose of the human.

[0052] Through the above steps, a set of image frames is obtained, in which the same biological object is displayed in the image frames. The monitored part of the biological object is the part whose three-dimensional pose needs to be adjusted. The pose of the biological object in the image frames is estimated to predict the initial pose features of the biological object. The initial pose features include at least: the initial three-dimensional pose and initial two-dimensional position of the biological object, and the ground contact state of the monitored part. This allows for adjustment of the three-dimensional pose of the monitored part in the initial three-dimensional pose based on the initial two-dimensional position and ground contact state, thereby achieving the goal of obtaining the target pose of the biological object. Since the adjustment of the three-dimensional pose of the monitored part is based on the initial two-dimensional position and ground contact state, the two-dimensional position and ground contact state can be used for supervision, making the position adjustment of the monitored part of the biological object more accurate. Moreover, the ground contact state is used to characterize the contact between the monitored part and the preset object in the image frame. Therefore, the gap problem of the monitored part can be better addressed, and the constraints of the ground contact state can be taken into account, achieving the effect of natural walking movement on the ground. This solves the technical problem of difficulty in adjusting the monitored part of the biological object in related technologies.

[0053] As an optional embodiment, in step S304, the biological object in the image frame is subjected to pose estimation to predict the initial pose features of the biological object, including: using a regression model to perform three-dimensional pose estimation on the biological object in the image frame to obtain the initial three-dimensional pose; using an extraction model to perform two-dimensional pose estimation on the biological object in the image frame to obtain the initial two-dimensional position and ground contact state.

[0054] In this embodiment, the regression model is used for 3D pose estimation of the biological object. By using this model, the 3D pose of the biological object can be estimated more accurately, resulting in a more accurate initial 3D pose and effectively avoiding misestimation. The extraction model is used for 2D pose estimation of the biological object. It can better estimate the 2D pose of the biological object, obtaining the initial 2D position and ground contact state, effectively avoiding errors between the identified initial 2D position and the actual 2D position, as well as misjudgments of the ground contact state.

[0055] It should be noted that during the process of pose estimation for biological objects in image frames and predicting their initial pose features, for 3D pose estimation, other models and algorithms with 3D pose estimation capabilities can be used. Similarly, for 2D pose estimation, other models and algorithms with 2D pose estimation capabilities can be used. There are no limitations here; adaptive settings can be made according to the actual application and scenario. After determining the model and algorithm, appropriate loss functions, including loss function terms and regularization terms, can be set for further training of the model or algorithm to achieve better prediction results.

[0056] As an optional embodiment, the extraction model is used to perform two-dimensional pose estimation of biological objects in the image frame to obtain the ground contact state, including: using the extraction model to estimate the ground contact of biological objects in the image frame to obtain the ground contact area in the image frame, wherein the ground contact area is used to characterize the area in the image frame where the monitored part comes into contact with the preset object; and identifying the image of the ground contact area in the image frame to obtain the ground contact state.

[0057] In this embodiment, when using the extraction model to estimate the ground contact of biological objects in an image frame, the ground contact area in the image frame is determined, that is, the ground contact area of ​​the monitored part of the biological object. For example, if the biological object is a person and the monitored part is the foot, the ground contact area of ​​the person's foot is the aforementioned ground contact area, and the ground contact state can be determined based on the ground contact area of ​​the person's foot.

[0058] It should be noted that when determining the ground contact state of the monitored part, the monitored part can be divided into multiple parts. Taking the example above, the foot can be divided into the front foot part and the back foot part, or the foot can be divided into more areas to determine the ground contact state. Determining the ground contact state in this way can make the subsequent adjustment of the three-dimensional pose of the monitored part in the initial three-dimensional pose more accurate.

[0059] As an optional embodiment, in step S306, adjusting the three-dimensional pose of the monitored part in the initial three-dimensional pose based on the initial two-dimensional position and the ground contact state to obtain the target pose of the object includes: adjusting the three-dimensional pose of at least one part of the biological object in the initial three-dimensional pose based on the initial two-dimensional position to obtain an intermediate three-dimensional pose, wherein at least one part in the intermediate three-dimensional pose coincides with at least one part in the two-dimensional position; and adjusting the three-dimensional pose of the monitored part in the intermediate three-dimensional pose based on the ground contact state to obtain the target pose.

[0060] In this embodiment, the above steps include a two-part adjustment process. First, the 3D pose of at least one part of the biological object in the initial 3D pose is adjusted based on the initial 2D position. Specifically, if the monitored part is the foot, the 3D pose of the foot is adjusted. This adjustment process can correct the spatial position of the initial 3D pose. Since the 2D position has more artificial mechanical constraints and is easier to train, adjusting the initial 3D pose based on the initial 2D position makes the 3D pose more accurate. Second, the 3D pose of the monitored part in the intermediate 3D pose is adjusted based on the ground contact state. This involves determining whether the part is in contact with the ground and adjusting the positional relationship between the monitored part and the ground plane. Adjusting the 3D pose of the monitored part in the intermediate 3D pose based on the ground contact state ensures that the biological object lands on the ground when necessary, preventing the monitored part from appearing to float on the floor or creating a visually discordant effect with the ground.

[0061] For example, when the biological object is a human and the monitoring part is the feet, the three-dimensional pose in space is adjusted based on the two-dimensional position of the feet in the image frame to obtain the intermediate three-dimensional pose; the display result of the feet in the intermediate three-dimensional pose is adjusted based on the ground contact state to obtain the target pose of the human.

[0062] As an optional embodiment, adjusting the three-dimensional pose of at least one part of a biological object in an initial three-dimensional pose based on the two-dimensional position to obtain an intermediate three-dimensional pose includes: performing a projection transformation on the initial three-dimensional pose to obtain a target two-dimensional position of the biological object; determining the distance between the target position of at least one part in the target two-dimensional position and the initial position of at least one part in the initial two-dimensional position; and adjusting the three-dimensional pose of at least one part in the initial three-dimensional pose at least once based on the distance to obtain an intermediate three-dimensional pose.

[0063] In this embodiment, the intermediate three-dimensional pose is obtained by adjusting the initial three-dimensional pose through projection transformation. Because two-dimensional positions are easily constrained and trained, using two-dimensional positions to constrain and adjust the three-dimensional pose makes the obtained intermediate three-dimensional pose more ergonomic and realistic. Therefore, the initial three-dimensional pose is projected to obtain the target two-dimensional position of the biological object. By determining the distance between the target position of at least one part of the target two-dimensional position and the initial position of at least one part of the initial two-dimensional position, the three-dimensional pose of at least one part of the initial three-dimensional pose is adjusted at least once based on the distance. In this process, the two-dimensional position is first adjusted based on the distance between the target two-dimensional position and the initial two-dimensional position, and then the two-dimensional position is back-projected into a three-dimensional pose for adjustment. This optional implementation not only allows for better adjustment but also, through projection, allows for simpler and faster adjustment of the three-dimensional pose of at least one part of the biological object in the initial three-dimensional pose to obtain the intermediate three-dimensional pose.

[0064] For example, when the biological object is a human and the monitoring part is the foot, the three-dimensional pose in space is projected and transformed to obtain the transformed two-dimensional position of the foot in the image frame. The distance between the transformed two-dimensional position and the two-dimensional position is determined. Based on the distance, the three-dimensional pose in space is adjusted to obtain the intermediate three-dimensional pose.

[0065] As an optional embodiment, adjusting the position data of the three-dimensional pose of the monitored part in the intermediate three-dimensional pose based on the ground contact state to obtain the target pose includes: when the ground contact state is that the monitored part is in contact with a preset object, determining the first position of the monitored part in the image frame and the second position relative to the preset object based on the intermediate three-dimensional pose; adjusting the height data of the monitored part in the intermediate three-dimensional pose and the three-dimensional pose of the preset part based on the first position and the second position to obtain the target pose, wherein the preset part is a part of the biological object connected to the monitored part.

[0066] In this embodiment, for example, when the biological object is a human and the monitored part is the foot, the constraint of the foot landing is considered when the foot is on the ground, that is, the constraint of the foot position relative to the ground plane is considered to make the foot position more coordinated with the ground plane position. During the adjustment process, the display results of the foot and leg in the intermediate three-dimensional pose are also adjusted based on the position of the foot and the ground plane to obtain the target pose, making the overall biological object more harmonious. Moreover, by adjusting the display results of the monitored part and the preset part in the intermediate three-dimensional pose, forward, backward, left and right slippage can also be eliminated. That is, by using inverse dynamics, the posture of the corresponding leg is finely adjusted to pull the foot to the same position, achieving a better stabilization effect.

[0067] As an optional embodiment, determining a first position of the monitored part in an image frame and a second position relative to a preset object based on an intermediate three-dimensional pose includes: determining the first position based on the intermediate three-dimensional pose; determining a ground contact position in the image frame based on the first position, wherein the ground contact position is used to characterize the position where the monitored part contacts the preset object; and determining the second position based on the ground contact position in the image frame.

[0068] In this embodiment, a first position is determined based on an intermediate three-dimensional pose. For example, when the biological object is a human and the monitored part is the foot, the spatial position of the foot is determined. Since the foot is in complete contact with the ground, the first position of the foot can determine the ground contact position of the image frame. Based on the ground contact position of the image frame, the position of the ground plane is found, and the second position can be determined. This method makes the determination of the first and second positions more reasonable and accurate, which is beneficial to the accuracy of subsequent target pose adjustments.

[0069] As an optional embodiment, adjusting the display results of the monitored parts and the preset parts in the intermediate three-dimensional pose based on the first position and the second position to obtain the target pose includes: unifying the height data of the monitored parts in the intermediate three-dimensional pose based on the first position to obtain the adjusted three-dimensional pose; and adjusting the three-dimensional pose of the preset parts in the adjusted three-dimensional pose based on the second position to obtain the target pose, wherein the positions of the monitored parts in the target pose are the same.

[0070] In this embodiment, for example, when the biological object is a human and the monitored part is the foot, in order to lock and constrain the spatial position and height of the foot when it touches the ground, considering the height constraint of the ground, the position of the ground plane is estimated from multiple consecutive landing points. Then, the landing height of all frames is unified to fall on the same ground. That is, the height data of the monitored part in the intermediate three-dimensional pose is unified based on the first position to obtain the adjusted three-dimensional pose. Then, the three-dimensional pose of the preset part in the adjusted three-dimensional pose is adjusted based on the second position to obtain the target pose, so that the position of the monitored part is the same in the target pose. In this way, the landing height of all frames can be unified to fall on the same ground plane, thus eliminating the phenomenon of vertical fluctuation and achieving the effect of natural walking movement on the ground.

[0071] Under the above operating environment, one embodiment of this application also provides another such... Figure 4 The attitude estimation method shown is illustrated. It should be noted that the attitude estimation method in this embodiment can be derived from... Figure 1 The mobile terminal of the illustrated embodiment is executed. Figure 4 This is a flowchart of another attitude estimation method according to one embodiment of this application. Figure 4 As shown, the method may include the following steps:

[0072] Step S402: In response to the input command applied to the operation interface, a set of image frames is displayed on the operation interface. The image frames in the set of image frames display the same biological object, and the monitoring part of the biological object is the part whose three-dimensional pose needs to be adjusted.

[0073] Step S404: In response to the pose estimation command applied to the operation interface, the target pose estimation of the biological object is displayed on the operation interface. The target pose is obtained by adjusting the three-dimensional pose of the monitoring part in the initial three-dimensional pose of the biological object through the initial two-dimensional position of the biological object and the ground contact state of the monitoring part. The initial three-dimensional pose, initial two-dimensional position and ground contact state are predicted by the pose estimation of the biological object in the image frame. The ground contact state is used to characterize the contact between the monitoring part and the preset object in the image frame.

[0074] Through the above steps, in response to input commands applied to the operating interface, a set of image frames is displayed on the operating interface. Each image frame in the set displays the same biological object, and the monitored part of the biological object is the part whose 3D pose needs adjustment. In response to pose estimation commands applied to the operating interface, the target pose estimate of the biological object is displayed on the operating interface, thus achieving the goal of obtaining the target pose of the biological object. Since the adjustment of the 3D pose of the monitored part in the initial 3D pose of the biological object is based on the initial 2D position and ground contact state, it can be supervised using the 2D position and ground contact state, ensuring accurate adjustment. Moreover, the ground contact state is used to characterize the contact between the monitored part and the preset object in the image frame, thus allowing for better attention to the gap problem of the monitored part, thereby solving the technical problem of difficulty in adjusting the monitored part of the biological object in related technologies.

[0075] According to the embodiments of this application, a pose estimation method applicable to virtual reality scenarios such as virtual reality (VR) devices and augmented reality (AR) devices is also provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order than that shown here.

[0076] Figure 5 This is a flowchart of another attitude estimation method according to one embodiment of this application. Figure 5 As shown, the method may include the following steps:

[0077] Step S502: Display a set of image frames on the presentation screen of a virtual reality (VR) device or an augmented reality (AR) device. The image frames in the set of image frames display the same biological object, and the monitoring part of the biological object is the part whose three-dimensional pose needs to be adjusted.

[0078] Step S504: Perform pose estimation on the biological object in the image frame to predict the initial pose features of the biological object. The initial pose features include at least: the initial three-dimensional pose and initial two-dimensional position of the biological object, and the ground contact state of the monitored part. The ground contact state is used to characterize the contact between the monitored part and the preset object in the image frame.

[0079] Step S506: Based on the initial two-dimensional position and ground contact state, adjust the three-dimensional pose of the monitored part in the initial three-dimensional pose to obtain the target pose of the biological object.

[0080] Step S508: Drive the VR device or AR device to render and display the target pose.

[0081] Through the above steps, a set of image frames is displayed on the screen of a virtual reality (VR) device or an augmented reality (AR) device. Each image frame in the set displays the same biological object, and the monitored part of the biological object is the part whose 3D pose needs adjustment. The pose of the biological object in the image frames is estimated to predict its initial pose features. These initial pose features include at least: the initial 3D pose and initial 2D position of the biological object, and the ground contact state of the monitored part. Based on the initial 2D position and ground contact state, the 3D pose of the monitored part in the initial 3D pose is adjusted to obtain the target pose of the biological object. This drives the VR or AR device to render and display the target pose, thus achieving the goal of obtaining the target pose of the biological object. Since the adjustment of the 3D pose of the monitored part is based on the initial 2D position and ground contact state, it can be supervised using the 2D position and ground contact state, ensuring accurate adjustment. Furthermore, the ground contact state characterizes the contact between the monitored part and the preset object in the image frame, thus allowing for better attention to the gap problem of the monitored part. This solves the technical problem of difficulty in adjusting the monitored part of a biological object in related technologies.

[0082] Optionally, in this embodiment, the above-described pose estimation method can be applied to a hardware environment consisting of a server and a virtual reality device. The target pose is displayed on the screen of the virtual reality (VR) device or augmented reality (AR) device. The server can be a server corresponding to a media file operator. The aforementioned network includes, but is not limited to, a wide area network (WAN), a metropolitan area network (MAN), or a local area network (LAN). The aforementioned virtual reality device is not limited to, for example, a virtual reality headset, virtual reality glasses, or a standalone virtual reality device.

[0083] Optionally, the virtual reality device includes: a memory, a processor, and a transmission device. The memory stores an application that can be used to perform: displaying a set of image frames on the presentation screen of a virtual reality (VR) device or an augmented reality (AR) device, wherein the image frames in the image frame set display the same biological object, and the monitored part of the biological object is the part whose three-dimensional pose needs to be adjusted; estimating the pose of the biological object in the image frames to predict the initial pose features of the biological object, wherein the initial pose features include at least: the initial three-dimensional pose and initial two-dimensional position of the biological object, and the ground contact state of the monitored part, the ground contact state being used to characterize the contact between the monitored part and a preset object in the image frame; adjusting the three-dimensional pose of the monitored part in the initial three-dimensional pose based on the initial two-dimensional position and the ground contact state to obtain the target pose of the biological object; and driving the VR device or AR device to render and display the target pose.

[0084] It should be noted that the pose estimation method described above in this embodiment, when applied to VR or AR devices, may include... Figure 5 The method of the illustrated embodiment is used to drive a VR device or AR device to display a target pose.

[0085] Optionally, the processor in this embodiment can invoke the application stored in the memory via the transmission device to execute the above steps. The transmission device can receive media files sent by the server via a network, and can also be used for data transmission between the processor and the memory.

[0086] Optionally, in a virtual reality device, there is a head-mounted display with eye tracking. The screen in the HMD is used to display the video footage. The eye tracking module in the HMD is used to acquire the real-time movement trajectory of the user's eyes. The tracking system is used to track the user's position and movement information in real three-dimensional space. The computing and processing unit is used to acquire the user's real-time position and movement information from the tracking system and calculate the three-dimensional coordinates of the user's head in the virtual three-dimensional space, as well as the user's field of vision orientation in the virtual three-dimensional space.

[0087] In this embodiment, the virtual reality device can be connected to a terminal, and the terminal and the server are connected via a network. The virtual reality device is not limited to virtual reality headsets, virtual reality glasses, virtual reality all-in-one machines, etc., and the terminal is not limited to PCs, mobile phones, tablets, etc. The server can be a server corresponding to a media file operator, and the network includes, but is not limited to, wide area networks, metropolitan area networks, or local area networks.

[0088] Under the above operating environment, one embodiment of this application also provides another such... Figure 6 The attitude estimation method shown is illustrated. It should be noted that the attitude estimation method in this embodiment can be derived from... Figure 1The mobile terminal of the illustrated embodiment is executed. Figure 6 This is a flowchart of another attitude estimation method provided according to one embodiment of this application. Figure 6 As shown, the method may include the following steps:

[0089] Step S602: Obtain an image frame set by calling the first interface. The image frames in the image frame set display the same biological object, and the monitoring part of the biological object is the part whose three-dimensional pose needs to be adjusted.

[0090] In the technical solution provided by step S602 of the present invention, the first interface can be an interface for data interaction between the server and the client. The client can use the set of image frames in the detection scene as a first parameter of the first interface to achieve the purpose of obtaining the set of image frames using the first interface.

[0091] Step S604: Perform pose estimation on the biological object in the image frame to predict the initial pose features of the biological object. The initial pose features include at least: the initial three-dimensional pose and initial two-dimensional position of the biological object, and the ground contact state of the monitored part. The ground contact state is used to characterize the contact between the monitored part and the preset object in the image frame.

[0092] Step S606: Adjust the three-dimensional pose of the monitored part in the initial three-dimensional pose based on the initial two-dimensional position and the ground contact state to obtain the target pose of the biological object.

[0093] Step S608: Output the target pose by calling the second interface, wherein the second interface includes a second parameter, and the parameter value of the second parameter is the target pose.

[0094] In the technical solution provided by step S608 of the present invention, the second interface can be an interface for data interaction between the server and the client. The server can pass the image frame set into the second interface as a parameter of the second interface to achieve the purpose of outputting the target pose.

[0095] Through the above steps, an image frame set is obtained by calling the first interface. The image frames in this set display the same biological object, and the monitored part of the biological object is the part whose 3D pose needs adjustment. Pose estimation is performed on the biological object in the image frames to predict its initial pose features. These initial pose features include at least: the biological object's initial 3D pose and initial 2D position, and the ground contact state of the monitored part. Based on the initial 2D position and ground contact state, the 3D pose of the monitored part in the initial 3D pose is adjusted to obtain the target pose of the biological object. The target pose is then output by calling the second interface, which includes a second parameter whose value is the target pose, thus achieving the goal of obtaining the target pose of the biological object. Since the adjustment of the 3D pose of the monitored part is based on the initial 2D position and ground contact state, it can be supervised using the 2D position and ground contact state, ensuring accurate adjustment. Furthermore, the ground contact state characterizes the contact between the monitored part and the preset object in the image frame, thus allowing for better attention to the gap problem of the monitored part. This solves the technical problem of difficulty in adjusting the monitored part of biological objects in related technologies.

[0096] Under the aforementioned operating environment, one embodiment of this application also provides a method such as... Figure 7 The driving method for the virtual object shown is illustrated. It should be noted that the driving method for the virtual object in this embodiment can be provided by... Figure 1 The mobile terminal of the illustrated embodiment is executed. Figure 7 This is a flowchart of a virtual object driving method according to one embodiment of this application. Figure 7 As shown, the method may include the following steps:

[0097] Step S702: A set of image frames is obtained by taking pictures of the biological object, wherein the monitored part of the biological object is the part whose three-dimensional pose needs to be adjusted;

[0098] Step S704: Perform pose estimation on the biological object in the image frame to predict the initial pose features of the biological object. The initial pose features include at least: the initial three-dimensional pose and initial two-dimensional position of the biological object, and the ground contact state of the monitored part. The ground contact state is used to characterize the contact between the monitored part and the preset object in the image frame.

[0099] Step S706: Based on the initial two-dimensional position and ground contact state, adjust the three-dimensional pose of the monitored part in the initial three-dimensional pose to obtain the target pose of the biological object.

[0100] Step S708: Drive the virtual object corresponding to the biological object based on the target pose result.

[0101] Through the above steps, an image frame set is obtained by capturing images of a biological object. The monitored part of the biological object is the part whose 3D pose needs to be adjusted. The pose of the biological object in the image frames is estimated to predict its initial pose features. These initial pose features include at least: the initial 3D pose and initial 2D position of the biological object, and the ground contact state of the monitored part. Based on the initial 2D position and ground contact state, the 3D pose of the monitored part in the initial 3D pose is adjusted to obtain the target pose of the biological object. Based on the target pose result, the corresponding virtual object is driven, achieving the goal of obtaining the target pose of the biological object. Since the adjustment of the 3D pose of the monitored part is based on the initial 2D position and ground contact state, it can be supervised using the 2D position and ground contact state, ensuring accurate adjustment. Furthermore, the ground contact state represents the contact between the monitored part and the preset object in the image frame, thus allowing for better attention to the gap problem of the monitored part, thereby solving the technical problem of difficulty in adjusting the monitored part of a biological object in related technologies.

[0102] In related technologies, monocular actuation methods have not yet addressed or can achieve a natural replication of foot-to-ground interaction. For example, taking a human as the biological object and the feet as the monitoring area, current methods pay little attention to foot posture. When the feet touch the ground, there are often noticeable and unreasonable shaking and sliding. This effect is even more pronounced when driving virtual humans. Because a person's center of gravity is on their feet, unreasonable shaking and sliding of the feet disrupts the naturalness of the overall replication effect, making it appear strange.

[0103] Based on this, this application provides an optional implementation method, taking a human body as the biological object and the foot as the monitoring part as an example, which is described in detail below.

[0104] In an optional embodiment of this application, an attitude estimation method is provided. Figure 8 This is a flowchart of an attitude estimation method provided according to an optional embodiment of this application, such as... Figure 8 As shown, this optional embodiment uses monocular estimation to estimate the 3D limb posture of the human body and estimates the interaction state between the feet and the ground in the scene, which improves the accuracy of posture estimation and the naturalness of motion replication. The optional implementation of this application will be described in detail below.

[0105] S1, Obtain an image frame set, wherein the image frames in the image frame set display the same user, and the monitored part of the user is the part whose three-dimensional pose needs to be adjusted. In this optional embodiment, the monitored part is the foot.

[0106] S2, perform pose estimation on the user in the video image frame, and predict the initial 3D pose and 2D position of the user's feet;

[0107] For example, based on the Transformer model of temporal machine translation, the initial 3D pose is obtained through the autoregression mechanism of the regression module, and the 2D position is extracted through the extraction module;

[0108] S3, adjust the initial 3D pose based on the 2D position to obtain the intermediate 3D pose;

[0109] For example, using the predicted 3D pose as the optimization object, the initial 3D pose is adjusted based on the 2D position, and the intermediate 3D pose is obtained through the pose optimizer.

[0110] The repair process is based on projection transformation, which projects the joints in 3D space onto the image coordinate system. The predicted 2D joint positions are used for constraint supervision to minimize the difference between the two, so that the projected joints are close to the predicted 2D joints. Through multiple iterations of optimization, a more accurate intermediate 3D pose is obtained.

[0111] It should be noted that adjusting the 3D pose based on the 2D position can correct the spatial position of the 3D pose. Since the 2D position has more artificial mechanical constraints and is easier to train, adjusting the 3D pose through the 2D position can make the subsequent intermediate 3D pose more accurate.

[0112] S4, use the extraction module to estimate the ground contact of the user in the image frame, obtain the ground contact area in the image frame, identify the image of the ground contact area in the image frame, and obtain the ground contact status;

[0113] The ground contact state includes toe contact and heel contact. No contact is recorded as 0, and contact is recorded as 1. (0-1 confidence level) is represented by four data points (left and right feet, front and back heels) (additional points can also be added).

[0114] S5, adjust the display results of the feet in the intermediate 3D pose based on the ground contact state to obtain the target pose.

[0115] For example, based on the ground contact state and intermediate 3D pose, the spatial position and height of the ground contact foot are locked and constrained. The target pose is obtained through the foot adjuster. For example, if the ground contact state is 1, the foot is adjusted to the ground plane through the foot adjuster to obtain the target pose.

[0116] When the foot lands, the constraints of the landing are taken into account. First, the height constraint of the ground is considered. The position of the ground plane is estimated from the landing points of multiple consecutive frames. Then, the landing height of all frames is unified to fall on the same ground, thus eliminating the phenomenon of vertical fluctuation. Then, when landing in consecutive frames, it is necessary to eliminate forward, backward and left and right sliding. Inverse dynamics is used to finely adjust the posture of the corresponding leg to pull the foot to the same position.

[0117] The following improvements and beneficial effects can be achieved through the above optional implementation methods:

[0118] 1) The deep learning model adds a branch that learns to determine whether the feet are on the ground. When the feet are on the ground, it will consider the constraints of the ground, so that the character can remain relatively still with the feet on the ground and can walk and move naturally on the ground.

[0119] 2) Since the adjustment of the foot display is based on 2D position and ground contact state, the 2D position and ground contact state can be used for supervision, making the adjustment of the user's foot position more accurate. Moreover, the ground contact state is used to represent the contact between the foot and the preset object in the image frame. Therefore, the gap problem of the foot can be better addressed, thus solving the technical problem of difficulty in adjusting the user's foot in related technologies.

[0120] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0121] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods of the various embodiments of this application.

[0122] According to an embodiment of this application, an apparatus for implementing the above-described attitude estimation method is provided. Figure 9 This is a structural block diagram of another attitude estimation device according to one embodiment of this application, such as... Figure 9 As shown, the device includes a first acquisition module 902, a first prediction module 904, and a first adjustment module 906. The device will be described below.

[0123] The first acquisition module 902 is used to acquire a set of image frames, wherein the image frames in the set display the same biological object, and the monitoring part of the biological object is the part whose three-dimensional pose needs to be adjusted; the first prediction module 904 is connected to the first acquisition module 902, and is used to perform pose estimation on the biological object in the image frames to predict the initial pose features of the biological object, wherein the initial pose features include at least: the initial three-dimensional pose and initial two-dimensional position of the biological object, and the ground contact state of the monitoring part, the ground contact state being used to characterize the contact between the monitoring part and a preset object in the image frame; the first adjustment module 906 is connected to the first prediction module 904, and is used to adjust the three-dimensional pose of the monitoring part in the initial three-dimensional pose based on the initial two-dimensional position and the ground contact state to obtain the target pose of the biological object.

[0124] It should be noted that the first acquisition module 902, the first prediction module 904, and the first adjustment module 906 mentioned above correspond to steps S302 to S306 in the embodiments. Multiple modules and their corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in the above embodiments. It should also be noted that the above modules, as part of the device, can run in the AR / VR device provided in the embodiments.

[0125] According to an embodiment of this application, another apparatus for implementing the above-described attitude estimation method is provided. Figure 10 This is a structural block diagram of another attitude estimation device provided according to one embodiment of this application, such as... Figure 10 As shown, the device includes a first display module 1002 and a second display module 1004. The device will be described below.

[0126] The first display module 1002 is used to respond to input commands applied to the operation interface and display a set of image frames on the operation interface. The image frames in the image frame set display the same biological object, and the monitoring part of the biological object is the part whose three-dimensional pose needs to be adjusted. The second display module 1004 is connected to the first display module 1002 and is used to respond to pose estimation commands applied to the operation interface and display the target pose estimation of the biological object on the operation interface. The target pose is obtained by adjusting the gap problem of the monitoring part to better focus on the monitoring part through the initial two-dimensional position of the biological object and the ground contact state of the monitoring part. The initial three-dimensional pose, initial two-dimensional position and ground contact state are predicted by the pose estimation of the biological object in the image frame. The ground contact state is used to characterize the contact between the monitoring part and the preset object in the image frame.

[0127] It should be noted that the first display module 1002 and the second display module 1004 mentioned above correspond to steps S402 to S404 in the embodiments. Multiple modules and their corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in the above embodiments. It should also be noted that the above modules, as part of the device, can run in the AR / VR device provided in the embodiments.

[0128] According to an embodiment of this application, another apparatus for implementing the above-described attitude estimation method is provided. Figure 11 This is a structural block diagram of another attitude estimation device according to one embodiment of the present application, such as... Figure 11 As shown, the device includes: a first display module 1102, a second prediction module 1104, a second adjustment module 1106, and a second display module 1108. The device will be described below.

[0129] The first display module 1102 is used to display a set of image frames on the display screen of a virtual reality (VR) device or an augmented reality (AR) device. The image frames in the set of image frames display the same biological object, and the monitoring part of the biological object is the part whose three-dimensional pose needs to be adjusted.

[0130] The second prediction module 1104, connected to the first display module 1102, is used to estimate the pose of the biological object in the image frame and predict the initial pose features of the biological object. The initial pose features include at least: the initial three-dimensional pose and initial two-dimensional position of the biological object, and the ground contact state of the monitored part. The ground contact state is used to characterize the contact between the monitored part and a preset object in the image frame. The second adjustment module 1106, connected to the second prediction module 1104, is used to adjust the three-dimensional pose of the monitored part in the initial three-dimensional pose based on the initial two-dimensional position and the ground contact state to obtain the target pose of the biological object. The second display module 1108, connected to the second adjustment module 1106, is used to drive a VR device or an AR device to render and display the target pose.

[0131] It should be noted that the first display module 1102, the second prediction module 1104, the second adjustment module 1106, and the second display module 1108 mentioned above correspond to steps S502 to S508 in the embodiments. Multiple modules and their corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in the above embodiments. It should also be noted that the above modules, as part of the device, can run in the AR / VR device provided in the embodiments.

[0132] According to an embodiment of this application, another apparatus for implementing the above-described attitude estimation method is provided. Figure 12 This is a structural block diagram of another attitude estimation device according to one embodiment of the present application, such as... Figure 12 As shown, the device includes: a second acquisition module 1202, a third prediction module 1204, a third adjustment module 1206, and a calling module 1208. The device will be described below.

[0133] The second acquisition module 1202 is used to acquire a set of image frames by calling the first interface, wherein the image frames in the set display the same biological object, and the monitoring part of the biological object is the part whose three-dimensional pose needs to be adjusted; the third prediction module 1204 is connected to the second acquisition module 1202, and is used to perform pose estimation on the biological object in the image frames to predict the initial pose features of the biological object, wherein the initial pose features include at least: the initial three-dimensional pose and initial two-dimensional position of the biological object, and the ground contact state of the monitoring part, the ground contact state being used to characterize the contact between the monitoring part and a preset object in the image frame; the third adjustment module 1206 is connected to the third prediction module 1204, and is used to adjust the three-dimensional pose of the monitoring part in the initial three-dimensional pose based on the initial two-dimensional position and the ground contact state to obtain the target pose of the biological object; the calling module 1208 is connected to the third adjustment module 1206, and is used to output the target pose by calling the second interface, wherein the second interface includes a second parameter, and the parameter value of the second parameter is the target pose.

[0134] It should be noted that the second acquisition module 1202, the third prediction module 1204, the third adjustment module 1206, and the calling module 1208 mentioned above correspond to steps S602 to S608 in the embodiments. Multiple modules and their corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in the above embodiments. It should also be noted that the above modules, as part of the device, can run in the AR / VR device provided in the embodiments.

[0135] According to an embodiment of this application, an apparatus is provided for implementing the above-described virtual object driving method. Figure 13 This is a structural block diagram of a virtual object driving device according to one embodiment of this application, such as... Figure 13 As shown, the device includes: a shooting module 1302, a fourth prediction module 1304, a fourth adjustment module 1306, and a driving module 1308. The device will be described below.

[0136] The imaging module 1302 is used to obtain a set of image frames by photographing the biological object, wherein the monitored part in the biological object is the part whose three-dimensional pose needs to be adjusted; the fourth prediction module 1304, connected to the imaging module 1302, is used to estimate the pose of the biological object in the image frames and predict the initial pose features of the biological object, wherein the initial pose features include at least: the initial three-dimensional pose and initial two-dimensional position of the biological object, and the ground contact state of the monitored part, the ground contact state being used to characterize the contact between the monitored part and a preset object in the image frame; the fourth adjustment module 1306, connected to the fourth prediction module 1304, is used to adjust the three-dimensional pose of the monitored part in the initial three-dimensional pose based on the initial two-dimensional position and the ground contact state to obtain the target pose of the biological object; the driving module 1308, connected to the fourth adjustment module 1306, is used to drive the virtual object corresponding to the biological object based on the target pose result.

[0137] It should be noted that the aforementioned shooting module 1302, fourth prediction module 1304, fourth adjustment module 1306, and driving module 1308 correspond to steps S702 to S708 in the embodiments. Multiple modules and their corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in the above embodiments. It should also be noted that the aforementioned modules, as part of the device, can operate in the AR / VR device provided in the embodiments.

[0138] Embodiments of this application can provide a pose estimation system. The computer terminal may include an AR / VR device, a server, and a client. The AR / VR device can be any AR / VR device from a group of AR / VR devices. Optionally, the device includes: a processor; and a memory connected to the processor, used to provide the processor with instructions to perform the following processing steps: acquiring a set of image frames, wherein the image frames in the image frame set display the same biological object, and the monitoring part of the biological object is the part whose three-dimensional pose needs to be adjusted; performing pose estimation on the biological object in the image frames to predict the initial pose features of the biological object, wherein the initial pose features include at least: the initial three-dimensional pose and initial two-dimensional position of the biological object, and the ground contact state of the monitoring part, the ground contact state being used to characterize the contact between the monitoring part and a preset object in the image frame; adjusting the three-dimensional pose of the monitoring part in the initial three-dimensional pose based on the initial two-dimensional position and the ground contact state to obtain the target pose of the biological object.

[0139] Embodiments of this application may provide an AR / VR device, which can be any AR / VR device from a group of AR / VR devices. Optionally, in this embodiment, the aforementioned AR / VR device may also be replaced by a terminal device such as a mobile terminal.

[0140] Optionally, in this embodiment, the AR / VR device described above may be located in at least one of a plurality of network devices in a computer network.

[0141] In this embodiment, the AR / VR device described above can execute the following steps in the pose estimation method: acquiring a set of image frames, wherein the image frames in the set display the same biological object, and the monitoring part of the biological object is the part whose three-dimensional pose needs to be adjusted; performing pose estimation on the biological object in the image frames to predict the initial pose features of the biological object, wherein the initial pose features include at least: the initial three-dimensional pose and initial two-dimensional position of the biological object, and the ground contact state of the monitoring part, the ground contact state being used to characterize the contact between the monitoring part and a preset object in the image frame; adjusting the three-dimensional pose of the monitoring part in the initial three-dimensional pose based on the initial two-dimensional position and the ground contact state to obtain the target pose of the biological object.

[0142] Optionally, Figure 14 This is a structural block diagram of a computer terminal according to an embodiment of this application. Figure 14 As shown, the computer terminal may include one or more (only one is shown in the figure) processors 1401, memory 1402, and other components as shown in the figure.

[0143] The memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the attitude estimation method and apparatus in this application embodiment. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, thereby implementing the aforementioned attitude estimation method. The memory may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include memory remotely located relative to the processor, and these remote memories can be connected to the terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0144] The processor can access information and application programs stored in memory via a transmission device to perform the following steps: acquiring a set of image frames, wherein the image frames in the set display the same biological object, and the monitoring part of the biological object is the part whose three-dimensional pose needs to be adjusted; performing pose estimation on the biological object in the image frames to predict the initial pose features of the biological object, wherein the initial pose features include at least: the initial three-dimensional pose and initial two-dimensional position of the biological object, and the ground contact state of the monitoring part, the ground contact state being used to characterize the contact between the monitoring part and a preset object in the image frame; adjusting the three-dimensional pose of the monitoring part in the initial three-dimensional pose based on the initial two-dimensional position and the ground contact state to obtain the target pose of the biological object.

[0145] Optionally, the processor may also execute program code for the following steps: performing pose estimation on the biological object in the image frame and predicting the initial pose features of the biological object, including: using a regression model to perform three-dimensional pose estimation on the biological object in the image frame to obtain the initial three-dimensional pose; using an extraction model to perform two-dimensional pose estimation on the biological object in the image frame to obtain the initial two-dimensional position and ground contact state.

[0146] Optionally, the processor may also execute program code for the following steps: using an extraction model to perform two-dimensional pose estimation on biological objects in an image frame to obtain a ground-touching state, including: using an extraction model to perform ground-touching estimation on biological objects in an image frame to obtain a ground-touching region in the image frame, wherein the ground-touching region is used to characterize the area in the image frame where the monitored part comes into contact with a preset object; and recognizing the image of the ground-touching region in the image frame to obtain the ground-touching state.

[0147] Optionally, the processor may also execute program code for the following steps: adjusting the three-dimensional pose of the monitored part in the initial three-dimensional pose based on the initial two-dimensional position and the ground contact state to obtain the target pose of the object, including: adjusting the three-dimensional pose of at least one part of the biological object in the initial three-dimensional pose based on the initial two-dimensional position to obtain an intermediate three-dimensional pose, wherein at least one part in the intermediate three-dimensional pose coincides with at least one part in the two-dimensional position; adjusting the position data of the three-dimensional pose of the monitored part in the intermediate three-dimensional pose based on the ground contact state to obtain the target pose.

[0148] Optionally, the processor may also execute program code for the following steps: adjusting the three-dimensional pose of at least one part of the biological object in the initial three-dimensional pose based on the two-dimensional position to obtain an intermediate three-dimensional pose, including: performing a projection transformation on the initial three-dimensional pose to obtain the target two-dimensional position of the biological object; determining the distance between the target position of at least one part in the target two-dimensional position and the initial position of at least one part in the initial two-dimensional position; and adjusting the three-dimensional pose of at least one part in the initial three-dimensional pose at least once based on the distance to obtain an intermediate three-dimensional pose.

[0149] Optionally, the processor may also execute program code for the following steps: adjusting the position data of the three-dimensional pose of the monitored part in the intermediate three-dimensional pose based on the ground contact state to obtain the target pose, including: when the ground contact state is that the monitored part is in contact with a preset object, determining the first position of the monitored part and the second position of the preset object in the image frame based on the intermediate three-dimensional pose; adjusting the height data of the monitored part and the three-dimensional pose of the preset part in the intermediate three-dimensional pose based on the first position and the second position to obtain the target pose, wherein the preset part is the part of the biological object connected to the monitored part.

[0150] Optionally, the processor may also execute program code for the following steps: determining a first position of the monitored part and a second position of the preset object in the image frame based on the intermediate three-dimensional pose, including: determining the first position based on the intermediate three-dimensional pose; determining the ground contact position in the image frame based on the first position, wherein the ground contact position is used to characterize the position where the monitored part contacts the preset object; and determining the second position based on the ground contact position in the image frame.

[0151] Optionally, the processor may also execute program code for the following steps: adjusting the height data of the monitored part in the intermediate three-dimensional pose and the three-dimensional pose of the preset part based on the first position and the second position to obtain the target pose, including: unifying the height data of the monitored part in the intermediate three-dimensional pose based on the first position to obtain the adjusted three-dimensional pose; adjusting the three-dimensional pose of the preset part in the adjusted three-dimensional pose based on the second position to obtain the target pose, wherein the positions of the monitored parts in the target pose are the same.

[0152] The processor can invoke information and application programs stored in memory via a transmission device to perform the following steps: In response to an input command applied to the operating interface, display a set of image frames on the operating interface, wherein the image frames in the image frame set display the same biological object, and the monitoring part of the biological object is the part whose three-dimensional pose needs to be adjusted; In response to a pose estimation command applied to the operating interface, display the target pose estimation of the biological object on the operating interface, wherein the target pose is obtained by adjusting the gap problem of the monitoring part to better focus on the initial two-dimensional position of the biological object and the ground contact state of the monitoring part, the initial three-dimensional pose, the initial two-dimensional position, and the ground contact state are predicted by the pose estimation of the biological object in the image frame, and the ground contact state is used to characterize the contact between the monitoring part and the preset object in the image frame.

[0153] The processor can access information and applications stored in memory via a transmission device to perform the following steps: displaying a set of image frames on the presentation screen of a virtual reality (VR) device or an augmented reality (AR) device, wherein the image frames in the set display the same biological object, and the monitored part of the biological object is the part whose three-dimensional pose needs to be adjusted; performing pose estimation on the biological object in the image frames to predict the initial pose features of the biological object, wherein the initial pose features include at least: the initial three-dimensional pose and initial two-dimensional position of the biological object, and the ground contact state of the monitored part, the ground contact state being used to characterize the contact between the monitored part and a preset object in the image frame; adjusting the three-dimensional pose of the monitored part in the initial three-dimensional pose based on the initial two-dimensional position and the ground contact state to obtain the target pose of the biological object; driving the VR device or AR device to render and display the target pose.

[0154] The processor can invoke information and application programs stored in the memory via a transmission device to perform the following steps: acquiring a set of image frames by calling a first interface, wherein the image frames in the set display the same biological object, and the monitored part of the biological object is the part whose three-dimensional pose needs to be adjusted; estimating the pose of the biological object in the image frames to predict the initial pose features of the biological object, wherein the initial pose features include at least: the initial three-dimensional pose and initial two-dimensional position of the biological object, and the ground contact state of the monitored part, the ground contact state being used to characterize the contact between the monitored part and a preset object in the image frame; adjusting the three-dimensional pose of the monitored part in the initial three-dimensional pose based on the initial two-dimensional position and ground contact state to obtain the target pose of the biological object; and outputting the target pose by calling a second interface, wherein the second interface includes a second parameter, the value of which is the target pose.

[0155] The processor can access information and application programs stored in memory via a transmission device to perform the following steps: obtaining a set of image frames by capturing images of a biological object, wherein the monitored part of the biological object is the part whose three-dimensional pose needs to be adjusted; estimating the pose of the biological object in the image frames to predict the initial pose features of the biological object, wherein the initial pose features include at least: the initial three-dimensional pose and initial two-dimensional position of the biological object, and the ground contact state of the monitored part, the ground contact state being used to characterize the contact between the monitored part and a preset object in the image frame; adjusting the three-dimensional pose of the monitored part in the initial three-dimensional pose based on the initial two-dimensional position and ground contact state to obtain the target pose of the biological object; and driving the virtual object corresponding to the biological object based on the target pose result.

[0156] Those skilled in the art will understand that Figure 14 The structure shown is for illustrative purposes only. The computer terminal can also be a smartphone (such as an Android phone, an iOS phone, etc.), a tablet computer, a mobile internet device (MID), a PAD, and other terminal devices. Figure 14 This does not limit the structure of the aforementioned electronic device. For example, computer terminal 14 may also include components that are more advanced than those described above. Figure 14 The more or fewer components shown (such as network interfaces, display devices, etc.), or having the same Figure 14 The different configurations shown.

[0157] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing the hardware related to the terminal device. The program can be stored in a computer-readable storage medium, which may include: flash drive, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0158] Embodiments of this application also provide a computer-readable storage medium. Optionally, in this embodiment, the computer-readable storage medium can be used to store the program code executed by the attitude estimation method provided in the above embodiments.

[0159] Optionally, in this embodiment, the computer-readable storage medium may be located in any computer terminal in the AR / VR device terminal group in the AR / VR device network, or in any mobile terminal in the mobile terminal group.

[0160] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for performing the following steps: acquiring a set of image frames, wherein the image frames in the set of image frames display the same biological object, and the monitoring part in the biological object is the part whose three-dimensional pose needs to be adjusted; performing pose estimation on the biological object in the image frames to predict the initial pose features of the biological object, wherein the initial pose features include at least: the initial three-dimensional pose and initial two-dimensional position of the biological object, and the ground contact state of the monitoring part, the ground contact state being used to characterize the contact between the monitoring part and a preset object in the image frame; adjusting the three-dimensional pose of the monitoring part in the initial three-dimensional pose based on the initial two-dimensional position and the ground contact state to obtain the target pose of the biological object.

[0161] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for performing the following steps: performing pose estimation on a biological object in an image frame to predict the initial pose features of the biological object, including: using a regression model to perform three-dimensional pose estimation on the biological object in the image frame to obtain an initial three-dimensional pose; and using an extraction model to perform two-dimensional pose estimation on the biological object in the image frame to obtain an initial two-dimensional position and ground contact state.

[0162] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for performing the following steps: using an extraction model to perform two-dimensional pose estimation of biological objects in an image frame to obtain a ground-touching state, including: using an extraction model to perform ground-touching estimation of biological objects in an image frame to obtain a ground-touching region in the image frame, wherein the ground-touching region is used to characterize the area in the image frame where the monitored part comes into contact with a preset object; and recognizing the image of the ground-touching region in the image frame to obtain the ground-touching state.

[0163] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for performing the following steps: adjusting the three-dimensional pose of the monitored part in the initial three-dimensional pose based on the initial two-dimensional position and the ground contact state to obtain the target pose of the object, including: adjusting the three-dimensional pose of at least one part of the biological object in the initial three-dimensional pose based on the initial two-dimensional position to obtain an intermediate three-dimensional pose, wherein at least one part in the intermediate three-dimensional pose coincides with at least one part in the two-dimensional position; adjusting the position data of the three-dimensional pose of the monitored part in the intermediate three-dimensional pose based on the ground contact state to obtain the target pose.

[0164] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for performing the following steps: adjusting the three-dimensional pose of at least one part of a biological object in an initial three-dimensional pose based on a two-dimensional position to obtain an intermediate three-dimensional pose, including: performing a projection transformation on the initial three-dimensional pose to obtain a target two-dimensional position of the biological object; determining the distance between the target position of at least one part in the target two-dimensional position and the initial position of at least one part in the initial two-dimensional position; and adjusting the three-dimensional pose of at least one part in the initial three-dimensional pose at least once based on the distance to obtain an intermediate three-dimensional pose.

[0165] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for performing the following steps: adjusting the position data of the three-dimensional pose of the monitoring part in the intermediate three-dimensional pose based on the ground contact state to obtain a target pose, including: when the ground contact state is that the monitoring part is in contact with a preset object, determining a first position of the monitoring part and a second position of the preset object in the image frame based on the intermediate three-dimensional pose; adjusting the height data of the monitoring part in the intermediate three-dimensional pose and the three-dimensional pose of the preset part based on the first position and the second position to obtain a target pose, wherein the preset part is a part of the biological object connected to the monitoring part.

[0166] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for performing the following steps: determining a first position of the monitored part and a second position of the preset object in an image frame based on an intermediate three-dimensional pose, including: determining the first position based on the intermediate three-dimensional pose; determining the ground contact position in the image frame based on the first position, wherein the ground contact position is used to characterize the position where the monitored part contacts the preset object; and determining the second position based on the ground contact position in the image frame.

[0167] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for performing the following steps: adjusting the height data of the monitored part in the intermediate three-dimensional pose and the three-dimensional pose of the preset part based on the first position and the second position to obtain a target pose, including: unifying the height data of the monitored part in the intermediate three-dimensional pose based on the first position to obtain an adjusted three-dimensional pose; adjusting the three-dimensional pose of the preset part in the adjusted three-dimensional pose based on the second position to obtain a target pose, wherein the positions of the monitored parts in the target pose are the same.

[0168] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for performing the following steps: in response to an input command applied to the operation interface, displaying a set of image frames on the operation interface, wherein the image frames in the image frame set display the same biological object, and the monitoring part in the biological object is the part whose three-dimensional pose needs to be adjusted; in response to a pose estimation command applied to the operation interface, displaying a target pose estimate of the biological object on the operation interface, wherein the target pose is obtained by adjusting the gap problem of the monitoring part to better focus on the initial two-dimensional position of the biological object and the ground contact state of the monitoring part, the initial three-dimensional pose, the initial two-dimensional position, and the ground contact state are predicted by the pose estimation of the biological object in the image frame, and the ground contact state is used to characterize the contact between the monitoring part and the preset object in the image frame.

[0169] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for performing the following steps: displaying a set of image frames on the presentation screen of a virtual reality (VR) device or an augmented reality (AR) device, wherein the image frames in the image frame set display the same biological object, and the monitoring part of the biological object is the part whose three-dimensional pose needs to be adjusted; performing pose estimation on the biological object in the image frames to predict the initial pose features of the biological object, wherein the initial pose features include at least: the initial three-dimensional pose and initial two-dimensional position of the biological object, and the ground contact state of the monitoring part, the ground contact state being used to characterize the contact between the monitoring part and a preset object in the image frame; adjusting the three-dimensional pose of the monitoring part in the initial three-dimensional pose based on the initial two-dimensional position and the ground contact state to obtain the target pose of the biological object; and driving the VR device or AR device to render and display the target pose.

[0170] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for performing the following steps: obtaining a set of image frames by calling a first interface, wherein the image frames in the image frame set display the same biological object, and the monitoring part in the biological object is the part whose three-dimensional pose needs to be adjusted; performing pose estimation on the biological object in the image frames to predict the initial pose features of the biological object, wherein the initial pose features include at least: the initial three-dimensional pose and initial two-dimensional position of the biological object, and the ground contact state of the monitoring part, the ground contact state being used to characterize the contact between the monitoring part and a preset object in the image frame; adjusting the three-dimensional pose of the monitoring part in the initial three-dimensional pose based on the initial two-dimensional position and the ground contact state to obtain the target pose of the biological object; and outputting the target pose by calling a second interface, wherein the second interface includes a second parameter, and the parameter value of the second parameter is the target pose.

[0171] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for performing the following steps: obtaining a set of image frames by photographing a biological object, wherein the monitored part in the biological object is the part whose three-dimensional pose needs to be adjusted; performing pose estimation on the biological object in the image frames to predict the initial pose features of the biological object, wherein the initial pose features include at least: the initial three-dimensional pose and initial two-dimensional position of the biological object, and the ground contact state of the monitored part, the ground contact state being used to characterize the contact between the monitored part and a preset object in the image frame; adjusting the three-dimensional pose of the monitored part in the initial three-dimensional pose based on the initial two-dimensional position and the ground contact state to obtain the target pose of the biological object; and driving the virtual object corresponding to the biological object based on the target pose result.

[0172] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0173] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0174] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0175] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0176] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0177] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A pose estimation method, characterized in that, include: Obtain a set of image frames, wherein the image frames in the set display the same biological object, and the monitoring part of the biological object is the part whose three-dimensional pose needs to be adjusted; The biological object in the image frame is subjected to pose estimation to obtain the initial pose features of the biological object. The initial pose features include at least: the initial three-dimensional pose and initial two-dimensional position of the biological object, and the ground contact state of the monitored part. The ground contact state is used to characterize whether the monitored part is in contact with a preset object in the image frame. Based on the initial two-dimensional position and the ground contact state, the three-dimensional pose of the monitored part in the initial three-dimensional pose is adjusted to obtain the target pose of the biological object. Wherein, in the ground contact state, the monitored part is in contact with the preset object, the target pose is obtained by adjusting the height data of the monitored part and the three-dimensional pose of the preset part in the intermediate three-dimensional pose based on the ground contact state, the first position of the monitored part, and the second position of the preset object. The preset part is a part of the biological object connected to the monitored part. The first and second positions are determined based on the intermediate three-dimensional pose, which is obtained by adjusting the three-dimensional pose of at least one part of the biological object in the initial three-dimensional pose.

2. The method according to claim 1, characterized in that, Pose estimation is performed on the biological object in the image frame to obtain the initial pose features of the biological object, including: The biological object in the image frame is estimated in three dimensions using a regression model to obtain the initial three-dimensional pose; The biological object in the image frame is estimated in two dimensions using an extraction model to obtain the initial two-dimensional position and the ground contact state.

3. The method according to claim 2, characterized in that, The biological object in the image frame is subjected to two-dimensional pose estimation using an extraction model to obtain the ground-touching state, including: The extraction model is used to estimate the ground contact area of ​​the biological object in the image frame to obtain the ground contact area in the image frame, wherein the ground contact area is used to characterize the area in the image frame where the monitored part comes into contact with the preset object; The ground-touching state is obtained by identifying the ground-touching area in the image frame.

4. The method according to claim 1, characterized in that, Based on the initial two-dimensional position and the ground contact state, the three-dimensional pose of the monitored part in the initial three-dimensional pose is adjusted to obtain the target pose of the biological object, including: Based on the initial two-dimensional position, the three-dimensional pose of at least one part of the biological object in the initial three-dimensional pose is adjusted to obtain an intermediate three-dimensional pose, wherein the at least one part in the intermediate three-dimensional pose coincides with the at least one part in the two-dimensional position. Based on the ground contact state, the position data of the three-dimensional pose of the monitored part in the intermediate three-dimensional pose are adjusted to obtain the target pose.

5. The method according to claim 4, characterized in that, Based on the two-dimensional position, the three-dimensional pose of at least one part of the biological object in the initial three-dimensional pose is adjusted to obtain an intermediate three-dimensional pose, including: The initial three-dimensional pose is projected and transformed to obtain the target two-dimensional position of the biological object; Determine the distance between the target position of at least one part in the target two-dimensional position and the initial position of at least one part in the initial two-dimensional position; Based on the distance, the three-dimensional pose of at least one part in the initial three-dimensional pose is adjusted at least once to obtain the intermediate three-dimensional pose.

6. The method according to claim 1, characterized in that, The method further includes: Based on the intermediate three-dimensional pose, the first position is determined; Based on the first position, the ground contact position in the image frame is determined, wherein the ground contact position is used to characterize the position where the monitored part comes into contact with the preset object; The second position is determined based on the ground contact position in the image frame.

7. The method according to claim 1, characterized in that, The method further includes: Based on the first position, the height data of the monitored part in the intermediate three-dimensional pose are unified to obtain the adjusted three-dimensional pose. Based on the second position, the three-dimensional pose of the preset part in the adjusted three-dimensional pose is adjusted to obtain the target pose, wherein the position of the monitored part in the target pose is the same.

8. A pose estimation method, characterized in that, include: In response to input commands applied to the operation interface, a set of image frames is displayed on the operation interface, wherein the image frames in the set of image frames display the same biological object, and the monitoring part of the biological object is the part whose three-dimensional pose needs to be adjusted; In response to a pose estimation command applied to the operation interface, a target pose estimate of the biological object is displayed on the operation interface. The target pose is obtained by adjusting the three-dimensional pose of the monitoring part in the initial three-dimensional pose of the biological object using the initial two-dimensional position of the biological object and the ground contact state of the monitoring part. The initial three-dimensional pose, the initial two-dimensional position, and the ground contact state are obtained by pose estimation of the biological object in the image frame. The ground contact state characterizes whether the monitoring part is in contact with a preset object in the image frame. When the ground contact state indicates that the monitoring part is in contact with the preset object, the target pose is obtained by adjusting the height data of the monitoring part and the three-dimensional pose of the preset part in the intermediate three-dimensional pose based on the ground contact state, the first position of the monitoring part, and the second position of the preset object. The preset part is a part of the biological object connected to the monitoring part. The first and second positions are determined based on the intermediate three-dimensional pose, which is obtained by adjusting the three-dimensional pose of at least one part of the biological object in the initial three-dimensional pose.

9. A posture estimation method, characterized in that, include: A set of image frames is displayed on the screen of a virtual reality (VR) device or an augmented reality (AR) device, wherein the image frames in the set of image frames display the same biological object, and the monitoring part of the biological object is the part whose three-dimensional pose needs to be adjusted. The biological object in the image frame is subjected to pose estimation to obtain the initial pose features of the biological object. The initial pose features include at least: the initial three-dimensional pose and initial two-dimensional position of the biological object, and the ground contact state of the monitored part. The ground contact state is used to characterize whether the monitored part is in contact with a preset object in the image frame. Based on the initial two-dimensional position and the ground contact state, the three-dimensional pose of the monitored part in the initial three-dimensional pose is adjusted to obtain the target pose of the biological object. Wherein, in the ground contact state, the monitored part is in contact with the preset object, the target pose is obtained by adjusting the height data of the monitored part and the three-dimensional pose of the preset part in the intermediate three-dimensional pose based on the ground contact state, the first position of the monitored part, and the second position of the preset object. The preset part is a part of the biological object connected to the monitored part. The first and second positions are determined based on the intermediate three-dimensional pose, which is obtained by adjusting the three-dimensional pose of at least one part of the biological object in the initial three-dimensional pose. The VR device or AR device is driven to render and display the target pose.

10. A pose estimation method, characterized in that, include: A set of image frames is obtained by calling the first interface. The image frames in the set of image frames display the same biological object, and the monitoring part of the biological object is the part whose three-dimensional pose needs to be adjusted. The biological object in the image frame is subjected to pose estimation to obtain the initial pose features of the biological object. The initial pose features include at least: the initial three-dimensional pose and initial two-dimensional position of the biological object, and the ground contact state of the monitored part. The ground contact state is used to characterize whether the monitored part is in contact with a preset object in the image frame. Based on the initial two-dimensional position and the ground contact state, the three-dimensional pose of the monitored part in the initial three-dimensional pose is adjusted to obtain the target pose of the biological object. Wherein, in the ground contact state, the monitored part is in contact with the preset object, the target pose is obtained by adjusting the height data of the monitored part and the three-dimensional pose of the preset part in the intermediate three-dimensional pose based on the ground contact state, the first position of the monitored part, and the second position of the preset object. The preset part is a part of the biological object connected to the monitored part. The first and second positions are determined based on the intermediate three-dimensional pose, which is obtained by adjusting the three-dimensional pose of at least one part of the biological object in the initial three-dimensional pose. The target pose is output by calling a second interface, wherein the second interface includes a second parameter, and the parameter value of the second parameter is the target pose.

11. A method for driving virtual objects, characterized in that, include: A set of image frames is obtained by photographing a biological object, wherein the monitored part of the biological object is the part whose three-dimensional pose needs to be adjusted; The biological object in the image frame is subjected to pose estimation to obtain the initial pose features of the biological object. The initial pose features include at least: the initial three-dimensional pose and initial two-dimensional position of the biological object, and the ground contact state of the monitored part. The ground contact state is used to characterize whether the monitored part is in contact with a preset object in the image frame. Based on the initial two-dimensional position and the ground contact state, the three-dimensional pose of the monitored part in the initial three-dimensional pose is adjusted to obtain the target pose of the biological object. Wherein, in the ground contact state, the monitored part is in contact with the preset object, the target pose is obtained by adjusting the height data of the monitored part and the three-dimensional pose of the preset part in the intermediate three-dimensional pose based on the ground contact state, the first position of the monitored part, and the second position of the preset object. The preset part is a part of the biological object connected to the monitored part. The first and second positions are determined based on the intermediate three-dimensional pose, which is obtained by adjusting the three-dimensional pose of at least one part of the biological object in the initial three-dimensional pose. The virtual object corresponding to the biological object is driven based on the target pose result.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored program, wherein, when the program is executed, it controls the device on which the computer-readable storage medium is located to perform the method according to any one of claims 1 to 11.

13. An electronic device, characterized in that, include: processor; A memory, connected to the processor, is used to provide the processor with instructions to perform the following processing steps: acquiring a set of image frames, wherein the image frames in the set display the same biological object, and the monitoring part of the biological object is the part whose three-dimensional pose needs to be adjusted; performing pose estimation on the biological object in the image frames to obtain the initial pose features of the biological object, wherein the initial pose features include at least: the initial three-dimensional pose and initial two-dimensional position of the biological object, and the ground contact state of the monitoring part, the ground contact state being used to characterize whether the monitoring part is in contact with a preset object in the image frame; and adjusting the initial three-dimensional pose based on the initial two-dimensional position and the ground contact state. The three-dimensional pose of the monitored part in the pose is adjusted to obtain the target pose of the biological object. In the ground-contact state, where the monitored part is in contact with the preset object, the target pose is obtained by adjusting the height data of the monitored part and the three-dimensional pose of the preset part in the intermediate three-dimensional pose based on the ground-contact state, the first position of the monitored part, and the second position of the preset object. The preset part is the part of the biological object connected to the monitored part. The first position and the second position are determined based on the intermediate three-dimensional pose. The intermediate three-dimensional pose is obtained by adjusting the three-dimensional pose of at least one part of the biological object in the initial three-dimensional pose.

Images