3D pose prediction apparatus and method

Abstract

To provide a 3D posture prediction device and method capable of minimizing an information loss in an encoding process to more precisely predict a 3D posture in predicting a posture of a user on the basis of posture information obtained by directly observing the user from a viewpoint of a first person.SOLUTION: A method includes: using a heat map estimator 210 to generate an extremity heat map and an articulation heat map from a plurality of image data 10, 20; inputting the articulation heat map to a grid heat map encoder to extract an articulation feature vector; propagating a relation feature vector between adjacent articulations generated on the basis of the articulation feature vector and the extremity heat map through a propagation network having a skeleton tree hierarchical structure to output a propagation feature vector; and predicting a 3D posture of a user on the basis of the propagation feature vector and the articulation feature vector.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The embodiments disclosed herein relate to a 3D posture prediction device and method, and more particularly, to a 3D posture prediction device and method that can minimize information loss during the encoding process and more accurately predict a 3D posture when predicting a user's posture based on posture information obtained by directly observing the user from a first-person perspective.

[0002] This research was conducted as a result of the "Artificial Intelligence Graduate School Support (Seoul National University)" project (IITP-2021-0-01343) under the Ministry of Science and ICT and the Institute for Information and Communications Technology Planning (IITP)'s ICT Broadcasting Innovation Talent Development Project.

[0003] This research was carried out as a result of the "Ultra-Realistic Continuous Hybrid Telepresence Platform" project (NRF-2022R1A2C3008495) of the Individual Basic Research Project and the "Foundation Model for Understanding 3D Human-Space Interactions" project (NRF-00218601) of the Group Research Support Project of the Ministry of Science and ICT and the National Research Foundation of Korea (NRF). [Background technology]

[0004] With the development of deep neural network technology, the market for virtual reality and augmented reality devices and software is developing rapidly.

[0005] Here, a head mounted display (hereinafter referred to as HMD) is a device that a user wears on their head to experience virtual reality and augmented reality, and technology that uses an internally mounted camera to precisely reproduce the user's movements is applied to virtual reality and augmented reality.

[0006] Meanwhile, to estimate a user's posture using information observed from a first-person perspective, the user directly observes their own body from a first-person perspective using the internal camera of the HMD, and the observed information can be used for 3D posture prediction. According to conventional technology, a heatmap, which is a probabilistic representation of 2D posture, is obtained from image data containing observed body information, and this is compressed at once using an encoder structure based on a convolutional neural network (CNN), and 3D posture is predicted using a decoder network that infers posture from the compressed information.

[0007] However, according to the above-mentioned conventional technology, the probabilistic information of the heat map is lost during the process of compressing the information by the encoder, and when a user observes body parts from a first-person perspective, parts far from the camera are decoded at once without taking into consideration that these parts are inaccurate because they are blocked by other body parts. This results in a problem of lower accuracy of pose prediction compared to pose prediction technology using a third-person camera.

[0008] Therefore, there is a need for a technique that can provide higher prediction accuracy when predicting 3D pose based on body information observed from a first-person perspective.

[0009] On the other hand, the above-mentioned background art is technical information that the inventor possessed for the purpose of deriving the present invention or acquired in the process of deriving the present invention, and it cannot necessarily be said to be publicly known art that was disclosed to the general public prior to the filing of the present invention. [Prior art documents] [Patent documents]

[0010] [Patent Document 1] Korean Patent Publication No. 10-2023-0066309 Summary of the Invention [Problem to be solved by the invention]

[0011] The embodiments disclosed herein aim to provide a 3D posture prediction device and method that can minimize information loss during the encoding process and predict 3D posture more accurately when predicting a user's posture based on posture information directly observed from a first-person perspective.

[0012] Other objects and advantages of the present invention can be understood from the following description and will become more apparent from the following examples. Also, it will be readily apparent that the objects and advantages of the present invention can be realized by the means and combinations thereof as claimed. [Means for solving the problem]

[0013] As a technical means for achieving the above technical object, a 3D posture prediction device includes an input / output unit for receiving a plurality of image data observed from a first-person perspective of a user's body parts and outputting the results of arithmetic processing of the image data; a memory for storing a program for executing a 3D posture prediction method; and a control unit for predicting the user's 3D posture based on the image data received through the input / output unit by executing the program, wherein the control unit generates limb heatmaps and joint heatmaps from the plurality of image data using a heatmap estimator, inputs the joint heatmaps to a grid heatmap encoder to extract joint feature vectors, propagates the joint feature vectors and relationship feature vectors between adjacent joints generated based on the limb heatmaps through a propagation network with a skeletal tree hierarchical structure, and outputs propagation feature vectors, and predicts the user's 3D posture based on the propagation feature vectors and the joint feature vectors.

[0014] According to another embodiment, a 3D posture prediction method performed by the 3D posture prediction device includes the steps of receiving input of a plurality of image data observing a user's body parts from a first-person perspective, generating limb heatmaps and joint heatmaps from the plurality of image data using a heatmap estimator, inputting the joint heatmaps to a grid heatmap encoder to extract joint feature vectors, propagating relationship feature vectors between adjacent joints generated based on the joint feature vectors and the limb heatmaps through a propagation network with a skeletal tree hierarchical structure to output propagation feature vectors, and predicting the user's 3D posture based on the propagation feature vectors and the joint feature vectors.

[0015] According to yet another embodiment, the recording medium is a computer-readable recording medium having a program recorded thereon for executing a 3D pose prediction method.

[0016] According to yet another embodiment, the computer program is a computer program recorded on a recording medium for being executed by a 3D pose prediction device to perform a 3D pose prediction method. [Effects of the Invention]

[0017] According to any one of the above-mentioned solutions, multiple heat maps obtained from image data containing body information are combined into a single image in a pixel-dimensional grid format and processed, thereby enabling efficient encoding by allowing a wide pixel area to exchange information simultaneously during encoding, thereby minimizing information loss that may occur during the encoding process and enabling more accurate 3D posture prediction.

[0018] In addition, among the information contained in image data obtained from a first-person perspective, information about body parts close to the camera (information about upper joints) is accurate, but information about body parts far from the camera (information about lower joints) may be inaccurate due to occlusion between bodies. However, by using a propagation network with a skeletal tree hierarchical structure to propagate information about upper joints (e.g., joint feature vectors of upper joints and relationship feature vectors between adjacent joints) to lower joints, it is possible to predict information about lower joints where occlusion occurs (e.g., joint feature vectors of lower joints), thereby enabling more accurate 3D posture prediction.

[0019] The effects obtained by the disclosed embodiments are not limited to those mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art to which the disclosed embodiments pertain in the following description.

[0020] The accompanying drawings below illustrate preferred embodiments disclosed in this specification, and serve to facilitate a better understanding of the technical ideas disclosed in this specification, along with specific content for implementing the invention. Therefore, the content disclosed in this specification should not be interpreted as being limited to only the matters shown in the drawings. [Brief explanation of the drawings]

[0021] [Figure 1] FIG. 1 is a functional block diagram of a 3D pose prediction device according to an embodiment. [Figure 2] 1 is an exemplary diagram illustrating a 3D pose prediction device according to an embodiment; [Figure 3] 1 is an exemplary diagram illustrating a 3D pose prediction device according to an embodiment; [Figure 4] 1 is a flowchart of a 3D pose prediction method performed by a 3D pose prediction device according to an embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0022] Various embodiments will be described in detail below with reference to the accompanying drawings. The embodiments described below may be implemented in various different forms. In order to more clearly describe the features of the embodiments, detailed descriptions of matters that are well known to those skilled in the art to which the following embodiments pertain will be omitted. In addition, parts in the drawings that are not relevant to the description of the embodiments will be omitted, and similar parts will be designated by similar reference numerals throughout the specification.

[0023] Throughout the specification, when a certain component is said to be connected to another component, this includes not only direct connection but also connection via an intermediate component. Furthermore, when a certain component includes another component, unless otherwise specified, it does not mean that the other component is excluded, but that the other component can also be included.

[0024] Hereinafter, the embodiments will be described in detail with reference to the accompanying drawings.

[0025] FIG. 1 is a functional block diagram of a 3D pose prediction device according to an embodiment, and FIGS. 2 and 3 are exemplary diagrams for explaining the 3D pose prediction device according to an embodiment.

[0026] The 3D pose prediction device may be implemented by an electronic device 100. Such an electronic device 100 may include a user terminal or may include a user terminal and a server. For example, the electronic device 100 may implement a method according to the embodiments disclosed herein by a user terminal or by a server-client system.

[0027] Here, according to an embodiment, the user terminal may be embodied as a computer, a portable terminal, a television, a wearable device, etc. that can be connected to a remote server via a network or can be connected to other terminals and servers. Here, the computer may include, for example, a notebook PC, a desktop PC, or a laptop PC equipped with a web browser, and the portable terminal may include, for example, a wireless communication device that ensures portability and mobility, such as a Personal Communication System (PCS), a Personal Digital Cellular (PDC), a Personal Handyphone System (PHS), a Personal Digital Assistant (PDA), a Global System for Mobile communications (GSM), an International Mobile Telecommunication (IMT)-2000, a Code Division Multiple Access (CDMA)-2000, a W-Code Division Multiple Access (W-CDMA), a Wireless Broadband Internet (Wibro), a smartphone, a Mobile Worldwide Interoperability for Microwave Access (Mobile WiMAX), etc. The television may include Internet Protocol Television (IPTV), Internet Television, terrestrial TV, cable TV, etc. Furthermore, a wearable device is a type of information processing device that can be worn directly on the human body, such as a watch, glasses, accessory, clothing, or footwear, and can be connected to a remote server or other terminal via a network, either directly or through another information processing device.

[0028] According to one embodiment, the electronic device 100 is implemented as a server-client system, and the user terminal included in the electronic device 100 is a wearable device and may be implemented as an HMD that displays a virtual environment image to a user and changes the image according to the user's movements. Such a user terminal may have a built-in / attached camera to acquire a camera view image showing the real world and communicate with a server.

[0029] The electronic device 100 may observe various body parts of the user through an input / output unit such as an internally mounted camera and generate the observed various body parts as a plurality of image data. Here, the generated plurality of image data may be image data including information on the body parts of the user observed in various directions from a first-person perspective.

[0030] The electronic device 100 may receive a plurality of image data obtained by observing a user's body parts from a first-person perspective and generate limb heatmaps and joint heatmaps from the plurality of image data using a heatmap estimator. Here, a heatmap represents a probabilistic 2D pose for each pixel from a camera perspective, and according to an embodiment, two types of heatmaps for joints and limbs may be used. A limb heatmap may represent a limb pose as a stereo pair, and a joint heatmap may represent a joint pose as a stereo pair. Here, the joint heatmap follows the standard definition, where pixel values ​​indicate the probability that a joint is at a given 2D coordinate (see Jonathan Tompson, Ross Goroshin, Arjun Jain, Yann Le-Cun, and Christoph Bregler (2015). Efficient object localization using convolutional networks. In CVPR, pages 648-656. IEEE Computer Society). The limb heatmap has two channels and can be used to derive a relationship feature vector between both joints. The limb heatmap can also show 3D information along with limb visibility using lines connecting joints (see Taeho Kang, Kyungjin Lee, Jinrui Zhang, and Youngki Lee (2023). Ego3dpose: Capturing 3D cues from binocular egocentric views. In SIGGRAPH Asia 2023 Conference Papers, New York, NY, USA. Association for Computing Machinery).

[0031] The electronic device 100 may input the joint heatmap to a grid heatmap encoder (Grid ViT (Vision Transformer) Encoder) to extract a joint feature vector. According to an embodiment, the grid heatmap encoder may receive the joint heatmap and generate a feature embedding for each joint. That is, the grid heatmap encoder may receive the joint heatmap and extract a joint feature vector for each joint.

[0032] The electronic device 100 may propagate the extracted joint feature vector and the relationship feature vector between adjacent joints generated based on the limb heat map through a propagation network to output a propagation feature vector. Here, the propagation network may have a skeletal tree hierarchy structure. According to an embodiment, the skeletal tree hierarchy structure may be in the form of a tree in which the head is the root and each joint is connected.

[0033] The electronic device 100 can predict the 3D pose of the user based on the propagation feature vector and the joint feature vector.

[0034] Hereinafter, the 3D pose prediction device refers to the electronic device 100 described above.

[0035] Referring to FIG. 1, a 3D pose prediction apparatus 100 according to an embodiment includes an input / output unit 110, a memory 120, a communication unit 130, and a control unit 140.

[0036] The input / output unit 110 may include an input unit for receiving input from a user and an output unit for displaying information such as a result of performing a task or a status of the 3D posture prediction device 100. For example, the input / output unit 110 may include an operation panel for receiving user input and a display panel for displaying a screen.

[0037] Specifically, the input unit may include devices capable of receiving various types of user input, such as a keyboard, physical buttons, a touch screen, a camera, a microphone, a sensor, etc. The output unit may include a display panel, a speaker, etc. However, the input / output unit 110 is not limited thereto and may include configurations supporting various inputs and outputs.

[0038] According to the embodiment, when the 3D posture prediction device 100 is implemented as an HMD and worn on the user's head, the input / output unit 110 may receive a plurality of image data obtained by observing various body parts of the user from a first-person perspective. For example, if the input unit is a camera, the camera may capture images of various body parts of the user to obtain the image data.

[0039] Various types of data, such as files, applications, and programs, may be embedded and stored in the memory 120. The controller 140 may access and use data stored in the memory 120, or may store new data in the memory 120. The controller 140 may also execute programs embedded in the memory 120. Referring to FIG. 1, the memory 120 may be embedded with a program for executing a 3D pose prediction method.

[0040] According to one embodiment, when an input requesting the start of a program is received from a user via the input / output unit 110, the control unit 140 may execute a program stored in the memory 120 to perform a 3D pose prediction method.

[0041] The communication unit 130 may perform wired or wireless communication with other devices or networks. To this end, the communication unit 130 may include a communication module that supports at least one of various wired or wireless communication methods. For example, the communication module may be implemented in the form of a chipset.

[0042] The wireless communication supported by the communication unit 130 may be, for example, Wi-Fi (Wireless Fidelity), Wi-Fi Direct, Bluetooth (registered trademark), UWB (Ultra Wide Band), or NFC (Near Field Communication), etc. The wired communication supported by the communication unit 130 may be, for example, USB or HDMI (High Definition Multimedia Interface), etc.

[0043] The control unit 140 includes at least one processor such as a CPU, a GPU, an Arduino, etc., and can control the overall operation of the 3D pose prediction apparatus 100. That is, the control unit 140 can control other components included in the 3D pose prediction apparatus 100 to perform operations for 3D pose prediction.

[0044] For example, the control unit 140 may execute a program stored in the memory 120 , read a file stored in the memory 120 , or store a new file in the memory 120 .

[0045] Hereinafter, a process in which the control unit 140 executes a program stored in the memory 120 to perform a 3D pose prediction method according to an embodiment will be described in detail.

[0046] The control unit 140 can generate limb heatmaps and joint heatmaps from a plurality of image data using a heatmap estimator.

[0047] For example, the control unit 140 may

[0048]

number

[0049]

number

[0050]

number

[0051] FIG. 2 illustrates a schematic structure of the 3D pose prediction device 100, which includes a grid heatmap encoder 220 and a propagation network 230. The grid heatmap encoder 220 receives a joint heatmap as input and generates feature embeddings for each joint. The propagation network 230 recognizes the skeletal structure and processes the generated embeddings to accurately predict the 3D pose. In this embodiment, embedding refers to converting raw data into a vector. For example, generating feature embeddings may refer to generating feature vectors. The joint feature vectors for each joint are propagated through a skeletal hierarchy. The skeletal hierarchy can be represented as a tree structure with the root representing the head. FIG. 2 illustrates a simplified skeleton (T) showing propagation from the head to the hands. Feature propagation utilizes a propagation unit (PU) 250.

[0052] Referring to FIG. 2, the control unit 140 receives input of multiple image data 10, 20 observing various body parts of the user from a first-person perspective, and can generate limb heat maps and joint heat maps from this using a heat map estimator 210.

[0053] Here, the heatmap estimator 210 can be trained using a script for a pre-input dataset, and its basic configuration uses a known optimizer (see the paper: Diederik Kingma and Jimmy Ba (2015). Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR), San Diego, CA, USA.) Since the heatmap estimator 210 is a known technique, a detailed description thereof will be omitted.

[0054] The control unit 140 may input the joint heatmap to the grid heatmap encoder 220 to extract a joint feature vector.

[0055] More specifically, the control unit 140 may combine the joint heat maps into a single image by connecting them in a grid form, divide the combined image into patches, which are joint heat maps for each joint, and extract joint feature vectors by encoding the divided patches.

[0056] In other words, all the joint heatmaps can be combined into a large single grid image, and the grid combined image can be divided into patches.

[0057] For example, a 64x64 joint heatmap can be saved as a single image with a resolution of 384x384. The image is a 6x6 grid, consisting of 36 regions. The joint heatmap fills the grid in order. Meanwhile, regions that do not correspond to the joint heatmap are masked by the Transformer Encoder and do not affect the output.

[0058] Referring to FIG. 2, the control unit 140 may linearly project the grid divided into patches, i.e., the joint heat map, to create an input embedding and input it to a Transformer Encoder. Here, the Transformer Encoder may include multi-head attention. Here, the Transformer Encoder according to the embodiment may be a known encoder (see, for example, Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby (2021). An image is worth 16x16 words: Transformers for image recognition at scale. ICLR.) with the Special Classification (CLS) token removed.

[0059] According to an embodiment, a Transformer Encoder can take an input embedding Z and output an output feature embedding Z'.

[0060] The transformer encoder is 32N. J patch,

[0061]

number

[0062] According to an embodiment, a patch-based communication with self-attention is used to maintain consistency between the joint heatmap and the feature embeddings, so the encoding process preserves the correspondence between the patches and the output input feature embeddings. That is, the output feature embeddings corresponding to the individual input patches are concatenated and encoded to generate the joint feature vector for the joint heatmap.

[0063] In this way, unlike a CNN encoder in which communication occurs within nearby pixels of different heatmaps, the grid heatmap encoder 220 according to the embodiment is capable of communication between heatmap patches that are spatially far apart, and therefore can share features without downsampling, thereby minimizing information loss.

[0064] Meanwhile, the process of the grid heatmap encoder 220 can be formulated as follows:

[0065]

number

[0066] The joint heatmaps are arranged as a single grid image. The aligned images are then arranged into a total of 4 × 4 × 2N J patches

[0067]

number

[0068]

number

[0069]

number

[0070]

number

[0071]

number

[0072] Here, W represents a learnable projection matrix, which may be a matrix in the form of a general layer in a deep learning network with non-linear parts removed. According to an embodiment, the patch may be encoded into a feature vector of a desired size using the above matrix. X represents pixels generated by dividing the joint heatmap by the patch, and may have a size of 4x4. p represents a learnable position encoding. In the transformer encoder structure according to an embodiment, since it does not receive position information as input but only the value of the patch itself, a position encoding value can be added to determine the position information.

[0073]

number

[0074]

number

[0075]

number

[0076]

number

[0077]

number

[0078]

number

[0079] Assuming that the 2i-1 and 2i-th heatmaps correspond to the i-th joint, the joint heatmap can be formulated as shown in Equation 17 below.

[0080]

number

[0081] The control unit 140 can propagate the relationship feature vector between adjacent joints generated based on the joint feature vector and the limb heat map through a propagation network with a skeletal tree hierarchical structure, and output the propagation feature vector.

[0082] Here, the propagation network can be composed of a two-layer propagation unit including a relational feature encoder that extracts relational feature vectors between adjacent joints using limb heat maps, and a Long Short-Term Memory (LSTM) structure that processes the propagation process for joint feature vectors and relational feature vectors.

[0083] Referring to FIG. 2, the propagation network 230 may include a relational feature encoder 240 and a propagation unit 250 .

[0084] The relational feature encoder 240 can output a relational feature vector between joints using the estimated limb heat map. Here, the estimated limb heat map can be flattened to create a vector of a specific dimension for input to the MLP layer. The propagation unit 250 can process a propagation process to generate a state of a sub-joint (child joint) using the upper state (parent state), relational features, and joint features of the sub-joint (child joint) as input. Here, the joint features are propagated from the head, to which the camera is directly connected, to the limbs via a skeletal tree hierarchical structure. During the propagation process, whether to reflect information about the upper joint (parent joint) can be flexibly determined depending on the reliability of the features of the upper joint (parent joint) and the sub-joint (child joint) by the propagation unit.

[0085] According to an embodiment, the limb heat map may include 3D information including a trigonometric function of the camera's viewing angle, and thus may be used to provide information about the connection between a higher joint (parent joint) and a lower joint (child joint).

[0086] On the other hand, a relational feature encoder E, which is composed of fully connected layers, R is the limb heatmap

[0087]

number

[0088] into limb features, where the limb features are encoded as relational feature vectors F R are linked to form H L,2i-1 and H L,2i Assuming that corresponds to a limb heat map connecting the i-th joint and its upper joint (parent joint), the process of providing information about the connection between the upper joint (parent joint) and the lower joint (upper joint), i.e., the process of generating a relational feature vector, can be formulated as follows:

[0089]

number

[0090] where E L is a relational feature encoder that can convert a limb heatmap into a one-dimensional vector and output the encoding result vector as an MLP. Here, an MLP can have a structure in which each layer is stacked on top of another layer, and the results of each layer are connected to the input of the upper layer. An MLP can stack many layers in succession.

[0091] Meanwhile, referring to FIGS. 2 and 3, the propagation unit 250 according to the embodiment can be configured in two layers.

[0092] The propagation unit 250 according to an embodiment can assign weights to the parent's hidden state and relational features to the articulation features.

[0093] According to the embodiment, in the skeletal tree hierarchy structure, parent(i) denotes the index of the upper joint (parent joint), and PropagationNet((H,C),R,J) denotes the propagation network. Here, the propagation network takes two PU layers, H=[h1,h2], C=[c1,c2], the hidden and cell states for the relation feature R and the joint feature J, and generates the i-th joint heatmap H i , C i The hidden and cell states for can be formulated as follows:

[0094]

number

[0095]

number

[0096]

number

[0097] On the other hand, as shown in Figure 2, the head, which is the root indicating the highest joint, is not seen from the egocentric perspective, i.e., the first-person perspective, so it has no features and is indexed to 0 and initialized to a 0 vector (Zero Init).

[0098]

number

[0099] is the propagation network h 2,i This is the hidden state from the second layer.

[0100] According to an embodiment, the propagation network can estimate the position of a lower joint (child joint) using the position of a higher joint (parent joint) and the joint relationship. Meanwhile, in addition to the LSTM structure, the propagation unit may further include a forget gate (not shown) that enables the joint features and relational features of the upper joint that are propagated based on the joint features of the lower joint to be ignored.

[0101] According to the embodiment, the forget gate ignores the upper joint (parent joint) and related features when the features of the lower joint (child joint) are clear. Therefore, when such a forget gate is included, it can predict the posture only for an ambiguous joint, i.e., a lower joint that is occluded by a higher joint.

[0102] The propagation unit can be formulated as follows:

[0103]

number

[0104]

number

[0105]

number

[0106]

number

[0107] where W can be a weight matrix and b can be a bias vector. JPEG0007827262000028.jpg56 indicates element-wise multiplication, + indicates element-wise addition, and σ indicates sigmoid activation.

[0108] The forget gate can be calculated from the joint features, JPEG0007827262000029.jpg54 and The forget gate controls the hidden state of the upper joint (parent joint) and the relationship between the two joints, and the modified hidden state JPEG0007827262000031.jpg65 and modified related features JPEG0007827262000032.jpg65 can be generated. These modified states and input processed joint features are then used in a standard LSTM structure to apply weights and then non-linearity to the four gates (input, candidate cell state, forgetting, and output).

[0109] According to an embodiment, in the second layer of the propagation network, only the hidden state of the previous layer exists without distinction of relational features or joint features, so the hidden state of the previous layer can be used to forget the hidden state of the upper joint (parent joint) in the current layer.

[0110] The control unit 140 can predict the 3D posture of the user based on the propagation feature vector and the joint feature vector.

[0111] As shown in FIG. 2, according to an embodiment, the output F of the propagation network P,i and the joint features F J,i By concatenating and projecting these, the 3D position of each joint can be estimated and the posture can be predicted.

[0112] FIG. 4 is a flowchart illustrating an example of a method for 3D pose prediction according to an embodiment.

[0113] The 3D pose prediction method according to the embodiment shown in Figure 4 includes time-series processing steps performed by the 3D pose prediction device shown in Figures 1 to 3. Therefore, although omitted below, the above description of the 3D pose prediction device shown in Figures 1 to 3 can also be applied to the 3D pose prediction method according to the embodiment shown in Figure 4.

[0114] Referring to FIG. 4, the 3D posture prediction device 100 receives input of a plurality of image data observing a user's body parts from a first-person perspective, and can generate limb heatmaps and joint heatmaps from the input plurality of image data using a heatmap estimator (S410).

[0115] In operation S420, the 3D pose prediction device 100 may extract a joint feature vector by inputting the joint heatmap to a grid heatmap encoder. Here, the 3D pose prediction device 100 may extract a joint feature vector by connecting the joint heatmaps in a grid form to combine them into a single image, dividing the combined image into patches, which are joint heatmaps for each joint, and encoding the resulting image.

[0116] In operation S430, the 3D pose prediction apparatus 100 may propagate the relationship feature vector between adjacent joints generated based on the joint feature vector and the limb heat map through a propagation network with a skeletal tree hierarchy structure to output a propagation feature vector. Here, the propagation network may be configured with a two-layer propagation unit including a relationship feature encoder that extracts the relationship feature vector between adjacent joints using the limb heat map and an LSTM structure that processes the propagation process for the joint feature vector and the relationship feature vector. In addition to the LSTM structure, the propagation unit may further include a forget gate that enables the joint feature vector and relationship feature vector of a higher joint that have been propagated based on the joint feature vector of a lower joint to be ignored.

[0117] In step S440, the 3D posture prediction device 100 can predict the 3D posture of the user based on the propagation feature vector and the joint feature vector.

[0118] The term 'module' used in the above embodiments refers to software or hardware components such as FPGAs (field programmable gate arrays) or ASICs, and a 'module' performs a certain function. However, 'module' is not limited to software or hardware. A 'module' may be configured to reside on an addressable storage medium or to execute one or more processors. Thus, by way of example, 'module' includes components such as software components, object-oriented software components, class components, and task components, as well as processes, functions, attributes, procedures, subroutines, segments of program code, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, and variables.

[0119] The functionality provided within the components and units may be combined into fewer components and units or separated into additional components and units.

[0120] Furthermore, the components and '~ units' can be implemented to replicate one or more CPUs within a device or a secure multimedia card.

[0121] Meanwhile, the 3D pose prediction method according to an embodiment described herein may also be embodied in the form of a computer-readable medium storing computer-executable instructions and data. Here, the instructions and data may be stored in the form of program code, which, when executed by a processor, may generate a predetermined program module and perform a predetermined operation. Furthermore, the computer-readable medium may be any available medium accessible by a computer, including both volatile and nonvolatile media, and both separable and non-separable media. Furthermore, the computer-readable medium may be a computer recording medium. The computer recording medium may be any volatile and non-volatile, separable and non-separable medium embodied by any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. For example, the computer recording medium may be a magnetic storage medium such as a HDD or SSD, an optical storage medium such as a CD, DVD, or Blu-ray disc, or a memory included in a server accessible via a network.

[0122] Furthermore, the 3D pose prediction method according to an embodiment described herein may be implemented as a computer program (or a computer program product) including computer-executable instructions. The computer program includes programmable machine instructions processed by a processor and may be implemented in a high-level programming language, an object-oriented programming language, an assembly language, a machine language, or the like. The computer program may be recorded on a computer-readable recording medium (e.g., a memory, a hard disk, a magnetic / optical medium, or a solid-state drive (SSD)).

[0123] Therefore, the 3D pose prediction method according to one embodiment described herein can be implemented by executing the above-described computer program on a computing device. The computing device may include a processor, a memory, a storage device, a high-speed interface connecting to the memory and a high-speed expansion port, and at least a portion of a low-speed interface connecting to a low-speed bus and the storage device. Each of these components is connected to each other using various buses and may be mounted on a common motherboard or in other suitable manners.

[0124] Here, the processor may process instructions within a computing device. Such instructions may include instructions stored in a memory or storage device for displaying graphic information to provide a GUI (Graphical User Interface) on an external input and output device, such as a display connected to a high-speed interface. In other embodiments, multiple processors and / or multiple buses may be used, along with multiple memories and memory types, as appropriate. The processor may also be implemented as a chipset, which may include multiple independent analog and / or digital processors.

[0125] Also, memory stores information within a computing device. As an example, memory may be comprised of a volatile memory unit or collection thereof. As another example, memory may be comprised of a non-volatile memory unit or collection thereof. Memory may also be in other forms of computer-readable media, such as, for example, a magnetic or optical disk.

[0126] A storage device can provide a large amount of storage space to a computing device. A storage device can be a computer-readable medium or a configuration that includes such a medium, such as a device in a Storage Area Network (SAN) or other configuration, and can be a floppy disk drive, hard disk drive, optical disk drive, tape drive, flash memory, or other similar semiconductor memory device or device array.

[0127] The above-described embodiments are merely illustrative, and those skilled in the art will understand that the above-described embodiments may be easily modified into other specific forms without changing the technical ideas or essential features of the above-described embodiments. Therefore, it should be understood that the above-described embodiments are illustrative in all respects and are not limiting. For example, each component described as a single component may be implemented in a distributed form, and similarly, each component described as a distributed component may be implemented in a combined form.

[0128] The scope of protection sought by this specification is determined by the claims set forth below rather than the above detailed description, and should be interpreted to include all modifications or variations derived from the meaning and scope of the claims and their equivalent concepts. [Explanation of symbols]

[0129] 100 3D posture predictor 110 Input / output section 120 memory 130 Communications Department 140 Control Unit

Claims

1. an input / output unit for receiving input of a plurality of image data obtained by observing body parts of a user from a first-person perspective and outputting the results of arithmetic processing of the image data; a memory for storing a program for performing a 3D pose prediction method; a control unit that executes the program to predict a 3D posture of a user based on image data received via the input / output unit, The control unit generating limb heatmaps and joint heatmaps from the plurality of image data using a heatmap estimator; inputting the joint heatmap into a grid heatmap encoder to extract a joint feature vector; propagating the joint feature vector and the relationship feature vector between adjacent joints generated based on the limb heat map through a propagation network having a skeletal tree hierarchical structure to output a propagation feature vector; A 3D posture prediction device that predicts a 3D posture of a user based on the propagation feature vector and the joint feature vector.

2. 2. The 3D pose prediction device of claim 1, wherein the control unit extracts a joint feature vector by combining the joint heat maps into a single image by connecting the joint heat maps in a grid form, dividing the combined image into patches, which are joint heat maps for each joint, and encoding the patches.

3. 2. The 3D pose prediction device of claim 1, wherein the propagation network is composed of a two-layer propagation unit including a relational feature encoder that extracts relational feature vectors between adjacent joints using limb heat maps, and an LSTM structure that processes the propagation process for the joint feature vectors and the relational feature vectors.

4. 4. The 3D pose prediction device of claim 3, wherein the propagation unit further comprises, in addition to the LSTM structure, a forget gate that enables the propagation unit to ignore the joint feature vectors and relational feature vectors of the upper joints propagated based on the joint feature vectors of the lower joints.

5. A 3D pose prediction method executed by a 3D pose prediction device, comprising: receiving input of a plurality of image data obtained by observing a user's body parts from a first-person perspective; generating limb heatmaps and joint heatmaps from the plurality of image data using a heatmap estimator; inputting the joint heatmap into a grid heatmap encoder to extract a joint feature vector; propagating the joint feature vector and the relationship feature vector between adjacent joints generated based on the limb heat map through a propagation network having a skeletal tree hierarchical structure to output a propagation feature vector; and predicting a 3D pose of a user based on the propagation feature vector and the joint feature vector.

6. The step of extracting the joint feature vector comprises: combining the joint heat maps into a single image by connecting them in a grid configuration; and extracting joint feature vectors by dividing the combined image into patches, which are joint heat maps for each joint, and encoding the patches.

7. 6. The 3D pose prediction method of claim 5, wherein the propagation network is composed of a two-layer propagation unit including a relational feature encoder that extracts relational feature vectors between adjacent joints using limb heat maps, and an LSTM structure that processes the propagation process for the joint feature vectors and the relational feature vectors.

8. 10. The 3D pose prediction method of claim 7, wherein the propagation unit further comprises, in addition to the LSTM structure, a forget gate that enables the propagation unit to ignore the joint feature vectors and relational feature vectors of the upper joint based on the joint feature vectors of the lower joint.

9. A computer-readable recording medium on which a program for executing the method according to claim 5 is recorded.

10. A computer program recorded on a recording medium for executing the method according to claim 5 when executed by a 3D pose prediction device.

Images