Motion capture method and apparatus based on an artificial neural network model.

Abstract

This invention provides a motion capture method and apparatus using an artificial neural network model. [Solution] The motion capture method includes the steps of: S110 inputting camera footage into a first artificial neural network model; S130 generating event information relating to the movement of one or more objects in the motion capture space based on the process of utilizing the first artificial neural network model; and S150 generating first motion capture data relating to one or more objects relating to the camera footage based on the event information.

Description

Technical Field

[0001] The present disclosure relates to a motion capture method and apparatus using an artificial neural network model. More specifically, the present disclosure relates to a method and apparatus for generating motion capture data related to camera images based on a mechanism that uses an artificial neural network model to sense the appearance and disappearance of objects in a motion capture space and determines whether to activate the skeleton data corresponding to each object accordingly.

Background Art

[0002] Motion capture technology is used in various industrial fields, and is considered to play an important role particularly in movies, games, virtual reality (VR), augmented reality (AR), and the like. Such technology uses cameras and sensors to track the movements of people and objects, and is utilized to generate motion data that can be used in 3D animations and virtual environments.

[0003] In conventional motion capture systems, various problems may occur in the process of generating motion data. In particular, when a large number of people appear or disappear in a motion capture space, it is essential to accurately retarget the skeleton data for each person who appears and generate appropriate motion capture data. However, such a process usually relies on manual work by engineers. When the addition and deletion of people who appear are frequently performed, it is necessary to manually set whether to activate the skeleton data for each object, so the process of the work becomes time-consuming and inefficient.

[0004] Furthermore, motion capture data generated using conventional technologies often contains physically impossible or unrealistic movements. In such cases, engineers must manually correct or adjust the data, which is time-consuming and requires considerable effort. This manual intervention not only reduces the efficiency of motion capture work but also makes it difficult to maintain the accuracy and reliability of the data.

[0005] Therefore, in situations where numerous objects appear and disappear, there is a demand in the industry for improved methods that can automatically activate or deactivate skeleton data to generate more realistic and accurate motion capture data.

[0006] Korean Patent No. KR2517395 discloses a system and method for creating metaverse characters using biometric information. [Overview of the project] [Problems that the invention aims to solve]

[0007] This disclosure was devised in response to the aforementioned background technology, and more specifically, aims to generate motion capture data related to camera footage based on a mechanism that uses an artificial neural network model to sense the appearance and departure of objects in a motion capture space and, accordingly, decides whether or not to activate the skeleton data corresponding to each object.

[0008] However, the technical challenges that this disclosure aims to solve are not limited to those mentioned above, and as described below, a variety of technical challenges may be included within the scope that is obvious to an average engineer. [Means for solving the problem]

[0009] Based on one embodiment of the present disclosure for achieving the aforementioned problems, a motion capture method and apparatus using an artificial neural network model are disclosed. The above method may include the steps of: inputting camera images into a first artificial neural network model; generating event information relating to the movement of one or more objects in the motion capture space based on the first artificial neural network model; and generating first motion capture data relating to one or more objects relating to the camera images based on the event information.

[0010] In one embodiment, the event information may include one or more of the following: first event information relating to a first object newly appearing in the motion capture space, or second event information relating to a second object leaving the motion capture space.

[0011] In one embodiment, the step of generating event information relating to the changes of one or more objects in the motion capture space based on the process of using the first artificial neural network model described above may include one or more steps of: generating first event information when the first object newly appears in the motion capture space, or generating second event information when the second object leaves the motion capture space.

[0012] In one embodiment, the step of generating first motion capture data for one or more objects related to the camera image based on the event information may include: determining whether or not to activate the skeleton asset corresponding to each of the one or more objects in the motion capture space based on the event information; and generating first motion capture data for one or more objects related to the camera image based on the camera image and the currently activated skeleton asset.

[0013] In one embodiment, the step of determining whether or not to activate the skeleton asset corresponding to each of one or more objects in the motion capture space based on the above event information may include: a step of activating the first skeleton asset, which is the skeleton asset corresponding to the first object, if the above event information includes first event information; and a step of deactivating the second skeleton asset, which is the skeleton asset corresponding to the second object, if the above event information includes second event information.

[0014] In one embodiment, the first artificial neural network model can be an artificial neural network model that has been trained to generate the event information when an object included in the motion capture space changes, based on object detection.

[0015] In one embodiment, the first artificial neural network model can be a trained artificial neural network model that includes the steps of generating a training dataset based on video data from a reference camera that captures the rotational movement of each object and label data corresponding to each object, and training the first artificial neural network model to identify and classify each object in the reference camera video based on the training dataset.

[0016] In one embodiment, the step of generating first motion capture data for one or more objects related to the camera image based on the event information may include: a step of generating first verification data based on a step of verifying the position of a marker for motion capture corresponding to an object included in the camera image, and a step of generating alarm information if there is an abnormality in the first verification data.

[0017] In one embodiment, the above method may further include a step of generating second motion capture data, which is corrected motion capture data, based on the step of inputting the first motion capture data into a second artificial neural network model.

[0018] In one embodiment, the step of generating second motion capture data, which is corrected motion capture data, based on the step of inputting the first motion capture data into the second artificial neural network model, may include: a step of filtering the data relating to the contact points of objects included in the first motion capture data; a step of generating second verification data based on a step of verifying the validity of the data relating to the contact points of objects; and a step of generating second motion capture data based on the second verification data.

[0019] In one embodiment, the step of generating second motion capture data, which is corrected motion capture data, based on the step of inputting the first motion capture data into the second artificial neural network model, may include: the step of generating virtual motion capture data for one or more third objects included in the first motion capture data based on reference motion capture data, and the step of generating second motion capture data based on the step of inputting the virtual motion capture data for one or more third objects and the first motion capture data into the second artificial neural network model.

[0020] In one embodiment, the step of generating virtual motion capture data for one or more third objects included in the first motion capture data based on reference motion capture data may include: the step of identifying one or more third objects included in the first motion capture data, and the step of generating virtual motion capture data for one or more third objects based on a comparison of the skeleton of the reference object included in the reference motion capture data with the skeleton of the one or more third objects.

[0021] Based on embodiments of this disclosure for achieving the aforementioned problems, a computer program stored on a computer-readable storage medium is disclosed that causes a computing device to perform a plurality of operations for motion capture using an artificial neural network model. These operations may include: an operation to input camera images into a first artificial neural network model; an operation to generate event information relating to the changes of one or more objects in the motion capture space based on the process of using the first artificial neural network model; and an operation to generate first motion capture data relating to one or more objects relating to the camera images based on the event information.

[0022] Based on one embodiment of the present disclosure for achieving the aforementioned problems, a computing device for motion capture using an artificial neural network model is disclosed. The computing device includes one or more processors and memory, and the one or more processors can input camera images into a first artificial neural network model, generate event information relating to the changes of one or more objects in the motion capture space based on the process of utilizing the first artificial neural network model, and generate first motion capture data relating to one or more objects relating to the camera images based on the event information. [Effects of the Invention]

[0023] According to the present disclosure, by automatically detecting the appearance and disappearance of an object in a motion capture space using an artificial neural network model and determining whether to activate the skeleton data corresponding to each object, the efficiency and accuracy of the motion capture operation can be greatly improved. In addition, by automating the operations that conventionally had to be processed manually by engineers, the labor of the operations can be saved, and data processing can be stabilized even in an environment where the addition and deletion of the appearing persons are frequently performed. Further, according to the present disclosure, by generating motion data that reflects only realistic movements, the necessity of additional correction operations due to the occurrence of unrealistic movements can be minimized.

[0024] On the other hand, the effects of the present disclosure are not limited to the above-described effects, and various effects can be included within a range obvious to ordinary engineers from the content described below.

Brief Description of the Drawings

[0025] [Figure 1] FIG. 1 is a block configuration diagram of a computing device for motion capture using an artificial neural network model according to an embodiment of the present disclosure. [Figure 2] FIG. 2 shows an example of the structure of an artificial intelligence base model according to an embodiment of the present disclosure. [Figure 3] FIG. 3 is a flowchart showing a process for motion capture using an artificial neural network model according to an embodiment of the present disclosure. [Figure 4] FIG. 4 is a conceptual diagram showing a process for generating first motion capture data according to an embodiment of the present disclosure. [Figure 5] FIG. 5 is a conceptual diagram showing a process for generating first motion capture data using an artificial neural network model when one or more objects newly appear in a motion capture space according to an embodiment of the present disclosure. [Figure 6]Figure 6 is a conceptual diagram illustrating the process of generating first motion capture data using an artificial neural network model when one or more objects exit the motion capture space, based on one embodiment of the present disclosure. [Figure 7] Figure 7 is a conceptual diagram illustrating the process of generating corrected motion capture data based on an embodiment of the present disclosure, which involves inputting virtual motion capture data of an object and first motion capture data into a second artificial neural network model. [Figure 8] Figure 8 is a conceptual diagram illustrating the process of generating corrected motion capture data based on a step of filtering data relating to object contact points included in the first motion capture data, according to one embodiment of the present disclosure. [Figure 9] Figure 9 is a simplified and general schematic diagram of an exemplary computing environment that can embody the embodiments of this disclosure. [Modes for carrying out the invention]

[0026] This disclosure provides a method and apparatus for generating motion capture data related to camera images, based on a mechanism that uses an artificial neural network model to sense the appearance and departure of objects in a motion capture space and determines whether or not to activate the skeleton data corresponding to each object accordingly.

[0027] Various embodiments will be described below with reference to the drawings. Various explanations are provided in this specification to facilitate understanding of the disclosure. However, it will be obvious that such embodiments can be carried out without such specific explanations.

[0028] As used herein, terms such as “component,” “module,” and “system” refer to computer-related entities, hardware, firmware, software, combinations of software and hardware, or software execution. For example, a component may be, but is not limited to, a processing procedure executed on a processor, a processor, an object, an execution thread, a program, and / or a computer. For example, both an application running on a computing device and the computing device itself may be components. One or more components may reside in a processor and / or an execution thread. A single component may be localized within a single computer. In this disclosure, one or more components may be distributed to two or more computers, i.e., multiple execution environments. Furthermore, such components may run on a variety of computer-readable media having a variety of data structures stored within them. Components may communicate locally and / or remotely using signals, for example, one or more data packets (e.g., data and / or signals from one component interacting with other components in a local or distributed system, and data transmitted over a network such as the Internet).

[0029] Furthermore, the words “or,” “or else,” and “or else” shall mean “or,” “or else,” and “or else” in an implicational sense, rather than in an exclusive sense. In other words, unless specifically specified or made clear in context, “X uses A or B” shall mean one of the natural implicational substitutions. That is, if X uses A; X uses B; or X uses both A and B, then “X uses A or (or else) B” can apply to any of these. Also, the term “and / or” in this specification shall refer to and include all possible combinations of one or more items from the multiple related items discussed.

[0030] Furthermore, the term “include” as a predicate and / or as a modifier should be understood to mean that the feature and / or component in question exists. However, the term “include” as a predicate and / or as a modifier should be understood not to exclude the existence or addition of one or more other features, components and / or groups thereof. Also, where the number is not specifically identified or where it is not clear from the context to indicate a singular form, “singular” in this specification and claims should generally be interpreted to mean “one or more.” Furthermore, the phrase "at least one of A or B" should be interpreted as meaning "including only A," "including only B," or "a combination of A and B."

[0031] A typical engineer should also recognize that the various exemplary logical blocks, configurations, modules, circuits, means, logic, and algorithmic stages described herein as relating to the embodiments disclosed herein can be implemented by electronic hardware, computer software, or a combination of both. To clearly illustrate the interoperability between hardware and software, various exemplary components, blocks, configurations, means, logic, modules, circuits, and stages have been generally described above in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and design constraints of the overall system. A skilled engineer can implement the described functionality in various ways for individual specific applications; however, such decisions on implementation should not be interpreted as exceeding the scope of this disclosure.

[0032] The descriptions relating to the embodiments presented herein are provided so that persons with ordinary skill in the art of this disclosure may utilize or implement this disclosure. Various variations of such embodiments will be readily apparent to persons with ordinary skill in the art of this disclosure. The general principles defined herein can be applied to other embodiments without departing from the scope of this disclosure. Thus, this disclosure is not limited to the embodiments presented herein. This disclosure should be interpreted in the broadest sense, consistent with the principles and novel features presented herein.

[0033] In this disclosure, "production data" can mean data used to create content, transmitted through multiple data streams included in a particular media protocol. For example, production data in this disclosure can be in the form of data streams that meet the standards of the SMPTE 2110 protocol. In this case, production data can include audio data streams, video data streams, ancillary data streams, and time data streams. In addition to data used to create content, production data in this disclosure can also include incidental data related to content creation, such as content metadata and lighting information in virtual production. However, in this disclosure, the format of production data and the types of data streams it includes are not limited to a specific protocol and include a variety of data forms that can be used in a virtual production environment or a broadcast environment.

[0034] Figure 1 is a block diagram of a computing device for motion capture using an artificial neural network model, based on one embodiment of the present disclosure.

[0035] The configuration of the computing device (100) shown in Figure 1 is merely a simplified example. In one embodiment of this disclosure, the computing device (100) may include other configurations for implementing the computing environment of the computing device (100), and it is also possible to configure the computing device (100) using only some of the disclosed configurations.

[0036] The computing device (100) may include a processor (110), memory (130), and a network unit (150).

[0037] The processor (100) can consist of one or more cores and may include processors for data analysis and deep learning, such as a central processing unit (CPU), a general-purpose graphics processing unit (GPGPU), and a tensor processing unit (TPU). The processor (110) can read computer programs stored in memory (130) and perform data processing for machine learning in one embodiment of the present disclosure. Based on one embodiment of the present disclosure, the processor (110) can perform calculations for learning a neural network. The processor (110) can perform calculations for learning a neural network, such as processing input data for learning in deep learning (DL), extracting feature values ​​from input data, calculating errors, and updating weights in the neural network using backpropagation.

[0038] At least one of the CPU, GPGPU, and TPU of the processor (110) is capable of processing network function training. For example, both the CPU and GPGPU can perform network function training and data classification using network functions. Furthermore, in one embodiment of this disclosure, it is possible to use the processors of multiple computing devices together to perform network function training and data classification using network functions. In addition, the computer program executed in the computing device in one embodiment of this disclosure can be a program that can be executed on the CPU, GPGPU, or TPU.

[0039] In this disclosure, “motion capture space” can mean a physical area in which a motion capture system can track motion and collect data, that is, an actual space within which the movement of a person or object can be sensed. For example, “motion capture space” can include a space in a film production studio that can be sensed by numerous cameras and sensors set up to capture the movements of an actor, or a specific area in a gym or training facility that records the movements of an athlete for sports-related analysis. Such a motion capture space is often defined by the arrangement of cameras and sensors, environmental conditions, tracking range, etc., and can include settings for capturing not only human motion but also the movement and interaction of objects.

[0040] The processor (110) can input camera images to a first artificial neural network model in order to generate motion capture data. In this disclosure, camera images can mean images acquired by cameras installed inside or outside the motion capture space.

[0041] In this disclosure, the first artificial neural network model can be an artificial neural network model that is trained to generate event information when objects included in the motion capture space change, based on object detection. In this disclosure, the first artificial neural network model can be trained using a training dataset generated based on reference camera video data and corresponding label data for each object included in the reference camera video data. For example, the reference camera video may include video of a first character with a marker for motion capture. In this case, the label data may be "first character," and the label data may be applied only to frames in the reference camera video in which the first character appears. In this case, the training data can be in the form of pairs of reference camera video data and label data. The first artificial neural network model can be trained to generate event information based on a training dataset, at the timing when a specific object appears in or leaves the reference camera footage. For example, the first artificial neural network model can be trained to take one or more frames of the reference video corresponding to each timing as input, output "B" if there is no change in objects in the frame, output "A" and the identifier (ID) of the object if a frame in which a new object appears is input, and output "C" and the identifier of the object if a frame in which an existing object leaves is input.

[0042] The processor (110) can generate event information related to one or more changes in the motion capture space based on a process utilizing a first artificial neural network model. At this time, the event information may include one or more of the following: first event information relating to a first object newly appearing in the motion capture space, or second event information relating to a second object leaving the motion capture space. The processor (110) can generate first event information when a first object newly appears in the motion capture space. Furthermore, the processor (110) can generate second event information when a second object leaves the motion capture space. The specific process by which the processor (110) generates event information will be described later with reference to Figure 4.

[0043] The processor (110) can generate first motion capture data for one or more objects related to the camera image based on event information. For example, the processor (110) can generate first motion capture data corresponding to the image of a first object included in the camera image based on event information.

[0044] To generate motion capture data, the processor (110) can determine, based on event information, whether or not to activate each of the skeleton assets corresponding to one or more objects in the motion capture space. For example, if the first event information is generated when the first object appears in the motion capture space, the processor (110) can activate the first skeleton asset corresponding to the first object. On the other hand, if the second event information is generated when the second object that was present in the motion capture space leaves, the processor (110) can deactivate the second skeleton asset corresponding to the second object.

[0045] After deciding whether or not to activate a skeleton asset based on event information, the processor (110) can generate motion capture data corresponding to the objects contained in the camera footage, based on the camera footage and the currently activated skeleton asset. For example, if a first object appears in the camera footage and a second object exits, the processor (110) can activate the first skeleton asset corresponding to the first object, deactivate the second skeleton asset corresponding to the second object, and then target the currently activated first skeleton asset to the first object contained in the camera footage.

[0046] In this disclosure, the processor (110) generates first verification data based on a process of verifying the position of a marker for motion capture that corresponds to an object included in the camera image, and if there is an abnormality in the first verification data, it is possible to generate alarm information.

[0047] For example, if the camera footage is identified as containing a first object, the processor (110) can compare the marker information of the first object registered in the database with the marker information of the first object contained in the camera footage, and generate first verification data based on this comparison process. Subsequently, if there is an abnormality in the first verification data, for example, if the marker information of the first object contained in the camera footage does not match the marker information of the first object registered in the database, the processor (110) can generate alarm information to warn the user that the object identification is incorrect.

[0048] The processor (110) can generate second motion capture data, which is corrected motion capture data, based on the process of inputting first motion capture data into a second artificial neural network model. The first and second embodiments for generating the second motion capture data will be described below.

[0049] In the first embodiment of this disclosure, the processor (110) can generate second verification data based on a step of verifying the validity of the data relating to the contact points of an object included in the first motion capture data. Subsequently, the processor (110) can generate second motion capture data by correcting the first motion capture data based on the second verification data. A specific method for generating second motion capture data based on the validity of the data relating to the contact points will be described later with reference to Figure 8.

[0050] In a second embodiment of this disclosure, the processor (110) can generate virtual motion capture data for a third object, which is an object included in the first motion capture data, based on reference motion capture data. In this case, the virtual motion capture data can be motion capture data generated considering the ratio difference between the skeleton of the object in the reference motion capture data and the skeleton of the third object.

[0051] Subsequently, the processor (110) can generate corrected second motion capture data based on the process of inputting all of the first motion capture data, which includes virtual motion capture data of the third object and actual motion capture data of the third object, into the second artificial neural network model. A specific method for generating second motion capture data based on virtual motion capture data will be described later with reference to Figure 7.

[0052] Through this disclosure, it becomes possible to automate the retargeting process—where engineers manually monitor camera footage and retarget each object with its corresponding skeleton asset—using an artificial neural network model. As a result, this disclosure yields a significant advantage in efficiently generating motion capture data.

[0053] In several embodiments of this disclosure, the memory (130) may include at least one type of storage medium from among flash memory type, hard disk type, multimedia card micro type, card type memory (e.g., SD or XD memory), Random Access Memory (RAM), Static Random Access Memory (SRAM), Read-Only Memory (ROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Programmable Read-Only Memory (PROM), magnetic memory, magnetic disk, and optical disk. The computing device (100) may also operate in conjunction with web storage that performs the storage function of the memory (130) over the internet. The foregoing descriptions of memory are illustrative, and this disclosure is not limited thereto.

[0054] In one embodiment of the present disclosure, the network unit (150) can use a variety of wired communication systems such as public switched telephone networks (PSTN), xDSL (x Digital Subscriber Line), RADSL (Rate Adaptive DSL), MDSL (Multi Rate DSL), VDSL (Very High Speed ​​DSL), UADSL (Universal Asymmetric DSL), HDSL (High Bit Rate DSL), and local area networks (LANs).

[0055] Furthermore, the network unit (150) in this specification can utilize a variety of wireless communication systems such as CDMA (Code Division Multi Access), TDMA (Time Division Multi Access), FDMA (Frequency Division Multi Access), OFDMA (Orthogonal Frequency Division Multi Access), SC-FDMA (Single Carrier-FDMA), and other systems.

[0056] In this disclosure, the network unit (150) can utilize any form of wired or wireless communication system.

[0057] The technologies described herein can be used not only in the aforementioned networks but also in other networks.

[0058] Figure 2 shows an example of the structure of an artificial intelligence platform model based on one embodiment of the present disclosure.

[0059] Throughout this specification, the terms artificial intelligence model, artificial intelligence infrastructure model, computational model, neural network, network function, and neural network can be used interchangeably.

[0060] Neural networks can generally be constructed as a collection of interconnected computational units called nodes. Such nodes may also be referred to as neurons. A neural network consists of at least one node. The nodes (or neurons) that make up a neural network can be interconnected by one or more links.

[0061] In a neural network, one or more nodes connected by links can, relatively speaking, be in an input-output node relationship. The concepts of input and output nodes are relative; any node that is an output node to one node can be an input node in relation to other nodes, and vice versa. As mentioned above, the relationship between input and output nodes can be established around links. One input node can be connected to one or more output nodes via links, and vice versa.

[0062] In a relationship between input and output nodes connected via a single link, the output node's data can be determined based on the data input to the input node. In this case, the link connecting the input and output nodes can have weights. These weights can be variable, but they can be adjusted according to the user or algorithm to perform the function required by the neural network. For example, if one or more input nodes are interconnected to one output node by their respective links, the output node can determine its value based on the values ​​input to the input nodes connected to it and the weights set for the links corresponding to each input node.

[0063] As mentioned above, a neural network consists of one or more nodes interconnected via one or more links, forming an input-output node relationship within the network. In a neural network, the characteristics of the network can be determined by the number of nodes and links, the correlation between nodes and links, and the weight values ​​assigned to each link. For example, if there are two neural networks with the same number of nodes and links but different link weight values, these two neural networks can be recognized as distinct.

[0064] A neural network can be composed of a set of one or more nodes. A subset of the nodes that make up a neural network can form a layer. Some of the nodes that make up a neural network can form a layer based on their distance from a first input node. For example, a set of nodes that are n in distance from a first input node can form an n-th layer. The distance from the first input node can be defined based on the minimum number of links that must be traversed to reach that node from the first input node. However, this definition of a layer is arbitrary for the sake of explanation, and the position of a layer in a neural network can be defined in a way different from the above explanation. For example, the layer of nodes can be defined based on their distance from the final output node.

[0065] In one embodiment of the content of this disclosure, a collection of neurons or nodes can be defined using the terms "hierarchy" or "layer."

[0066] A first input node can refer to one or more nodes in a neural network that receive data directly without going through links in relation to other nodes. Alternatively, it can refer to a node in a neural network that does not have other input nodes connected via links in relation to other nodes based on links. Similarly, a final output node can refer to one or more nodes in a neural network that do not have an output node in relation to other nodes. Furthermore, a hidden node can refer to a node that does not fall under the category of first input node or final output node, but is still part of the neural network.

[0067] In one embodiment of this disclosure, the neural network may have the same number of nodes in the input layer as in the output layer, and the number of nodes may decrease once before increasing again as one progresses from the input layer to the hidden layer. In another embodiment of this disclosure, the neural network may have fewer nodes in the input layer than in the output layer, and the number of nodes may increase as one progresses from the input layer to the hidden layer. In yet another embodiment of this disclosure, the neural network may have more nodes in the input layer than in the output layer, and the number of nodes may decrease as one progresses from the input layer to the hidden layer. In yet another embodiment of this disclosure, the neural network may be a combination of the above-described neural networks.

[0068] An artificial intelligence platform model based on one embodiment of the disclosed material may include a deep neural network (DNN). A deep neural network can mean a neural network that includes multiple hidden layers in addition to the input and output layers. By using a deep neural network, it is possible to grasp the latent structures of data. That is, it is possible to grasp the latent structures of photographs, text, videos, audio, protein sequence structures, gene sequence structures, peptide sequence structures, music (for example, what is depicted in a photograph, what is the content and emotion of a text, what is the content and emotion of an audio recording, etc.), and / or the binding affinity between peptides and MHC. Deep neural networks can include convolutional neural networks (CNNs), recurrent neural networks (RNNs), autoencoders, restricted Boltzmann machines (RBMs), deep belief networks (DBNs), Q networks, U networks, Siam networks, generative adversarial networks (GANs), transformers, and the like. The deep neural networks mentioned above are merely examples, and this disclosure is not limited to them.

[0069] The artificial intelligence platform model described herein can be represented by any of the aforementioned network structures, including an input layer, a hidden layer, and an output layer.

[0070] The neural networks used in the artificial intelligence platform models described herein may be trained using at least one of the following methods: supervised learning, unsupervised learning, semi-supervised learning, transfer learning, active learning, or reinforcement learning. Learning a neural network can be the process of applying knowledge to the neural network to enable it to perform specific actions.

[0071] Neural networks can be trained to minimize the error in their output. During neural network training, training data is repeatedly input into the network, the network's output and target error for the training data are calculated, and the error is backpropagated from the output layer to the input layer to update the weights of each node in the neural network, in order to reduce the error. In supervised learning, training data with the correct answer labeled is used (i.e., labeled training data), while in unsupervised learning, the correct answer may not be labeled for each training data point. For example, in supervised learning for data classification, the training data could be data with a category labeled for each training data point. Labeled training data is input into the neural network, and the error can be calculated by comparing the neural network's output (category) with the labels of the training data. As another example, in unsupervised learning for data classification, the error can be calculated by comparing the input training data with the output of the neural network. The calculated error is backpropagated in the neural network in the reverse direction (i.e., from the output layer to the input layer), and through backpropagation, the connection weights of each node in each layer of the neural network can be updated. The amount of change in the connection weights of each node that are updated can be determined by the learning rate. The calculation of the neural network on the input data and the backpropagation of the error can constitute a learning cycle (epoch). The learning rate can be applied in a way that changes depending on the number of iterations of the neural network's learning cycle. For example, in the early stages of learning the neural network, a high learning rate can be used to increase efficiency by enabling the neural network to quickly achieve a certain level of performance, while in the later stages of learning, a low learning rate can be used to improve accuracy.

[0072] In neural network training, training data can generally be a subset of real-world data (i.e., data that the trained neural network intends to process). This can lead to a training cycle where errors on the training data decrease, but errors on the real-world data increase. Overfitting is the phenomenon where errors on real-world data increase due to excessive training on the training data. For example, a neural network that has learned to recognize cats by observing yellow cats may fail to recognize cats of other colors as cats; this is a type of overfitting. Overfitting can increase errors in machine learning algorithms. Various optimization methods can be used to prevent such overfitting. These methods include increasing the amount of training data, regularization, dropout (deactivating some of the network nodes during the training process), and the use of a batch normalization layer.

[0073] An artificial intelligence platform model based on one embodiment of the present disclosure may include a Large Language Model (LLM). In this disclosure, a Large Language Model can mean an artificial intelligence platform model trained using a vast amount of training data to enable Natural Language Processing. A Large Language Model may include a transformer, a transformer encoder system model, and / or a transformer decoder system model. A transformer encoder system model can correspond to an artificial intelligence model that uses the transformer encoder structure. A transformer decoder system model can correspond to an artificial intelligence model that uses the transformer decoder structure.

[0074] In one embodiment, the transformer may consist of an encoder that encodes input data and a decoder that decodes the encoded data. The transformer may have a structure that receives a series of input data and outputs a series of output data through the stages of encoding and decoding. In one embodiment, the series of input data may be processed into a form that the transformer can process. The process of processing the series of input data into a form that the transformer can process may include a tokenization process and an embedding process. The tokenization process may mean the process of dividing the series of input data into tokens of a certain unit. For example, the certain unit may include a word unit. The embedding process may mean the process of converting at least one tokenized from the series of input data into an embedding vector.

[0075] In one embodiment, the transformer can obtain an embedding vector that is input to the encoder, which includes a token embedding vector containing a series of input data and at least one corresponding token, a segment embedding vector that distinguishes sentences containing tokens by token, and a position embedding vector that reflects the position of the tokens. The encoder and decoder models of the transformer can also obtain the embedding vector in the same manner.

[0076] In one embodiment, the encoder and decoder within the transformer can utilize an attention algorithm to encode and decode a series of input data. An attention algorithm can be defined as an algorithm that calculates the similarity of a given query by calculating the product of the matrix of the query and the key, applying the softmax function to the resulting attention score, and then calculating the product of the matrix of the calculated similarity and the value to calculate the attention value for the query.

[0077] In one embodiment, a self-attention algorithm can refer to an attention algorithm that uses queries, keys, and values ​​generated by multiplying the same embedding vector by numerical values ​​for query weights, key weights, and value weights, respectively. A cross-attention algorithm can refer to an attention algorithm that uses queries generated by multiplying a first embedding vector by query weights, and keys and values ​​generated by multiplying a second embedding vector by key weights and value weights, respectively. The query weights, key weights, and value weights can be trainable parameters that are updated through the learning process of a large-scale language model.

[0078] In one embodiment, the transformer's encoder may include an embedding layer, a self-attention layer that applies a self-attention algorithm to the embedding vector, a normalization layer, and a feed-forward neural network (FFN). The encoder may also take the form of N interconnected unit structures, each containing a self-attention layer, a normalization layer, and a feed-forward neural network. The transformer's decoder may include an embedding layer, a masked self-attention layer, a normalization layer, a cross-attention layer that applies a cross-attention algorithm, and a feed-forward neural network. The decoder may also take the form of N interconnected unit structures, each containing a masked self-attention layer, a normalization layer, a cross-attention layer, and a feed-forward neural network. The masked self-attention layer can correspond to a layer that determines the attention value for each sequence containing multiple words in a series of input data.

[0079] A transformer may include not only an encoder and a decoder, but also ancillary components such as a linear layer and a softmax layer. Models of the encoder system and the decoder system of a transformer may also include not only an encoder and a decoder, but also the aforementioned ancillary components. Methods for constructing a transformer using an attention algorithm may include the method disclosed in Vaswani et al., Attention Is All You Need, 2017 NIPS, which is incorporated hereby by reference.

[0080] In one embodiment, attention layers such as a self-attention layer, a masked self-attention layer, and a cross-attention layer can correspond to a multi-head attention layer that includes multiple attention layers in parallel. The multi-head attention layer can output an output attention value by concatenating the attention values ​​output from each of the multiple attention layers and calculating the product of the concatenated matrix and the output weight matrix. The output attention value output from the multi-head attention layer can have the same magnitude as the attention value output from a single attention layer.

[0081] In one embodiment, the transformer can be trained through processes such as a Masked Language Model (MLM) process and a Next Sentence Prediction (NSP) process. The MLM process can be interpreted as a learning process in which the transformer predicts masked words using a set of training data in which some words are masked. The NSP process can be interpreted as a learning process in which the transformer determines whether or not two sentences are connected in a set of training data containing any two sentences.

[0082] In one embodiment, the large-scale language model can process not only natural language text but also a variety of data formats such as image data, audio data, and video data. The large-scale language model can embed data to convert multiple data with diverse data formats into a series of data that can be computed. The large-scale language model can process additional data that represents the relative positional or phase relationships between a series of input data. Alternatively, a series of input data can be embedded by further reflecting multiple vectors that represent the relative positional or phase relationships between multiple input data. In one example, the relative positional relationships between a series of input data can include, but are not limited to, word order in a natural language sentence, the relative positional relationships of each of the divided images, or the temporal order of divided audio waveforms. The process of adding information that represents the relative positional or phase relationships between a series of input data can be called positional encoding.

[0083] An example of a large-scale language model for processing image data (Vision Transformer, ViT) is disclosed in Dosovitskiy, et al., AN IMAGE IS WORTH 16X16 WORDS: TRANSFORMERS FOR IMAGE RECOGNITION AT SCALE, which is incorporated herein by reference.

[0084] An artificial intelligence model based on one embodiment of the present disclosure may include a multimodal large-scale language model. A multimodal large-scale language model can mean a large-scale language model that can understand and process the relationships between different data formats, such as natural language text data, image data, audio data, and video data. A multimodal language model may include multiple encoders that encode each data format and the corresponding input data. A multimodal language model can be trained to calculate the similarity between multiple embedding vectors encoded from the encoders of each data format using training data containing data in multiple different data formats, such that the similarity between identical pairs is calculated to be higher and the similarity between different pairs is calculated to be lower. An example of a multimodal large-scale language model that understands and processes the relationships between image data and natural language text data (Contrastive Language-Image Pre-training, CLIP) is disclosed in Alec Radford, et al., LEARNING TRANSFERABLE VISUAL MODELS FROM NATURAL LANGUAGE SUPERVISION, which is incorporated herein by reference.

[0085] Figure 3 is a flowchart showing the process for motion capture using an artificial neural network model, based on one embodiment of the present disclosure.

[0086] As shown in Figure 3, the process for motion capture using the artificial neural network model of this disclosure may include the steps of inputting camera images into the first artificial neural network model (S110), generating event information relating to the movement of one or more objects in the motion capture space based on the process of utilizing the first artificial neural network model (S130), and generating first motion capture data relating to one or more objects relating to the camera images based on the event information (S150).

[0087] In step S110, the processor (110) can input camera images to the first artificial neural network model. As previously mentioned with reference to Figure 1, the first artificial neural network model can be an artificial neural network model that has been trained to generate event information when objects included in the motion capture space change, based on object detection.

[0088] In step S130, the processor (110) can generate event information related to changes in one or more objects within the motion capture space based on a process utilizing the first artificial neural network model. The first artificial neural network model can output first event information indicating the appearance of a new object when that object appears in the motion capture space, and second event information indicating the departure of an object when that object leaves the motion capture space.

[0089] In step S150, the processor (110) can generate first motion capture data for one or more objects related to the camera image. For example, the processor (110) can target an activated skeleton asset to each object included in the camera image and generate motion data of the skeleton asset corresponding to the movement of each object included in the camera image as first motion capture data. The specific method for generating the first motion capture data was described above with reference to Figure 1.

[0090] Figure 4 is a conceptual diagram showing the process of generating first motion capture data based on one embodiment of the present disclosure.

[0091] In this disclosure, multiple cameras (411) are installed in the motion capture space (410) to capture images of an object (412).

[0092] The video captured by each camera (411) can be transmitted to the video processing unit (420). In the video processing unit (420), the video captured by each camera can be combined into one, and the marker data contained in each video can be processed.

[0093] The motion capture unit (430) can receive data as input from the video processing unit (420) and generate first motion capture data (440) for each video. If an artificial neural network model is not used, an engineer can check the motion capture markers attached to each object based on the camera footage, assign a corresponding skeleton asset to each object, and then the motion capture unit (430) can generate first motion capture data (440) corresponding to each object.

[0094] Figure 5 is a conceptual diagram illustrating the process of generating first motion capture data using an artificial neural network model when one or more objects newly appear in the motion capture space, based on one embodiment of the present disclosure.

[0095] In this disclosure, multiple cameras (511) are installed in the motion capture space (510), and it is possible to capture images of a newly appearing first object (512) in the motion capture space. In the video processing unit (520), the images captured by each camera are combined into one, and the marker data contained in each image is processed.

[0096] The artificial neural network model (550) receives video transmitted from the video processing unit (520) as input, identifies the frame in which the first object appears, and generates first event information. At this time, the first event information may include information relating to the timing of the appearance of the new object, as well as information relating to the identifier of the newly appeared object (in this case, the first object) and its position in the video.

[0097] The motion capture unit (530) receives the first event information output by the artificial neural network as input, activates the skeleton asset corresponding to the newly appearing object in the camera image, i.e., the first object, and then targets the skeleton asset to the first object. Subsequently, the motion capture unit (530) can generate first motion capture data (540) corresponding to the object included in the camera image, based on the targeted skeleton asset and the camera image.

[0098] Figure 6 is a conceptual diagram illustrating the process of generating first motion capture data using an artificial neural network model when one or more objects exit the motion capture space, based on one embodiment of the present disclosure.

[0099] In this disclosure, multiple cameras (611) are installed in the motion capture space (610), and it is possible to capture video of a second object (612) that has left the motion capture space. In the video processing unit (620), the video captured by each camera is combined into one, and the marker data contained in each video is processed.

[0100] The artificial neural network model (650) receives video transmitted from the video processing unit (620) as input, identifies frames in which the second object no longer exists, and generates second event information. At this time, the second event information may include information related to the timing of the second object's departure and information related to the identifier of the departed object (in this case, the second object).

[0101] The motion capture unit (630) receives the second event information output by the artificial neural network as input and can deactivate the skeleton asset corresponding to the object that has disappeared from the camera image, i.e., the second object. Subsequently, the motion capture unit (630) can generate first motion capture data (540) corresponding to the object included in the camera image, based on the skeleton asset that is currently activated and targeted to another object and the camera image.

[0102] Figure 7 is a conceptual diagram illustrating the process of generating corrected motion capture data based on an embodiment of the present disclosure, which involves inputting virtual motion capture data of an object and first motion capture data into a second artificial neural network model.

[0103] In a second embodiment of this disclosure, the reference motion capture data (720) may include accurate motion capture data that has been pre-recorded and generated to correspond to each movement of a motion actor.

[0104] The processor (110) can generate virtual motion capture data (730) relating to one or more third objects included in the first motion capture data (710) generated based on camera footage, based on the reference motion capture data (720). Specifically, the processor (110) can generate virtual motion capture data (730) for the third objects based on the process of identifying one or more third objects included in the first motion capture data (710) and comparing the skeleton of the reference object included in the reference motion capture data (720) with the skeleton of the third object. At this time, the virtual motion capture data (730) can be generated while reflecting the ratio difference between the skeleton of the reference object and the skeleton of the third object included in the camera footage.

[0105] Subsequently, the processor (110) can generate corrected second motion capture data (740) based on the process of inputting virtual motion capture data (730) of a third object generated by referencing reference motion capture data (720) and the actual first motion capture data (710) into the second artificial neural network model. In this case, even if the first motion capture data initially captured contains errors that do not match the actual movement, it is possible to generate corrected second motion capture data that closely matches the actual movement by referring to the reference motion capture data (720) which accurately records the movement.

[0106] Figure 8 is a conceptual diagram illustrating the process of generating corrected motion capture data based on a step of filtering data relating to object contact points included in the first motion capture data, according to one embodiment of the present disclosure.

[0107] In the first embodiment of this disclosure, the processor (110) can filter data relating to object contact points included in the first motion capture data (810) generated based on camera images. For example, the processor (110) can identify in the first motion capture data (810) the parts where an object is in contact with the floor surface, contact points between multiple different objects, and parts of the same object that need to be in contact and parts that should not be in contact, and extract such data to generate filtered data (820).

[0108] Subsequently, the processor (110) can generate second verification data based on a process of verifying the validity of the data relating to the object's contact points in the filtered data (820). For example, based on the filtered data (820), the processor (110) can identify cases such as when an object is floating above the floor, when an object has its hands on its hips and the hands are penetrating the hips, or when multiple different objects overlap, and generate second verification data by determining that each motion capture data is invalid.

[0109] Subsequently, the processor (110) can generate second motion capture data (830), which is motion capture data corrected based on the first motion capture data (810), using the second verification data as a reference. In this case, the processor (110) can verify whether the motion capture data is generated as data consistent with the actual data by repeating the process of generating second verification data based on the second motion capture data (830).

[0110] On the other hand, a computer-readable medium storing a data structure is disclosed based on the embodiments of this disclosure.

[0111] A data structure can refer to the organization, management, and storage of data, enabling efficient access to and modification of that data. A data structure can also refer to the organization of data to solve a specific problem (e.g., data retrieval, data storage, data modification in the shortest possible time). A data structure can also be defined as the physical or logical relationships between multiple data elements designed to support specific data processing functions. Logical relationships between multiple data elements can include linking relationships between multiple user-defined data elements. Physical relationships between multiple data elements can include actual relationships between multiple data elements physically stored in a computer-readable storage medium (e.g., a permanent storage device). Specifically, a data structure can include a collection of data, relationships between data, and functions or instructions that can be applied to data. By leveraging effectively designed data structures, computing devices can perform calculations while minimizing the use of their resources. Specifically, computing devices can improve the efficiency of calculations, reading, ingesting, deleting, comparing, exchanging, and retrieving through effectively designed data structures.

[0112] Data structures can be classified into linear and nonlinear data structures based on their form. A linear data structure can be a structure in which only one piece of data follows one piece of data. Linear data structures can include lists, stacks, queues, and deques. A list can represent a set of data that has an internal order. Lists can include linked lists. A linked list can be a data structure in which data is linked in a linear fashion, with each piece of data having a pointer. In a linked list, pointers can contain information about the link to the next or previous piece of data. Linked lists can be represented as one-way linked lists, two-way linked lists, or circular linked lists, depending on their form. A stack can be a data array structure with restricted access to data. A stack can be a linear data structure in which data can only be processed (e.g., inserted or deleted) at one end of the data structure. Data stored in a stack can be a last-in, first-out (LIFO) data structure. A queue is also a data array structure with restrictions on access to data, but the difference from a stack is that it can be a first-in, first-out (FIFO) data structure. A deck can be a data structure that allows data to be processed at both ends of the data structure.

[0113] Nonlinear data structures can be structures where one data point is followed by multiple data points. Nonlinear data structures can include graph data structures. Graph data structures can be defined using vertices and edges, where edges can include lines connecting two different vertices. Graph data structures can also include tree data structures. A tree data structure can be a data structure where there is one path connecting two different vertices among the multiple vertices contained in the tree. In other words, it can be a data structure that does not form loops within a graph data structure.

[0114] Throughout this specification, the terms computational model, neural network, network function, and neural network can be used interchangeably. Hereafter, they will be consistently referred to as neural networks. A data structure can include a neural network. A data structure including a neural network can be stored on a computer-readable medium. Furthermore, a data structure including a neural network can include data preprocessed for processing by the neural network, data input to the neural network, neural network weights, neural network hyperparameters, data obtained from the neural network, activation functions associated with each node and hierarchy of the neural network, loss functions for learning the neural network, etc. A data structure including a neural network can include any of the components of the configuration disclosed above. In other words, when constructing a data structure containing a neural network, it is possible to include all of the following, or any combination thereof: data preprocessed for processing by the neural network, data input to the neural network, neural network weights, neural network hyperparameters, data obtained from the neural network, activation functions associated with each node and hierarchy of the neural network, loss functions for learning the neural network, etc. Beyond the aforementioned configurations, a data structure containing a neural network can also include any other information that determines the characteristics of the neural network. Furthermore, the data structure can include, and is not limited to, any form of data used or generated during the computational process of the neural network. Computer-readable media can include computer-readable recording media and / or computer-readable transmission media. A neural network can generally be composed of a set of interconnected computational units called nodes. Such nodes can also be called neurons. A neural network is composed of at least one node.

[0115] A data structure can include data to be input to a neural network. A data structure including data to be input to a neural network can be stored on a computer-readable medium. The data to be input to a neural network can include learning data input during the learning process of the neural network and / or input data to be input to a neural network after learning is complete. The data to be input to a neural network can include pre-processed data and / or data subject to pre-processing. Pre-processing can include data processing processes for inputting data into a neural network. Therefore, a data structure can include data subject to pre-processing and data generated by pre-processing. The aforementioned data structures are merely examples, and this disclosure is not limited thereto.

[0116] The data structure may include the weights of a neural network. (In this specification, weights and parameters can be considered to have the same meaning.) The data structure including the weights of a neural network may be stored in a computer-readable medium. A neural network may include multiple weights. The weights may be variable, but can be varied according to the user or algorithm in order to perform the function required by the neural network. For example, if one or more input nodes are interconnected to one output node by their respective links, the output node can determine the value of the data output from the output node based on multiple values ​​input to the input nodes connected to the output node and the weights set for the links corresponding to each input node. The data structures described above are illustrative examples, and this disclosure is not limited thereto.

[0117] As an example rather than a limitation, the weights may include weights that change during the learning process of the neural network and / or weights after the neural network has finished learning. The weights that change during the learning process of the neural network may include the weights at the start of the learning cycle and / or weights that change during the learning cycle. The weights after the neural network has finished learning may include weights after the learning cycle has finished. Therefore, a data structure containing the weights of a neural network may include a data structure containing weights that change during the learning process of the neural network and / or weights after the neural network has finished learning. Accordingly, the aforementioned weights and / or each combination of weights shall be included in the data structure containing the weights of a neural network. The aforementioned data structures are merely examples, and this disclosure is not limited thereto.

[0118] Data structures containing neural network weights can be stored on a computer-readable storage medium (e.g., memory, hard disk) after undergoing a serialization process. Serialization can be a process of converting data structures into a form that can be stored on the same or different computing devices and later reconfigured for use. Computing devices can serialize data structures and send and receive data over a network. Serialized data structures containing neural network weights can be reconfigured on the same or other computing devices through deserialization. Data structures containing neural network weights are not limited to serialization. Furthermore, data structures containing neural network weights can include data structures that enhance computational efficiency while minimizing the use of computing device resources (e.g., B-trees, tries, m-way search trees, AVL trees, Red-Black trees in nonlinear data structures). The foregoing are illustrative examples, and this disclosure is not limited thereto.

[0119] The data structure can include the hyperparameters of the neural network. The data structure containing the neural network's hyperparameters can be stored in a computer-readable medium. The hyperparameters can be variable depending on the user. Examples of hyperparameters include the learning rate, cost function, number of learning cycle iterations, weight initialization (e.g., setting the range of weight values ​​to be initialized), and number of hidden units (e.g., number of hidden layers, number of nodes in hidden layers). The aforementioned data structures are illustrative and the disclosure is not limited thereto.

[0120] Figure 9 is a simplified and general schematic diagram of an exemplary computing environment in which embodiments of the present disclosure can be realized.

[0121] While it has been stated that this disclosure can generally be embodied by computing devices, a typical engineer would understand that it can also be embodied as a combination of computer executable instructions and / or other program modules and / or a combination of hardware and software that can be executed on one or more computers.

[0122] Generally, a program module includes routines, programs, components, data structures, and so on, that perform a specific task or implement a specific abstract data type. Furthermore, a typical engineer would understand that the methods described herein can be implemented in configurations of other computer systems, including single-processor or multi-processor computer systems, minicomputers, mainframe computers, as well as personal computers, handheld computing devices, microprocessor-based consumer electronics, or programmable consumer electronics, and so on (all of which can operate in conjunction with one or more associated devices).

[0123] The embodiments described herein can further be implemented in a distributed computing environment in which a task is performed by remote processing units connected via a communication network. In a distributed computing environment, program modules can reside both locally and in remote memory storage devices.

[0124] Computers typically include a variety of computer-readable media. Any media accessible by a computer can be computer-readable, but such computer-readable media include volatile and non-volatile media, transient and non-transitory media, and portable and non-portable media. By example, not by limitation, computer-readable media may include computer-readable storage media and computer-readable transmission media. Computer-readable storage media include volatile and non-volatile media, transient and non-transitory media, portable and non-portable media, embodied by any method or technique for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer-readable storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory, or other memory technologies, CD-ROM, DVD (digital video disk), or other optical storage devices, magnetic cassettes, magnetic tapes, magnetic disk storage devices, or other magnetic storage devices, or any other media that can be accessed by a computer and used to store information.

[0125] Computer-readable transmission media typically include all information transmission media that implement computer-readable instructions, data structures, program modules, or other data on a modulated data signal, such as a carrier wave or other transmission mechanism. The term modulated data signal means a signal in which one or more of its characteristics have been set or modified in order to encode information within the signal. By example, rather than by limitation, computer-readable transmission media include wired media such as wired networks or direct-wired connections, and wireless media such as acoustic, RF, infrared, and other wireless media. Any combination of the aforementioned media is also included in the scope of computer-readable transmission media.

[0126] Exemplary environments embodying various aspects of this disclosure, including a computer (1102), are shown, the computer (1102) including a processor (1104), system memory (1106), and a system bus (1108). The system bus (1108) connects system components, including (but not limited to) system memory (1106), to the processor (1104). The processor (1104) can be any of a variety of commercial processors. Dual-processor and other multi-processor architectures can also be used as the processor (1104).

[0127] The system bus (1108) can be one of several types of bus structures that can be further interconnected with local buses using either a memory bus, a peripheral bus, or a variety of commercial bus architectures. System memory (1106) includes read-only memory (ROM) (1110) and random access memory (RAM) (1112). The basic input / output system (BIOS) is stored in non-volatile memory (1110) such as ROM, EPROM, or EEPROM, and this BIOS includes basic routines that support the exchange of information between multiple components within the computer (1102) during startup, etc. RAM (1112) may also include high-speed RAM such as static RAM for caching data.

[0128] The computer (1102) also includes an internal hard disk drive (HDD) (1114) (e.g., EIDE, SATA)—this internal hard disk drive (1114) can be used as an external drive in a suitable chassis (not shown), a magnetic floppy disk drive (FDD) (1116) (e.g., for reading from and writing to a portable diskette (1118)), and an optical disk drive (1120) (e.g., for reading CD-ROM disks (1122) or reading from and writing to other high-capacity optical media such as DVDs). The hard disk drive (1114), magnetic disk drive (1116), and optical disk drive (1120) can be connected to a system bus (1108) by a hard disk drive interface (1124), a magnetic disk drive interface (1126), and an optical drive interface (1128), respectively. The interface (1124) for implementing external drives includes at least one or both of the following: USB (Universal Serial Bus) and / or IEEE 1394 interface technologies.

[0129] These drives and computer-readable media relating thereto provide non-volatile storage for data, data structures, computer-executable instructions, and so on. In the case of a computer (1102), the drives and media correspond to storing any data in a suitable digital format. While the above description of computer-readable storage media refers to HDDs, portable magnetic disks, and portable optical media such as CDs or DVDs, it is obvious to the average technician that other types of computer-readable storage media, such as zip drives, magnetic cassettes, flash memory cards, cartridges, and so on, can also be used in the exemplary operating environment, and that any of these media can contain computer-executable instructions for performing the methods of the Disclosure.

[0130] Numerous program modules, including an operating system (1130), one or more application programs (1132), other program modules (1134), and program data (1136), can be stored in the drive and RAM (1112). All or part of the operating system, applications, modules, and / or data can also be cached in RAM (1112). It is obvious that this disclosure can be implemented using various commercially available operating systems or combinations of operating systems.

[0131] The user can input commands and information to the computer (1102) through one or more wired or wireless input devices, such as a keyboard (1138) and a pointing device such as a mouse (1140). Other input devices (not shown in the diagram) may include a microphone, IR remote control, joystick, gamepad, stylus pen, touchscreen, and so on. These and other input devices are often connected to the processing unit (1104) via an input device interface (1142) connected to the system bus (1108), but they can also be connected via other interfaces such as parallel ports, IEEE1394 serial ports, game ports, USB ports, IR interfaces, and so on.

[0132] A monitor (1144) or other type of display device is also connected to the system bus (1108) via an interface such as a video adapter (1146). In addition to the monitor (1144), the computer generally includes other peripheral output devices such as speakers, printers, and so on (not shown in the illustration).

[0133] A computer (1102) can operate in a networked environment by utilizing logical connections to one or more remote computers, such as (multiple) remote computers (1148), via wired and / or wireless communication. These (multiple) remote computers (1148) can be workstations, computing device computers, routers, personal computers, portable computers, microprocessor-based entertainment devices, peer devices, or other typical network nodes, and include many or all of the components generally associated with a computer (1102), although for simplification only a memory storage device (1150) is illustrated. The illustrated logical connections include wired or wireless connections to a short-range network (LAN) (1152) and / or a larger network, such as a long-range network (WAN) (1154). Such LAN and WAN networking environments are common in offices and companies, facilitating enterprise-wide computer networks such as intranets, all of which can connect to global computer networks, such as the Internet.

[0134] When used in a LAN networking environment, the computer (1102) connects to the local network (1152) via a wired and / or wireless network interface, or an adapter (1156). The adapter (1156) facilitates wired or wireless communication to the LAN (1152), which also includes a wireless access point installed therein to communicate with the wireless adapter (1156). When used in a WAN networking environment, the computer (1102) may include a modem (1158), or have other means of establishing communication through the WAN (1154), such as connecting to a communication computing device in the WAN (1154) or via the Internet. The modem (1158), which can be internal or external, and wired or wireless, connects to the system bus (1108) via a serial port interface (1142). In a networked environment, a program module or a portion thereof, as described in relation to a computer (1102), can be stored in a remote memory / storage device (1150). It is obvious that the illustrated network connection is illustrative, and other means of establishing communication links between multiple computers may also be used.

[0135] The computer (1102) operates to communicate with any wireless device or unit that is arranged and operates in wireless communication, such as a printer, scanner, desktop and / or portable computer, PDA (portable data assistant), communication satellite, any equipment or location relating to a wirelessly discoverable tag, and a telephone. This includes at least Wi-Fi and Bluetooth® wireless technologies. Thus, the communication may be a predefined structure, like a conventional network, or simply ad hoc communication between at least two devices.

[0136] Wi-Fi (Wireless Fidelity) enables internet access and other connectivity even without a wired connection. Wi-Fi is a wireless technology, similar to cell phone communication, that allows devices like computers to send and receive data from anywhere within the base station's coverage area, whether indoors or outdoors. Wi-Fi networks utilize IEEE 802.11 (a, b, g, etc.) wireless technology to provide secure, reliable, and high-speed wireless connectivity. Wi-Fi can be used to connect computers to each other via the internet and wired networks (using IEEE 802.3 or Ethernet). Wi-Fi networks can operate in unlicensed 2.4 and 5 GHz wireless bands at data rates of, for example, 11 Mbps (802.11a) or 54 Mbps (802.11b), or in products that include both bands (dual band).

[0137] A person with ordinary skill in the art of this disclosure will be able to understand that information and signals can be represented using any variety of different techniques and methods. For example, the data, instructions, commands, information, signals, bits, symbols and chips referenced in the above description can be represented by voltage, electric current, electromagnetic waves, magnetic fields or particles, optical fields or particles, or any combination thereof.

[0138] A person with ordinary skill in the art of this disclosure will understand that the various exemplary logic blocks, modules, processors, means, circuits, and algorithmic stages described herein may be embodied in electronic hardware, various forms of programs (referred to herein as “software” for convenience of explanation), design code, or a combination of all of these. To clearly illustrate this interoperability of hardware and software, various exemplary components, blocks, modules, circuits, and stages have been generally described above, focusing on their functions. Whether such functions are implemented in hardware or software depends on the design constraints imposed on the particular application and the overall system. A person with ordinary skill in the art of this disclosure may embodied the functions described in various ways for individual specific applications, but such decisions should not be construed as departing from the scope of this disclosure.

[0139] The various embodiments described herein can be embodied as methods, apparatus, or manufactured articles using standard programming and / or engineering techniques. For the purposes of this term, “manufactured article” includes computer programs, carriers, or media accessible from any computer-readable device. For example, computer-readable storage media include, but are not limited to, magnetic storage devices (e.g., hard disks, floppy disks, magnetic strips, etc.), optical disks (e.g., CDs, DVDs, etc.), smart cards, and flash memory devices (e.g., EEPROMs, cards, sticks, key drives, etc.). Furthermore, the various storage media described herein include media readable by one or more devices and / or other machines for storing information.

[0140] It should be understood that the specific order or hierarchical structure of the steps in the process shown herein is an example of an exemplary approach. It should be understood that, based on design priorities, the specific order or hierarchical structure of the steps in the process can be rearranged within the scope of this disclosure. The appended method claims provide a variety of step elements in sample order, but are not limited to the specific order or hierarchical structure shown.

[0141] The descriptions relating to the embodiments presented herein are provided so that any person with ordinary skill in the art of this disclosure may utilize or implement the disclosure. Various variations of such embodiments are readily apparent to a person with ordinary skill in the art of this disclosure, and the general principles defined herein can be applied to other embodiments without departing from the scope of this disclosure. Accordingly, this disclosure is not limited by the embodiments presented herein and should be interpreted in the broadest sense consistent with the principles and novel features presented herein.

Claims

1. A method performed in a computing device for motion capture using an artificial neural network model, The stage of inputting camera images into the first artificial neural network model, Based on the process of utilizing the first artificial neural network model, the process involves generating event information relating to the changes of one or more objects in the motion capture space, Based on the event information, a step is to generate first motion capture data relating to one or more objects related to the camera image, Includes, The aforementioned event information is, The first event information relating to the first object that newly appeared in the motion capture space, The second event information relating to the second object that has left the motion capture space includes one or more of the following: Based on the process of utilizing the first artificial neural network model, the step of generating event information relating to the changes of one or more objects in the motion capture space is: When the first object newly appears in the motion capture space, the operation of generating first event information is performed, The process includes one or more steps of the following: when the second object leaves the motion capture space, an action to generate second event information; Based on the event information, the step of generating first motion capture data relating to one or more objects in the camera image is: Based on the event information, the step of determining whether or not to activate the skeleton asset corresponding to each of one or more objects in the motion capture space, The process includes the step of generating first motion capture data relating to one or more objects related to the camera image, based on the camera image and the currently activated skeleton asset. Based on the event information, the step of determining whether or not to activate each of the skeleton assets corresponding to one or more objects in the motion capture space is as follows: If the event information includes first event information, the first skeleton asset, which is the skeleton asset corresponding to the first object, is activated. If the event information includes a second event information, the process includes the step of deactivating the second skeleton asset, which is the skeleton asset corresponding to the second object. method.

2. In the method according to claim 1, The first artificial neural network model is an artificial neural network model that has been trained to generate event information when an object included in the motion capture space changes based on object detection. method.

3. In the method of claim 2, The first artificial neural network model is, The process involves generating a training dataset based on video data from a reference camera capturing the rotational movement of each object, and corresponding label data for each object. The steps include training the first artificial neural network model to identify and classify each object in the reference camera image based on the aforementioned training dataset, and the trained neural network model is... method.

4. In the method according to claim 1, Based on the event information, the step of generating first motion capture data relating to one or more objects in the camera image is: A step of generating first verification data based on a process of verifying the position of a marker for motion capture that corresponds to an object included in the aforementioned camera image, The steps include: if there is an abnormality in the first verification data, generating alarm information; method.

5. In the method according to claim 1, Furthermore, the process includes a step of generating second motion capture data, which is corrected motion capture data, based on the step of inputting the first motion capture data into the second artificial neural network model. method.

6. In the method according to claim 5, Based on the step of inputting the first motion capture data into the second artificial neural network model, the step of generating second motion capture data, which is corrected motion capture data, is: The first motion capture data includes a step of filtering the data relating to the contact points of the object, Based on the process of verifying the validity of the data relating to the contact points of the aforementioned object, a step is made to generate second verification data, The process includes the step of generating the second motion capture data based on the second verification data, method.

7. In the method according to claim 6, Based on the step of inputting the first motion capture data into the second artificial neural network model, the step of generating the second motion capture data, which is corrected motion capture data, is: A step of generating virtual motion capture data for one or more third objects included in the first motion capture data based on reference motion capture data, The process includes a step of generating second motion capture data based on a step of inputting virtual motion capture data of one or more third objects and the first motion capture data into a second artificial neural network model, method.

8. In the method of claim 7, The step of generating virtual motion capture data for one or more third objects included in the first motion capture data, based on reference motion capture data, The steps include identifying one or more third objects included in the first motion capture data, The process includes a step of generating virtual motion capture data for one or more third objects based on a step of comparing the skeleton of a reference object included in the reference motion capture data with the skeleton of one or more third objects, method.

9. A computer program stored on a computer-readable storage medium that causes a computing device to perform actions for motion capture using an artificial neural network model, The aforementioned operation is, The operation of inputting camera images into the first artificial neural network model, Based on the process of utilizing the first artificial neural network model, an operation is performed to generate event information relating to the changes of one or more objects in the motion capture space, Based on the aforementioned event information, an operation is performed to generate first motion capture data relating to one or more objects in the camera image, Includes, The aforementioned event information is, The first event information relating to the first object that newly appeared in the motion capture space, The information includes one or more of the following: second event information relating to a second object exiting the motion capture space, Based on the process of utilizing the first artificial neural network model, the operation of generating event information related to the changes of one or more objects in the motion capture space is: When the first object newly appears in the motion capture space, the operation of generating first event information is performed, When the second object leaves the motion capture space, the operation includes one or more of the following: an operation to generate second event information, Based on the aforementioned event information, the operation to generate first motion capture data relating to one or more objects in the camera image is as follows: Based on the aforementioned event information, an action is taken to determine whether or not to activate the skeleton asset corresponding to each of one or more objects in the motion capture space, This includes the operation of generating first motion capture data relating to one or more objects related to the camera image, based on the camera image and the currently activated skeleton asset, Based on the aforementioned event information, the operation of determining whether or not to activate each of the skeleton assets corresponding to one or more objects in the motion capture space is as follows: If the event information includes the first event information, the first skeleton asset, which is the skeleton asset corresponding to the first object, is activated. If the event information includes a second event, the operation includes deactivating the second skeleton asset, which is the skeleton asset corresponding to the second object. A computer program stored on a computer-readable storage medium.

10. A computing device for motion capture using an artificial neural network model, One or more processors, memory, Includes, The one or more processors described above are: The camera image is input into the first artificial neural network model. Based on the process of utilizing the first artificial neural network model, event information relating to the changes of one or more objects in the motion capture space is generated. Based on the aforementioned event information, first motion capture data relating to one or more objects in the camera image is generated. The aforementioned event information is, The first event information relating to the first object that newly appeared in the motion capture space, The information includes one or more of the following: second event information relating to a second object exiting the motion capture space, Based on the process of utilizing the first artificial neural network model, generating event information related to the changes of one or more objects in the motion capture space is: When the first object newly appears in the motion capture space, the operation of generating first event information is performed, The operation includes one or more of the following: when the second object leaves the motion capture space, an operation to generate second event information; Based on the aforementioned event information, generating first motion capture data relating to one or more objects in the camera image is: Based on the aforementioned event information, it is determined whether or not to activate the skeleton asset corresponding to each of one or more objects in the motion capture space. This includes generating first motion capture data relating to one or more objects related to the camera image based on the camera image and the currently activated skeleton asset, Based on the aforementioned event information, determining whether or not to activate the skeleton asset corresponding to each of one or more objects in the motion capture space is: If the event information includes the first event information, the first skeleton asset, which is the skeleton asset corresponding to the first object, is activated. If the event information includes a second event, the second skeleton asset, which is the skeleton asset corresponding to the second object, is deactivated. Computing device.

Images