Kinematic joint and chain physical tool for augmented reality

Abstract

Examples related to kinematic joint and chain physical tools for augmented reality implementations are provided. One aspect includes a computing device for kinematic joint simulation, the computing device comprising a display and a processor coupled to a storage system storing instructions that, when executed by the processor, cause the processor to: present a chain simulation interface comprising a plurality of graphical control elements, each graphical control element is configured to adjust at least one physical parameter, receive a selection from a user using a plurality of graphical control elements, update a chain model based on the received selection, and display and animation the updated chain model on a display using a physical engine during kinematic motion simulation, wherein the chain model comprises a plurality of rigid mesh models linked together by spring elements arranged in a tree configuration, thereby forming an assembly skeleton.

Description

Cross-reference to related applications

[0001] This application claims priority to U.S. Provisional Patent Application Serial No. 63 / 493,296, filed March 30, 2023, and U.S. Patent Application Serial No. 18 / 317,802, filed May 15, 2023, the entire contents of which are incorporated herein by reference for all purposes. Background Technology

[0002] In recent years, the field of augmented reality (AR) has made significant progress. AR is an interactive integration of digital content through various sensory forms. One implementation of AR involves visually overlaying digital content onto the physical world, typically the user's environment, in real time. This experience can be provided, for example, in mobile devices equipped with cameras that capture images of the environment and display the overlaid digital content, or through transparent displays such as glasses and head-up displays that display digital content while allowing the user to view the physical environment through transparent lenses. Summary of the Invention

[0003] Examples related to kinematic joint and chain physics tools for augmented reality implementation are provided. One aspect includes a computing device for kinematic joint simulation, the computing device comprising: a display and a processor coupled to a storage system of stored instructions, which, when executed by the processor, cause the processor to: present a chain simulation interface comprising a plurality of graphical controls, each graphical control configured to adjust at least one physical parameter; receive selections from a user using the plurality of graphical controls; update a chain model based on the received selections; and, during kinematic motion simulation, display and animate the updated chain model on the display using a physics engine, wherein the chain model comprises a plurality of rigid body mesh models linked together by spring elements arranged in a tree configuration to form an assembly skeleton, wherein during kinematic motion simulation, movement of a parent rigid body mesh model generates a force in at least one adjacent spring element, which in turn causes movement of a subordinate rigid body mesh model.

[0004] This summary is provided to introduce, in a simplified form, the selection of concepts that will be further described in the detailed embodiments described below. This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter. Furthermore, the claimed subject matter is not limited to the implementation of solutions to any or all the shortcomings mentioned in any part of this disclosure. Attached Figure Description

[0005] Figure 1 An implementation of an example kinematic model development tool across various platforms is shown.

[0006] Figure 2A and2B It shows in Figure 1 The example chain model rendered in the graphical user interface of the Dynamic Chain Tool development tool, and its usage. Figure 1 The simulation of augmented reality in augmented reality environments on social media platforms.

[0007] Figure 3A and 3B It shows the use of Figure 1 The augmented reality program displays the force exerted on the example chain model by the tracked object in the augmented reality environment through a social media platform.

[0008] Figure 4A and 4B Shown Figure 1 The example hair chain model and sub-models are rendered in the graphical user interface of the Dynamic Chain Tool development tool.

[0009] Figure 5A and 5B It shows in Figure 1 The development tools' dynamic chain tool graphical user interface attaches hair chain models and earring chain models to the head object, and uses... Figure 1 The corresponding head object tracked by the augmented reality program on the graphical user interface of the social media platform.

[0010] Figure 6A and Figure 6B It shows the use of Figure 1 An earring model in world space that is affected by applied forces in an augmented reality application on a social media platform's graphical user interface.

[0011] Figure 7 Shown Figure 1 The development tool provides an example chain simulation graphical user interface with graphical control elements for adjusting various chain simulation parameters.

[0012] Figure 8 An example data structure for representing a chain model is shown, which can be implemented in Figure 1 In the development tools.

[0013] Figure 9 The flowchart illustrates an example method for kinematic joint simulation, which can be implemented using... Figure 1 The development tools can be implemented using any suitable hardware and software.

[0014] Figure 10 It shows where it can be deployed. Figure 1 A schematic diagram of an example computing environment for the development tools. Detailed Implementation

[0015] Augmented reality (AR) is an expanding field with many different applications across various platforms. As an emerging technology, several aspects of AR applications remain underdeveloped. For example, the physical simulation of objects in AR display applications can be a difficult and time-consuming task. As a more concrete example, the implementation of kinematic joints and chain physics, such as the representation of physical hinges using dynamic chains, in AR may require a high level of technical expertise and complex coding. This may limit the types of interactions and movements that can be achieved in AR experiences.

[0016] In light of the above observations, examples related to kinematic joint and chain physics tools for augmented reality applications are provided. Among many implementations, kinematic model development kits are provided. These kits can include a variety of tools enabling developers and users to implement kinematic modeling and simulation. In some examples, the kits include dynamic chain tools for implementing kinematic joint and chain physics for various applications, including but not limited to AR. Dynamic chain tools address the current challenges of implementing kinematic joint and chain physics in AR by simplifying the technical process of generating and simulating chain models. Some aspects of this implementation include the use of customizable kinematic joints and dynamic chain elements. By enabling the adjustment of various physical parameters, the tool can be implemented with additional flexibility and customization, allowing for a wider range of dynamic movement and interaction in AR experiences. By presenting a simplified user interface with predefined options, such tools make it easier for a wider range of developers and users to implement kinematic joint and chain physics, allowing them to create more realistic and engaging AR experiences with less technical expertise and complexity required to implement physical simulations.

[0017] Figure 1 An implementation of an example kinematic model development kit 102 across various platforms 104 is illustrated. In the depicted example, the kinematic model development kit 102 includes a dynamic chain tool 106 and a physics engine 108 for implementing kinematic joint and chain physics. Various types of physics engines can be utilized, including computer software capable of simulating certain physical systems. To enable ease of use and greater flexibility for developers and users, the dynamic chain tool 106 includes a graphical user interface (GUI) 110 for creating, modifying, and / or customizing dynamic chain components, as well as adjusting the physical parameters of these components.

[0018] A kinematic model development kit 102 is available, enabling developers and users across different platforms 104 to implement kinematic modeling for various applications, such as kinematic joints and chain physics. For example, the kinematic model development kit 102 can be provided to various media content platforms 104, such as social media platform 104A and gaming platform 104B. In a more specific example, the kinematic model development kit 102 can be provided to short-video social media platforms to enable the simulation of kinematic joints and chain physics in AR. One use case includes implementing an AR program 112 on a user's mobile device on such a platform. The AR program 112 can utilize the dynamic chain tool 106 in the kinematic model development kit 102 to simulate and render a chain model 114. A chain model refers to a graphical computer model comprising a set of mesh models arranged and configured as objects in a tree structure. Objects can be interconnected through parent / subordinate object relationships, creating a hierarchy of objects. The edges of the tree representing parent / subordinate relationships can be logically represented using spring elements that link objects or mesh models together. For kinematic joint and chain physics simulations, the movement of a parent object affects the movement of its subordinate objects.

[0019] The chain model can be generated in various ways, including by using the dynamic chaining tool 106, and can be provided from various sources. In some implementations, the chain model 114 is imported from an external source. For example, the dynamic chaining tool 106 can be used on an external device such as a laptop / desktop computer to generate the dynamic chain model 114. Users can adjust and modify the dynamic chain settings on such devices, and the model 114 can be imported into mobile devices for use with AR applications. In some implementations, the dynamic chain settings can be modified on the mobile device receiving the chain model 114. The chain model 114 can also be generated on a local device. For example, the dynamic chaining tool 106 and the provided GUI 110 can be used to generate the chain model 114 locally according to user preferences.

[0020] In some implementations, the chain model 114 is displayed and animated as an overlay on video data recorded from a mobile device's camera. For example, users on short-video social media platforms can create videos by combining video data received from their mobile device's camera with a chain model displayed overlaid on said video data. The dynamic chain tool 106 can be implemented using various systems on mobile devices to render the chain model 114 and animate it in a manner that interacts with content from video data from augmented reality applications. For example, objects can be detected within the video data, and interactions between objects and the simulated chain model 114 can be tracked. Spatial interactions (such as collisions) between the chain model 114 and detected objects can cause the chain model 114 to behave differently, where such behavior can be simulated using a physics engine 108. Figure 1In the illustrated example, the device implementing the dynamic chain tool includes a pose tracking program 116 for detecting and tracking objects such as heads and hands in video data. Such detection programs can be implemented using various techniques, including machine vision algorithms. In some implementations, a trained machine learning model is applied to detect and track objects within the video data. It is readily understood that the above implementation can be applied to video data in real-time or offline.

[0021] Figure 2A An example dynamic chain tool GUI 200 depicting a 3D chain model 202 is shown. Using the dynamic chain tool 106, the rendered chain model 202 can be presented to the user for various modifications and configurations. The dynamic chain tool GUI 200 can be implemented to provide developers or users with various options and presets to generate chain models of different types and configurations. Figure 2A In the illustrated example, chain model 202 comprises four objects, represented by a linearly arranged cubic mesh model, where each object (except the rightmost one) is the parent object of its immediately right-hand neighbor. For kinematic joint and chain physics simulations, the movement of the parent object affects the movement of the subordinate objects. An object can also have multiple subordinate objects, creating a branching structure. This allows for a large number of degrees of freedom, making it possible to simulate complex movements.

[0022] Figure 2B An example social media platform GUI 250 is shown, which depicts a chain model 202 as a rendered overlay on top of video data. In the depicted example, the social media platform is a short-video social media platform with a user interface 250 having various button icons 252 for navigation and performing various actions. Example actions include liking, commenting, favorited, and sharing the currently displayed short video. In short-video social media platforms, video data is typically received from the camera of a mobile device implementing an AR application. In the depicted example, the video data is displaying the user's upper body 254. The chain model 202 can be rendered and displayed as an overlay on the video data, allowing it to be simulated according to the user's preferences. For example, the chain model 202 can be simulated as representing forces acting on it. In some implementations, the chain model 202 is anchored to objects in the video data, and simulations of the movement of objects according to various physical systems can be performed using, for example, physics engines and dynamic chaining tools.

[0023] As described above, the content in the video data can influence the movement of the chain model 202. Various techniques, such as objects, can be used to detect content in the video data, including but not limited to machine vision and machine learning techniques. In some implementations, object detection algorithms are used to detect and track body parts, such as a person's hands, face, eyes, ears, nose, mouth, torso, arms, legs, feet, or hips. Collision detection can be performed to determine the interaction between the object and the chain model 202, and the final movement of the chain model 202 can be simulated and displayed accordingly.

[0024] The chain model can be configured in various ways. For example, Figure 2A and 2B The chain model 202 depicted consists of four distinct mesh models positioned separately from each other. Relationships between objects (mesh models) can be defined through parent / subordinate object relationships, where the movement of a parent object affects the movement of subordinate objects in the simulation environment. The relationships between objects and their associated physical parameters can be described and implemented as virtual spring elements, which include various physical parameter properties that can be used to simulate objects according to the physical system. These properties can be configured according to the preferences of the developer or user. For example, a user can generate chain models with certain physical parameters to mimic real-life objects such as hair, wigs, and accessories.

[0025] Movement in a chain model can be simulated using kinematic joints and dynamic chain physics, where the mesh model constituting the chain model is simulated as interconnected elements, such as joints and bones, in a virtual environment. Example use cases for chain model simulation can include simulating the movement of various objects that can be represented using chain structures, including human body parts such as hair and limbs. The dynamics of objects within a chain model are determined by several physical parameters. In a dynamic chain system, the motion of an object is determined by the relative positions and velocities of the elements that make up the object. The movement of a parent object will affect the movement of its subordinate objects. For example, in a character's arm, the movement of the shoulder joint will affect the movement of the elbow joint, which in turn will affect the movement of the wrist joint. This creates a cascading effect, where the movement of the root object (in this case, the shoulder joint) propagates through the remaining objects in the chain, ultimately affecting the movement of the end effector (in this case, the hand).

[0026] Simulated movement of a chain model can be based on forces applied to the chain model, such as forces generated by collisions with objects. Movement of subordinate objects can be generated by the movement of their respective parent objects. Collisions with other objects can include objects such as another computer-generated model or objects detected in video data. Figure 3A and 3B The diagram illustrates the force exerted on an example chain model 300 by a tracked object 302 in an augmented reality environment displayed via a social media platform GUI 250. The chain model 300 comprises a linearly arranged mesh of five spheres. Figure 3A A chain model 300 in a stationary position is shown. Upon collision with the tracked object 302 (a finger), its physical parameters and a physics engine are used to simulate the movement of the chain model 300. The simulated motion is rendered and displayed accordingly, as shown... Figure 3B As shown, various machine vision and / or machine learning techniques can be used to detect and track fingers in video data. In some implementations, pose tracking algorithms are applied to detect and track objects in video data.

[0027] To achieve realistic movement, each object can have various physical parameters that determine its behavior. Example physical parameters include stiffness, damping, elasticity, inertia, and applied force. Stiffness determines the resistance to two adjacent objects returning to their original relative distance. Damping adjusts the speed at which an object decelerates. Elasticity controls the degree of elasticity with which two adjacent objects return to their original relative direction. Inertia affects the effort required to move an object. Force, depending on the settings, controls the magnitude of the force applied to world space, local space, or relative to the object. Depending on the application, different constraints may also be applied. The simulation of some objects may have different constraints. For example, a chain model simulating a realistic human arm may include adjusting certain physical parameters to appropriate settings to maintain a constant relative distance between the parent object and its subordinate objects(s), similar to simulating a physical rigid hinge.

[0028] Depending on the application and user preferences, other forms and structures of chain models can be generated. In some implementations, multiple chain models are rendered together to simulate the desired object. Figure 4A An example hair chain model 400 rendered in the Dynamic Chain Tool GUI 200 is shown. The hair chain model 400 can be configured in several ways. In the depicted example, the hair chain model 400 includes multiple sub-models 402, each of which can be configured as a chain model. Each sub-model 402 corresponds to a different part of the hair and can be made from multiple mesh models. For illustrative purposes, the hair chain model 300 and its constituent sub-models 402 are typically not depicted as mesh models. Multiple sub-models 402 can also be configured as a single chain model by associating each chain model with a single invisible parent node (i.e., a parent node without a mesh model).

[0029] Hair can be simulated by applying certain physical parameters to the chain model. For example, to simulate hair as an unstretched rigid object, virtual spring elements between mesh models within the chain model can be configured with physical parameters that describe properties more like rods than springs. This can cause the chain model to behave as a physically rigid hinge, where the relative positions between predetermined points in the mesh model pair are constant. Figure 4BThe sub-model 402 rendered in the Dynamic Chain Tool GUI 200 is shown. Sub-model 402 comprises five mesh models 404 arranged in chains of adjacent objects. The relationship between two adjacent objects, which can be represented as virtual spring elements, can have physical parameters describing rigid bar elements. For example, in some implementations, the virtual spring elements within sub-model 402 can have elastic parameters with zero values ​​to simulate rigid elements.

[0030] As mentioned above, the simulation of kinematic joints and dynamic chain physics can be affected by a variety of physical parameters, including but not limited to stiffness, damping, elasticity, and inertia. Other considerations affecting the simulation can include the properties of the mesh model within the chain model. For example, each object in the chain model can be simulated as a rigid body or a deformable body. Other types of forces can also be applied, such as constant forces applied to objects in world space. For example, gravity can be applied to the chain model. Forces can also be applied in local space or in a frame of reference relative to another object. In some implementations, forces are applied in camera space. Another consideration includes chain models with fixed ends. In AR applications, displaying the chain model as an overlay on video data allows the movement of the chain model to be influenced by the content within the video data. In some implementations, one or more objects within the chain model are anchored to detected objects in the video data. For example, objects within the chain model can have fixed ends, such that the position of the object relative to the detected object is constant. This can be used to simulate various scenarios in AR applications. Examples include simulating digital hair, earrings, and other items attached relative to anchor objects. The movement of the anchor objects can be tracked, and the chain model can be simulated to move accordingly.

[0031] Figure 5A The diagram illustrates a hair chain model 400 and an earring chain model 500 attached to a head object 502 in the Dynamic Chain Tool GUI 200. Chain models 400 and 500 can be configured in various ways. For example, the hair model 400 can be implemented as one or more chain models. Each strand of hair can be a separate chain model, or a strand of hair can be a dependent object of a single, invisible parent object. It is readily understood that any type and number of chain models can be attached to any type of anchor object. For example, a chain model can be an accessory or headdress. In some implementations, the chain model is one of the following: earrings, necklaces, pendants, bracelets, ties, wigs, crowns, hats, and hoods. Furthermore, the anchor object can be any object, including various body parts such as a person's hands, face, eyes, ears, nose, mouth, torso, arms, legs, feet, or hips.

[0032] Figure 5BA hair chain model 400 and an earring chain model 500 attached to a tracked head object 504 are shown on a social media platform GUI 250. The head object 504 is detected and tracked from video data, which can be performed using various tracking algorithms and machine vision techniques. In the depicted example, the video data contains two people. In such cases, determining which head object to track and anchor the hair and earring models 500, 502 can be decided by various methods, such as by the first detected object or by user selection. Once the chain models 500, 502 are attached to the tracked object 504, movement of the object (i.e., movement of the person in the video) can cause a necessary movement of the chain models 500, 502, because the positions of the fixed ends of the chain models 500, 502 are constrained to be constant relative to the head object.

[0033] In addition to the movement of the tracked anchor object, other forces can also be applied to the chain model, causing it to move. Examples of such forces include collisions with other tracked objects and constant forces relative to a predetermined frame of reference. Figure 6A and 6B An earring model 500 is shown, which is affected by applied forces in world space on the social media platform GUI 250. Figure 6A An earring model 500 is shown in a stationary position, fixed to the tracked head object 504. Figure 6B The image shows an earring model 502 that is stretched due to an applied downward force. Figure 6B In this case, the downward force can be used to simulate gravity. Since the tracked object 504 does not move, the fixed end remains in place (near the ear), while the free end stretches downward. The stretching capacity of the earring model 500 and the amount of stretching due to the applied force can be determined based on the model's physical parameters.

[0034] The applied force can be configured using a dynamic chain model tool, which provides various configuration options for implementing the applied force. The force can also be specified relative to different frames of reference, such as world space, local space, or another object. In some implementations, the applied force is relative to camera space. Different configurations can be used to simulate different scenes. For example, gravity can be simulated by applying a force relative to world space. In some implementations, the direction of the force can be determined using various methods that determine orientation in the real environment. Example methods include using the mobile device's gyroscope to determine the mobile device's orientation, and thus its camera's orientation. This information can be used, for example, to determine the world space frame of reference. In another example, the applied force can be relative to another object to simulate a magnetic object.

[0035] Various types of physical parameters can be adjusted for each object and / or spring element in each chain model. In some implementations, physical parameters are adjusted for each chain model. The physics engine can then use the adjusted parameters to simulate and render the chain models accordingly. The user interface can be equipped with graphical controls for adjusting various parameters associated with one or more chain models, objects, and / or spring elements. Examples of graphical controls include radio buttons, drop-down menus, checkboxes, and text boxes. In some implementations, physical parameters can be adjusted in real time as the chain model is displayed as an overlay on video data.

[0036] Figure 7 An example chain simulation GUI 700 is shown, featuring graphical controls for adjusting various chain simulation parameters. By providing these graphical controls, developers or users can quickly adjust the behavior of (multiple) objects, as well as the simulated kinematic joints and chain physics, offline or in real-time. The graphical controls can be configured to adjust the physical parameters of objects such as mesh models, spring elements, and chain models. Thus, in some implementations, each mesh model and / or spring element can be configured with different physical parameters. New changes to parameter settings can be used to update the chain model, and the physics engine can be used to simulate the updated chain model accordingly. This makes it easier for developers and users to create realistic and engaging AR experiences with less technical expertise and complexity required for physics simulations, while also allowing for greater flexibility and customization within a user-friendly interface. In some implementations, the interface includes a visual representation of the simulated objects and connections, allowing users to easily see the effects of their adjustments in real-time. Chain models can also be simulated and displayed with textures applied to their mesh models.

[0037] In the depicted example, the chain simulation GUI 700 includes graphical controls for adjusting physical parameters including damping, elasticity, stiffness, inertia, and force. For each of these physical parameters except "force," the chain simulation GUI 700 includes a slider graphical control 702 and a corresponding text box 704 displaying the value. The "Damping" control allows the user to control the amount of resistance an object exerts against movement. A high damping value will cause the motion between objects to decelerate rapidly. The "Elasticity" control allows the user to control the magnitude of the force applied to return an object to its original direction. A higher elasticity value will cause the object to accelerate back to its starting position more quickly. The "Stiffness" control allows the user to control the amount of resistance an object exerts against its original direction. The "Inertia" control allows the user to control the amount of movement required to move an object. The "Force" control allows the user to control the amount of force applied to the object. Any other parameters can also be implemented.

[0038] Forces can be applied in different frames of reference, such as in local space, world space, or relative to another object. The chain simulation GUI 700 includes textbox graphical controls 706-710 for determining the X, Y, and Z magnitudes of the force to be applied. The chain simulation GUI 700 also includes checkboxes 712 for determining whether the force is applied in local space or world space, and checkboxes 714 for determining whether the force is relative to another object. Furthermore, the example GUI includes graphical controls in the form of drop-down menus 716 for selecting the object associated with that object. The ability to apply forces in local or world space and to make the force relative to another object provides more possibilities for dynamic movement. It can be readily understood that any type of graphical control can be used for parameter adjustment.

[0039] In some implementations, multiple presets are provided to further simplify the adjustment process. Presets can be a set of customizable or pre-defined adjustable parameters. The chain simulation GUI 700 includes a drop-down menu 718 for selecting presets. As shown, the "Custom" preset is currently selected. Presets can be selected to quickly adjust parameters to pre-defined values. In further implementations, parameters can be further adjusted after selecting a preset. Examples of a set of named presets may include "Loose," "Dampened," "Springy," "Elastic," "Stiff," and "Rigid." "Loose" simulates objects with low resistance to movement, allowing for smooth and easy motion. "Dampened" simulates objects with a damping effect to reduce oscillations. "Springy" simulates objects with spring-like behavior. "Elastic" simulates objects capable of stretching and compressing, similar to a rubber band. "Stiff" simulates objects with high resistance to movement. "Rigid" simulates objects that are fixed and do not allow any movement, similar to welded objects, but may break or shatter under strong forces.

[0040] A chain simulation GUI 700 can be implemented to adjust parameters of a chain model or individual elements within a chain model. In some implementations, the chain simulation GUI 700 includes options for configuring the parameters of the chain model to vary according to a predetermined function. For example, options for changing parameter values ​​of elements within a cross-chain model can be implemented. These options can include different functions or patterns where values ​​can vary. Example patterns include linear, logarithmic, and exponential increases and decreases in the value of a given parameter of an element within a cross-chain model.

[0041] Figure 8An example data structure 800 for representing a chain model is shown. As described above, a chain model can be represented as multiple mesh models and spring elements configured and arranged in a tree structure. Thus, the data structure can include mesh models classified as base objects 802, intermediate objects 804, or leaf objects 806. In the depicted example, the mesh model is a rigid body model. A deformable mesh model can also be implemented. To define the tree structure, the mesh model data structure can include information about its parent object 808 and dependent objects(s) 810. Spring elements 812 logically represent the relationship between parent and dependent objects. Spring elements 812 can include a list 814 of attached objects describing two objects linked together by corresponding spring elements.

[0042] The mesh model data structure may include additional information, such as physical parameters 816 and mesh geometry 818. In some implementations, the mesh model data structure includes texture information. In the depicted example, physical parameters 816 include information describing the inertia of the mesh model. Spring elements include physical parameter information 820 describing their damping, elasticity, stiffness, and inertial properties. The data structure may also include additional or different physical parameters. Any set of physical parameters can be used for the mesh model and spring elements. For example, the data structure may include information describing the mass of an object. Physical parameters can also be correlated with each other. For example, since inertia can be represented as the work required to change the velocity of an object of a given mass, mass and inertia can be correlated with each other according to a predefined relationship. Other configurations of the data structure can be implemented. For example, the chain model can be described using physical parameters that affect all objects and elements within the chain model. In this case, physical parameters can be recorded in a single instance, such as in the base mesh model data structure. In some implementations, the mesh model data structure stores physical parameter information applied to one or more adjacent spring elements.

[0043] Figure 9 A flowchart of an example method 900 for kinematic joint simulation is shown. In 902, method 900 includes providing a chain simulation interface comprising multiple graphical control elements. The chain simulation interface may be a GUI that allows a user to select a chain model or elements within a chain model for configuration. For example, the chain simulation interface may include graphical control elements for selecting which element (such as a spring element, mesh model, or chain model) to configure. The graphical control elements can be implemented for various functions. In some implementations, the multiple graphical control elements include at least one graphical control element for adjusting one or more physical parameters. Example physical parameters include, but are not limited to, stiffness, damping, elasticity, and inertia.

[0044] In some implementations, the plurality of graphics control elements includes a graphics control element for selecting a preset. The preset corresponds to a predetermined selection for at least one of the plurality of graphics control elements. In some implementations, after selecting a preset, the user can further adjust parameters. Another type of graphics control element that can be implemented includes a graphics control element for indicating whether an applied force exists in the simulation. The applied force can be included in the simulation of the chain model relative to a reference frame. For example, the simulation can include applied forces in local space, world space, or relative to an object. In some implementations, the applied force is relative to camera space. The chain simulation interface can include at least one graphics control element for indicating whether a force is applied in local space, world space, or relative to an object.

[0045] In some implementations, multiple graphical control elements include graphical control elements for indicating anchor objects. For example, a drop-down menu can be implemented for the user to select an object to which a mesh model within a chain model is attached. This anchoring relationship defines and implements how the mesh model is positioned relative to the anchor object. Example anchor objects can include various objects or body parts, such as the user's hands, face, eyes, ears, nose, mouth, torso, arms, legs, feet, and hips within the camera's field of view. During simulation, anchor objects can be determined from video data received from the camera. Various methods can be implemented to determine anchor objects. In some implementations, a pose tracking algorithm is implemented to detect and track anchor objects. Machine learning algorithms can also be used to detect and track anchor objects. For example, the position of the anchor object can be determined by applying a trained machine learning model.

[0046] In method 904, method 900 includes receiving selections from a user using multiple graphical control elements. These selections may include adjusted parameter values ​​corresponding to elements such as spring elements, mesh models, or chain models.

[0047] In 906, method 900 includes updating the chain model based on the received selection. Updating the chain model may include updating the parameters that were adjusted in the received selection. The update method may vary depending on how the chain model is implemented. For example, in some implementations, the chain model is represented by a data structure that stores parameter information. In such implementations, updating the chain model may include updating the parameter information.

[0048] In method 900 (908), the updated chain model is displayed and animated using a physics engine during kinematic simulation. The chain model can be any 3D computer model. For example, a chain model can represent real-world items such as earrings, necklaces, pendants, bracelets, ties, wigs, crowns, hats, and hoods. In some implementations, the chain model is displayed and animated in real-time on a monitor as an overlay on video data received from a camera. Such implementations can be performed on a variety of devices, including computing devices such as mobile devices. The chain model can be implemented in many different ways. In some implementations, the chain model comprises multiple mesh models linked together by spring elements arranged in a tree configuration. The mesh model can be simulated as a rigid body model or a deformable body model.

[0049] In some embodiments, the methods and processes described herein may be bound to a computing system with one or more computing devices. In particular, such methods and processes may be implemented as computer applications or services, application programming interfaces (APIs), libraries, and / or other computer-program products.

[0050] Figure 10 A non-limiting embodiment of a computing system 1000 capable of implementing one or more of the methods and processes described above is illustrated schematically. The computing system 1000 is shown in a simplified form. The computing system 1000 can be embodied in its deployable... Figure 1 The development tools provide an example computing environment. The computing system 1000 may take one or more of the following forms: personal computer, server computer, tablet computer, home entertainment computer, network computing device, gaming device, mobile computing device, mobile communication device (e.g., smartphone) and / or other computing device, as well as wearable computing devices such as smartwatches and head-mounted augmented reality devices.

[0051] The computing system 1000 includes a logic processor 1002, volatile memory 1004, and non-volatile storage device 1006. The computing system 1000 may optionally include a display subsystem 1008, an input subsystem 1010, a communication subsystem 1012, and / or... Figure 10 Other components not shown.

[0052] The logic processor 1002 includes one or more physical devices configured to execute instructions. For example, the logic processor 1002 may be configured to execute instructions that are part of one or more of the following: applications, programs, routines, libraries, objects, components, data structures, or other logical constructs. Such instructions may be implemented to perform tasks, implement data types, transition the state of one or more components, achieve technical effects, or otherwise achieve desired results.

[0053] The logic processor 1002 may include one or more physical processors (hardware) configured to execute software instructions. Additionally or alternatively, the logic processor 1002 may include one or more hardware logic circuits or firmware devices configured to execute hardware-implemented logic or firmware instructions. The processor of the logic processor 1002 may be single-core or multi-core, and the instructions executed thereon may be configured for sequential, parallel, and / or distributed processing. The various components of the logic processor 1002 may optionally be distributed across two or more separate devices that may be remotely located and / or configured for collaborative processing. Aspects of the logic processor 1002 may be virtualized and executed by remotely accessible networked computing devices configured in a cloud computing configuration. It is understood that, in this case, these virtualized aspects run on different physical logic processors on various different machines.

[0054] The non-volatile storage device 1006 includes one or more physical devices configured to store instructions executable by a logic processor to implement the methods and processes described herein. When implementing such methods and processes, the state of the non-volatile storage device 1006 can be changed, for example, to store different data.

[0055] The non-volatile storage device 1006 may include removable and / or built-in physical devices. The non-volatile storage device 1006 may include optical memory (e.g., CD, DVD, HD-DVD, Blu-ray disc, etc.), semiconductor memory (e.g., ROM, EPROM, EEPROM, flash memory, etc.), and / or magnetic memory (e.g., hard disk drive, floppy disk drive, magnetic tape drive, MRAM, etc.), or other high-capacity storage technologies. The non-volatile storage device 1006 may include non-volatile, dynamic, static, read / write, read-only, sequential access, location-addressable, file-addressable, and / or content-addressable devices. It should be understood that the non-volatile storage device 1006 is configured to retain instructions even when the non-volatile storage device 1006 is powered off.

[0056] Volatile memory 1004 may include a physical device containing random access memory. Volatile memory 1004 is typically used by logic processor 1002 to temporarily store information during software instruction processing. It should be understood that volatile memory 1004 typically does not continue storing instructions when power is lost.

[0057] The logic processor 1002, volatile memory 100, and non-volatile storage device 1006 can be integrated together into one or more hardware logic components. For example, such hardware logic components may include field-programmable gate arrays (FPGAs), application-specific integrated circuits (PASICs / ASICs), application-specific standard products (PSSPs / ASSPs), systems-on-a-chip (SOCs), and complex programmable logic devices (CPLDs).

[0058] The terms "module," "program," and "engine" can be used to describe aspects of the computing system 1000. These terms are typically implemented in software by a processor to perform specific functions using portions of volatile memory. These functions involve special configuration of the processor to perform translational processing for those functions. Therefore, a module, program, or engine can be instantiated using portions of volatile memory 1004 via a logical processor 1002 that executes instructions stored in non-volatile storage device 1006. It should be understood that different modules, programs, and / or engines can be instantiated from the same applications, services, code blocks, objects, libraries, routines, APIs, functions, etc. Similarly, the same module, program, and / or engine can be instantiated from different applications, services, code blocks, objects, routines, APIs, functions, etc. The terms "module," "program," and "engine" can encompass individual or grouped elements such as: executable files, data files, libraries, drivers, scripts, database records, etc.

[0059] When a display subsystem 1008 is included, it can be used to present a visual representation of data stored by the non-volatile storage device 1006. The visual representation may take the form of a graphical user interface (GUI). Since the methods and processes described herein change the data stored in the non-volatile storage device and thus transform the state of the non-volatile storage device, the state of the display subsystem 1008 can also be transformed to visually represent changes in the underlying data. The display subsystem 1008 may include one or more display devices utilizing virtually any type of technology. Such display devices may be combined with the logic processor 1002, the volatile memory 1004, and / or the non-volatile storage device 1006 in a shared housing, or such display devices may be peripheral display devices.

[0060] When the input subsystem 1010 is included, it may include one or more user input devices, such as a keyboard, mouse, touchscreen, or game controller, or an interface thereto. In some embodiments, the input subsystem 1010 may include or interface with selected Natural User Input (NUI) element portions. Such element portions may be integrated or peripheral, and the translation and / or processing of input actions may be handled on-board or off-board. Exemplary NUI element portions may include microphones for speech and / or voice recognition; infrared, color, stereo, and / or depth cameras for machine vision and / or gesture recognition; head trackers, eye trackers, accelerometers, and / or gyroscopes for motion detection and / or intent recognition; and electric field sensing element portions for assessing brain activity; and / or any other suitable sensors.

[0061] When the communication subsystem 1012 is included, it can be configured to communicatively couple the various computing devices described herein to each other and to other devices. The communication subsystem 1012 may include wired and / or wireless communication devices compatible with one or more different communication protocols. As a non-limiting example, the communication subsystem may be configured to communicate via a wireless telephone network, or a wired or wireless local area network or wide area network (such as an HDMI connected via Wi-Fi). In some embodiments, the communication subsystem may allow the computing system 1000 to send and / or receive messages to and / or from other devices via a network such as the Internet.

[0062] The following paragraphs provide additional description of the subject matter of this disclosure. One aspect provides a computational apparatus for kinematic joint simulation, the apparatus including a display and a processor coupled to a storage system of stored instructions, which, when executed by the processor, cause the processor to: present a chain simulation interface comprising a plurality of graphical control elements, each graphical control element configured to adjust at least one physical parameter; receive selections from a user using the plurality of graphical control elements; update a chain model based on the received selections; and, during kinematic motion simulation, display and animate the updated chain model on the display using a physics engine, wherein the chain model comprises a plurality of rigid body mesh models linked together by spring elements arranged in a tree configuration to form an assembly skeleton, wherein during kinematic motion simulation, movement of a parent rigid body mesh model generates a force in at least one adjacent spring element, which in turn causes movement of a subordinate rigid body mesh model. In this respect, additionally or alternatively, the selection includes values ​​for physical parameters for the spring elements. In this respect, additionally or alternatively, the plurality of graphical control elements includes graphical control elements configured to adjust physical parameters selected from the group consisting of stiffness, damping, elasticity, and inertia. In this respect, additionally or alternatively, the computing device also includes a camera, wherein the chain model is displayed and animated in real time on a display as an overlay on video data received from the camera. In this respect, additionally or alternatively, the plurality of graphical control elements includes a plurality of preset graphical control elements, wherein each preset corresponds to a predetermined selection for at least one of the plurality of graphical control elements. In this respect, additionally or alternatively, the plurality of graphical control elements includes a graphical control element configured to indicate forces applied to the chain model, and at least one graphical control element for indicating whether the force is applied in local space, world space, or relative to an object. In this respect, additionally or alternatively, the plurality of graphical control elements includes a graphical control element for indicating the position of a rigid body mesh model relative to an anchor object in which it is positioned. In this respect, additionally or alternatively, the computing device also includes a camera, wherein the position of the anchor object is determined by applying a trained machine learning model to video data received from the camera. In this respect, additionally or alternatively, the anchor object is a human body part selected from the group consisting of: the user's hands, face, eyes, ears, nose, mouth, torso, arms, legs, feet, and hips. In this regard, additionally or alternatively, the chain model is for items selected from the following groups: earrings, necklaces, pendants, bracelets, ties, wigs, crowns, hats, and hoods.

[0063] On the other hand, a method for simulating kinematic joints is provided, comprising: providing a chain simulation interface including multiple graphical control elements, each graphical control element configured to adjust at least one physical parameter; receiving selections from a user using the multiple graphical control elements; updating the chain model based on the received selections; and displaying and animates the updated chain model using a physics engine during kinematic motion simulation, wherein the chain model includes multiple rigid body mesh models linked together by spring elements arranged in a tree configuration to form an assembly skeleton, wherein during kinematic motion simulation, movement of a parent rigid body mesh model generates a force in at least one adjacent spring element, which in turn causes movement of a subordinate rigid body mesh model. In this respect, additionally or alternatively, the multiple graphical control elements include graphical control elements configured to adjust physical parameters selected from the group consisting of stiffness, damping, elasticity, and inertia. In this respect, additionally or alternatively, the chain model is displayed and animate in real time on a display of a computing device as an overlay of video data received from a camera of the computing device. In this respect, additionally or alternatively, the plurality of graphical control elements includes a plurality of preset graphical control elements, wherein each preset corresponds to a predetermined selection for at least one of the plurality of graphical control elements. In this respect, additionally or alternatively, the plurality of graphical control elements includes a graphical control element configured to indicate a force applied to a chain model, and at least one graphical control element for indicating whether the force is applied in local space, world space, or relative to an object. In this respect, additionally or alternatively, the plurality of graphical control elements includes a graphical control element for indicating a rigid body mesh model among a plurality of rigid body mesh models relative to an anchor object in which it is positioned. In this respect, additionally or alternatively, the position of the anchor object is determined by applying a trained machine learning model to video data received from a camera. In this respect, additionally or alternatively, the anchor object is a human body part selected from the group consisting of: the user's hands, face, eyes, ears, nose, mouth, torso, arms, legs, feet, and hips. In this respect, additionally or alternatively, the chain model is an item selected from the group consisting of: earrings, necklaces, pendants, bracelets, ties, wigs, crowns, hats, and hoods.

[0064] Another aspect provides a mobile device for kinematic joint simulation, the mobile device including a display, a camera, and a processor coupled to a storage system of stored instructions, which, when executed by the processor, cause the processor to: present a chain simulation interface using the display, the chain simulation interface including a plurality of graphical control elements, each graphical control element configured to adjust at least one physical parameter selected from the group consisting of stiffness, damping, elasticity, and inertia; receive selections from a user using the plurality of graphical control elements; update the chain model based on the received selections; and display and animate the updated chain model in real time on the display as an overlay on video data received from the camera, wherein the updated chain model is animated using a physics engine during kinematic motion simulation, and wherein the chain model includes a plurality of rigid body mesh models linked together by spring elements arranged in a tree configuration to form an assembly skeleton, wherein during kinematic motion simulation, movement of a parent rigid body mesh model generates a force in at least one adjacent spring element, which in turn causes movement of a subordinate rigid body mesh model.

[0065] It should be understood that the configurations and / or methods described herein are exemplary in nature, and these specific embodiments or examples should not be considered limiting, as various variations are possible. The specific routines or methods described herein may represent one or more of any number of processing strategies. Thus, the various actions described and / or illustrated may be performed in the order described and / or illustrated, in a different order, in parallel, or omitted. Similarly, the order of the above processes may be changed.

[0066] The subject matter of this disclosure includes all novel and non-obvious combinations and sub-combinations of various processes, systems and configurations, as well as other features, functions, actions and / or properties disclosed herein, and any and all equivalents thereof.

Claims

1. A computing device for simulating kinematic joints, the computing device comprising: monitor; as well as A processor coupled to a storage system of storage instructions, which, when executed by the processor, cause the processor to: The interface presents a chain simulation that includes multiple graphical control elements, each of which is configured to adjust at least one physical parameter. Receive selections from users using the plurality of graphical control elements; Update the chain model based on the received selection; as well as During the kinematic motion simulation, the updated chain model is displayed and animated on the display using a physics engine. The chain model comprises multiple rigid body mesh models linked together by spring elements arranged in a tree configuration to form an assembly skeleton. During the kinematic motion simulation, movement of a parent rigid body mesh model generates a force in at least one adjacent spring element, which in turn causes movement of a subordinate rigid body mesh model.

2. The computing device of claim 1, wherein the selection includes values ​​for physical parameters of the spring element.

3. The computing device of claim 1, wherein the plurality of graphics control elements includes graphics control elements configured to adjust physical parameters selected from the group consisting of: stiffness, damping, elasticity, and inertia.

4. The computing device of claim 1, further comprising a camera, wherein the chain model is displayed and animated in real time on the display as an overlay on video data received from the camera.

5. The computing device of claim 1, wherein the plurality of graphics control elements comprises a plurality of preset graphics control elements, wherein each preset corresponds to a predetermined selection of at least one of the plurality of graphics control elements.

6. The computing device of claim 1, wherein the plurality of graphics control elements includes a graphics control element configured to indicate a force applied to the chain model, and at least one graphics control element for indicating whether the force is applied in local space, world space, or relative to an object.

7. The computing device of claim 1, wherein the plurality of graphics control elements includes a graphics control element for indicating the rigid mesh model of the plurality of rigid mesh models relative to the anchor object in which it is positioned.

8. The computing device of claim 7, further comprising a camera, wherein the position of the anchor object is determined by applying a trained machine learning model to video data received from the camera.

9. The computing device of claim 7, wherein the anchor object is a human body part selected from the group consisting of: the user's hands, face, eyes, ears, nose, mouth, torso, arms, legs, feet, and buttocks.

10. The computing device of claim 7, wherein the chain model is an item selected from the group consisting of: earrings, necklaces, pendants, bracelets, ties, wigs, crowns, hats, and hoods.

11. A method for simulating kinematic joints, the method comprising: Provides a chain simulation interface that includes multiple graphical control elements, each of which is configured to adjust at least one physical parameter; Receive selections from users using the plurality of graphical control elements; The chain model is updated based on the selection received by the lock; as well as During the kinematic motion simulation, the updated chain model is displayed and animated using a physics engine. The chain model comprises multiple rigid body mesh models linked together by spring elements arranged in a tree configuration to form an assembly skeleton. During the kinematic motion simulation, movement of a parent rigid body mesh model generates a force in at least one adjacent spring element, which in turn causes movement of a subordinate rigid body mesh model.

12. The method of claim 11, wherein the plurality of graphic control elements includes a graphic control element configured to adjust physical parameters selected from the group consisting of: stiffness, damping, elasticity, and inertia.

13. The method of claim 11, wherein the chain model is displayed and animated in real time on a display of a computing device as an overlay on video data received from a camera of the computing device.

14. The method of claim 11, wherein the plurality of graphical control elements comprises a plurality of preset graphical control elements, wherein each preset corresponds to a predetermined selection of at least one of the plurality of graphical control elements.

15. The method of claim 11, wherein the plurality of graphical control elements includes a graphical control element configured to indicate a force applied to the chain model, and at least one graphical control element for indicating whether the force is applied in local space, world space, or relative to an object.

16. The method of claim 11, wherein the plurality of graphical control elements includes graphical control elements for indicating the positioning of the rigid mesh models among the plurality of rigid mesh models relative to the anchor object in which they are positioned.

17. The method of claim 16, wherein the position of the anchor object is determined by applying a trained machine learning model to video data received from a camera.

18. The method of claim 16, wherein the anchor object is a human body part selected from the group consisting of: the user's hands, face, eyes, ears, nose, mouth, torso, arms, legs, feet, and hips.

19. The method of claim 16, wherein the chain model is an item selected from the group consisting of: earrings, necklaces, pendants, bracelets, ties, wigs, crowns, hats, and hoods.

20. A mobile device for simulating kinematic joints, the mobile device comprising: monitor; camera; as well as A processor coupled to a storage system of storage instructions, which, when executed by the processor, cause the processor to: The display presents a chain simulation interface, which includes a plurality of graphical control elements, each configured to adjust at least one physical parameter selected from the group consisting of: stiffness, damping, elasticity, and inertia. Receive selections from users using the plurality of graphical control elements; Update the chain model based on the received selection; as well as On the display, the updated chain model is displayed and animated in real time as an overlay on video data received from the camera, wherein the updated chain model is animated using a physics engine during kinematic motion simulation, and wherein the chain model comprises multiple rigid body mesh models linked together by spring elements arranged in a tree configuration to form an assembly skeleton, wherein during the kinematic motion simulation, movement of a parent rigid body mesh model generates a force in at least one adjacent spring element, which in turn causes movement of a subordinate rigid body mesh model.

Images