Electronic device for training model using sketch data and operating method thereof

Abstract

Disclosed are an electronic device for training a model by using sketch data, and an operating method thereof. The disclosed electronic device comprises a processor and a memory for storing instructions, wherein the instructions, when executed by the processor, cause the electronic device to: acquire sketch data about links of a three-dimensional virtual object located in a virtual space and joints connecting two of the links; acquire physical characteristics of the links and the joints; and train, by using the sketch data and the physical characteristics, a model to determine control for the joints so that the virtual object performs a walking motion.

Description

Electronic device for training a model using sketch data and its operating method

[0001] The disclosure below relates to an electronic device and a method of operating the same for training a model using sketch data.

[0002] Many products feature moving parts, and these moving parts can transform the product into various poses. Products with these multiple poses exhibit distinct shapes for each pose. Because shape changes with movement, designers must consider both shape and motion when designing a product. However, it's difficult to consider both shape and motion simultaneously during the initial sketching phase of the design process, when exploring ideas. Furthermore, the process of designing a product and training a model to perform the desired movements can be resource-intensive and time-consuming.

[0003] The background technology described above is possessed or acquired during the process of deriving the present disclosure, and cannot necessarily be said to be a publicly known technology disclosed to the general public prior to the filing of the present disclosure.

[0004] Various embodiments can utilize sketch data for a three-dimensional virtual object to train a model to determine controls for the virtual object to perform a walking motion.

[0005] Various embodiments can acquire physical properties of links and joints of sketch data and train a model to determine controls for a virtual object to perform a walking motion based on the physical properties.

[0006] Various embodiments can determine pairs of links and joints based on the connection relationship between the links and joints of a virtual object, and determine control for the virtual object to perform a walking motion based on the current state of the determined pair and a learned model.

[0007] Other objects and advantages of the present invention can be understood through the following description and will be more clearly understood through the embodiments of the present invention. Furthermore, it will be readily apparent that the objects and advantages of the present invention can be realized by the means and combinations thereof set forth in the claims.

[0008] An electronic device according to one embodiment includes a processor and a memory storing instructions, wherein the instructions, when executed by the processor, cause the electronic device to obtain sketch data regarding links of a three-dimensional virtual object located in a virtual space and joints connecting two of the links, obtain physical characteristics of the links and the joints, and train a model to determine control of the joints for the virtual object to perform a walking motion using the sketch data and the physical characteristics.

[0009] The above instructions, when executed by the processor, may cause the electronic device to obtain a mass of each of the links, and, based on the mass of the link and the sketch data, determine a moment of inertia of the link about each axis of the three-dimensional plane of the virtual object.

[0010] The above instructions, when executed by the processor, may cause the electronic device to determine, for each of the links, a plurality of points on the sketch data included in the link, and to determine a centroid of the link based on the plurality of points.

[0011] The instructions, when executed by the processor, may cause the electronic device to determine one or more segments in the link to which the plurality of points are connected, determine a mass of each of the one or more segments based on a mass of the link and an overall length of the link, determine moments of inertia of each of the one or more segments about each axis of the link based on the mass of each of the one or more segments, and determine a moment of inertia of the link about each axis by adding up the moments of inertia of each of the one or more segments about each axis.

[0012] The above instructions, when executed by the processor, may cause the electronic device to train the model to determine a connection relationship between the links and joints based on the sketch data, determine pairs corresponding to each of the links and the joints connected to the respective links based on the connection relationship, and determine actions of the pairs based on the states of the pairs.

[0013] The above model may be a graph-based transformer model including a plurality of encoders and a plurality of decoders corresponding to each of the above pairs.

[0014] The instructions, when executed by the processor, may cause the electronic device to train the model by determining a reward for an action of the links and the joints based on at least one of the number of the links and the number of the joints.

[0015] The above instructions, when executed by the processor, may cause the electronic device to obtain the sketch data by drawing a first line on an image plane on which the virtual object is projected, and drawing a second line corresponding to the first line on the virtual object projected on the image plane, based on a pen input input from a user to a touch screen of the tablet.

[0016] According to one embodiment, an electronic device includes a processor and a memory storing instructions, wherein the instructions, when executed by the processor, cause the electronic device to obtain sketch data regarding links of a three-dimensional virtual object located in a virtual space and joints connecting two of the links, obtain current states of the links and the joints and a command for the virtual object, and determine control of the joints for the virtual object to perform a walking motion based on the current state and the command through a model learned using the sketch data and physical characteristics of the links and the joints.

[0017] The model can be trained by obtaining the mass of each of the links and, based on the mass of the link and the sketch data, determining the moment of inertia of the link for each axis on the three-dimensional surface of the virtual object.

[0018] The above model can be trained by determining, for each of the above links, a plurality of points on the sketch data included in the link, and determining the center point of the link based on the plurality of points.

[0019] The model can be trained by determining one or more segments to which the plurality of points are connected in the link, determining a mass of each of the one or more segments based on a mass of the link and a total length of the link, determining moments of inertia of each of the one or more segments for each axis of the link based on the mass of each of the one or more segments, and determining a moment of inertia of the link for each axis by adding up the moments of inertia of each of the one or more segments for each axis.

[0020] A method of operating an electronic device according to one embodiment may include an operation of obtaining sketch data regarding links of a three-dimensional virtual object located in a virtual space and joints connecting two of the links, an operation of obtaining physical characteristics of the links and the joints, and an operation of training a model to determine control of the joints for the virtual object to perform a walking motion using the sketch data and the physical characteristics.

[0021] The operation of acquiring the above physical characteristics can acquire the mass of each of the links, and based on the mass of the link and the sketch data, determine the moment of inertia of the link for each axis on the three-dimensional surface of the virtual object.

[0022] The operation of acquiring the above physical characteristics may include, for each of the links, determining a plurality of points on the sketch data included in the link, and, based on the plurality of points, determining the center point of the link.

[0023] The operation of acquiring the above physical characteristics may include determining one or more segments to which the plurality of points are connected in the link, determining a mass of each of the one or more segments based on a mass of the link and a total length of the link, determining moments of inertia of each of the one or more segments for each axis of the link based on the mass of each of the one or more segments, and adding up the moments of inertia of each of the one or more segments for each axis to determine a moment of inertia of the link for each axis.

[0024] The operation of training the above model may train the model to determine the connection relationship between the links and joints based on the sketch data, determine pairs corresponding to each of the links and the joints connected to the respective links based on the connection relationship, and determine the actions of the pairs based on the states of the pairs.

[0025] The above model may be a graph-based transformer model including a plurality of encoders and a plurality of decoders corresponding to each of the above pairs.

[0026] The operation of training the above model can train the model by determining a reward for the actions of the links and the joints based on at least one of the number of the links and the number of the joints.

[0027] Various embodiments can utilize sketch data for a three-dimensional virtual object to easily and quickly learn a model for the virtual object to perform a walking motion.

[0028] Various embodiments can efficiently train a model and design an object that performs walking motions without actually designing and manufacturing the object, such as a robot, by utilizing sketch data sketched by the user in a virtual space.

[0029] FIG. 1 is a drawing for explaining an electronic device according to one embodiment.

[0030] FIG. 2 is a drawing for explaining operations provided by an electronic device according to one embodiment.

[0031] FIG. 3 is a drawing for explaining an operation of obtaining sketch data according to one embodiment.

[0032] FIG. 4 is a diagram for explaining an operation of acquiring physical characteristics of links and joints according to one embodiment.

[0033] Figure 5 is a diagram for explaining the learning and inference process of a model according to one embodiment.

[0034] FIG. 6 is a drawing for explaining a pair of links and joints according to one embodiment.

[0035] FIG. 7 and FIG. 8 are drawings for explaining the connection relationship between links and joints according to one embodiment.

[0036] FIG. 9 is a drawing for explaining an operation in which a pair of links and joints is input into a model according to one embodiment.

[0037] Figures 10 to 12 are drawings for explaining a transformer model according to one embodiment.

[0038] FIG. 13 is a drawing for explaining an operation of determining control of joints using a model according to one embodiment.

[0039] FIG. 14 is a diagram illustrating an operation method of an electronic device in a learning process according to one embodiment.

[0040] FIG. 15 is a diagram illustrating an operation method of an electronic device in an inference process according to one embodiment.

[0041] Specific structural or functional descriptions of the embodiments are disclosed for illustrative purposes only and may be modified and implemented in various forms. Therefore, the actual implementation is not limited to the specific embodiments disclosed, and the scope of this specification includes modifications, equivalents, or alternatives within the technical concepts described in the embodiments.

[0042] In this document, phrases such as "A or B", "at least one of A and B", "at least one of A or B", "A, B, or C", "at least one of A, B, or C", and "a combination of one or more of A, B, and C" can each include any one of the items listed together in that phrase, or all possible combinations thereof. Although terms such as first or second may be used to describe various components, such terms should be construed only for the purpose of distinguishing one component from another. For example, a first component may be referred to as a second component, and similarly, a second component may also be referred to as a first component.

[0043] When it is said that a component is "connected" to another component, it should be understood that it may be directly connected or connected to that other component, but there may also be other components in between.

[0044] Singular expressions include plural expressions unless the context clearly dictates otherwise. In this specification, the terms "comprises" or "has" should be understood to indicate the presence of a described feature, number, step, operation, component, part, or combination thereof, but not to exclude the possibility of the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0045] Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by a person of ordinary skill in the art. Terms defined in commonly used dictionaries should be interpreted to have a meaning consistent with their meaning in the context of the relevant technology, and will not be interpreted in an idealized or overly formal sense unless explicitly defined herein.

[0046] Hereinafter, embodiments will be described in detail with reference to the attached drawings. In the description with reference to the attached drawings, identical components are assigned the same reference numerals regardless of the drawing numbers, and redundant descriptions thereof will be omitted.

[0047]

[0048] FIG. 1 is a drawing for explaining an electronic device according to one embodiment.

[0049] Referring to FIG. 1, an electronic device (100) may include a processor (110) and a memory (120).

[0050] The electronic device (100) can train a model (115) for a three-dimensional virtual object to perform a predetermined motion (e.g., a walking motion) and determine control for the virtual object to perform the predetermined motion based on the current state of the virtual object using the trained model (115). The electronic device (100) may include, but is not limited to, various computing devices such as a mobile phone, a smart phone, a tablet, an e-book device, a laptop, a personal computer, a desktop, a workstation, or a server, various wearable devices such as a smart watch, smart glasses, a head-mounted display (HMD), or smart clothing, various home appliances such as a smart speaker, a smart TV, or a smart refrigerator, a smart car, a smart kiosk, an Internet of Things (IoT) device, a walking assist device (WAD), a drone, or a robot.

[0051] The electronic device (100) can create a virtual space in which a user can sketch and control virtual objects. The virtual space can be implemented as, for example, augmented reality (AR), virtual reality (VR), mixed reality (MR), or extended reality (XR), but the embodiment is not limited thereto.

[0052] The electronic device (100) can be connected to separately provided devices to transmit and receive data. For example, the electronic device (100) can be connected to a tablet, an HMD, and a controller to transmit and receive data. The tablet includes a touchscreen capable of receiving a user's touch input, pen input, and control input, and can transmit user inputs via the touchscreen to the electronic device. The HMD can display a virtual space created by the electronic device (100) to the user. The controller can receive user inputs for controlling the virtual space or virtual objects and transmit them to the electronic device. Depending on the embodiment, the HMD and the tablet can communicate directly or through the electronic device (100). In another embodiment, the operations of the electronic device described below can be performed by the HMD and / or the tablet, in which case the electronic device (100) can be omitted.

[0053] The processor (110) can process data or perform given operations and / or tasks, and may include various processors such as a central processing unit (CPU), a graphics processing unit (GPU), a neural processing unit (NPU), a tensor processing unit (TPU), and a digital signal processor (DSP). In addition, the processor (110) can execute commands or programs, or control the electronic device (100).

[0054] The processor (110) may include a model (115). For example, the processor (110) may execute a model (115) implemented in software form to perform a given operation and / or task. The model (115) may be an artificial intelligence (AI) model, for example, a transformer model, but the embodiment is not limited thereto. The model (115) may include a plurality of encoders and a plurality of decoders. According to one embodiment, the model (115) may learn a control policy that can be generally applied without limitation on the shape of a virtual object by the processor (110). For example, the model (115) may cause a virtual object to perform a walking motion according to a pre-learned control policy, regardless of the shape and morphology of the virtual object, such as a biped robot, a tripped robot, or a quadruped robot.

[0055] In FIG. 1, for the sake of explanation, the model (115) is illustrated as being implemented by the processor (110). However, depending on the embodiment, the model (115) may be implemented by separate hardware. For example, the model (115) may be implemented by a dedicated processor included in an electronic device (100) different from the processor (110) or a processor provided separately from the electronic device (100). For example, the model (115) may be implemented by a hardware accelerator for computation.

[0056] The memory (120) can store data processed by the processor (110). In addition, the memory (120) can store instructions (e.g., a program) executable by the processor (110). For example, the instructions may include instructions for executing operations of the processor (110) and / or operations of each component of the processor (110).

[0057] According to one embodiment, the processor (110) can obtain sketch data regarding links of a three-dimensional virtual object and joints connecting two of the links. In addition, the processor (110) can obtain physical characteristics of the links and joints of the virtual object. For example, the processor can obtain sketch data and physical characteristics from a user through a touchscreen of a tablet. The operation of obtaining the sketch data and physical characteristics is described in detail with reference to FIGS. 3 and 4 below. The processor (110) can train a model (115) using the sketch data and physical characteristics to determine control of joints for the virtual object to perform a walking motion. The operation of training the model (115) is described in detail with reference to FIGS. 6 to 12 below.

[0058] According to one embodiment, the processor (110) can obtain the current state of links and joints of a virtual object and commands for the virtual object. The processor (110) can determine control of joints for the virtual object to perform a walking motion based on the current state and commands through a learned model (115). The virtual object can perform a walking motion within a virtual space according to the control determined by the model (115). The operation of determining control using the model (115) is described in detail below with reference to FIG. 13.

[0059] For example, in response to a user sketching a three-dimensional virtual object in a virtual space and inputting physical characteristics of the virtual object, the electronic device (100) can train a model (115) so that the virtual object can perform a predetermined motion. In response to the user inputting control for the virtual object through the controller, the electronic device (100) can control the virtual object to perform a predetermined motion according to the user's control using the trained model (115). The user can easily and quickly train a model (115) for the virtual object to perform a walking motion by utilizing sketch data in the virtual space without actually designing or manufacturing the object. In the present specification, the predetermined motion for the virtual object has been described as a walking motion for the purpose of explanation, but the embodiment is not limited thereto and may be determined differently depending on the purpose of designing the virtual object or the use of the virtual object. For example, if the virtual object is a walking robot, the predetermined motion may be a walking motion, but if the virtual object is a robot arm, the predetermined motion may be a carrying motion.

[0060]

[0061] FIG. 2 is a drawing for explaining operations provided by an electronic device according to one embodiment.

[0062] Referring to FIG. 2, in operation (210), the electronic device may draw a first line on an image plane according to a user's pen input, and may draw a second line corresponding to the first line on a virtual object projected on the image plane. The image plane may represent a plane created in a virtual space corresponding to the touchscreen of the tablet. The image plane may be a virtual transparent plane. The image plane may be synchronized with the touchscreen so that an object drawn on the touchscreen is displayed. In one embodiment, the first line may represent a line drawn on an image of a virtual object projected on the image plane according to the user's sketch, and the second line may represent a line drawn on a three-dimensional virtual object according to the user's sketch. The virtual object may include a plurality of parts (e.g., a torso, legs), and the plurality of parts may be drawn by a plurality of second lines. The sketch data may represent the second lines representing the virtual object. In the present specification, for convenience of description, operation (210) may also be referred to as a sketching operation.

[0063] In operation (220), the electronic device can provide a user with a three-dimensional virtual object generated in a virtual space. The electronic device can perform at least one of a segmenting operation, a rigging operation, a posing operation, and a filming operation based on the user's control input. The electronic device can distinguish parts of the virtual object based on the user's input, and can receive input of the connection relationship and physical characteristics of each part. For example, the electronic device can distinguish parts of the virtual object into links and joints, and can receive input of the mass of each part. In the present specification, for convenience of explanation, operations (210) and (220) may also be referred to as design operations.

[0064] In operation (230), the electronic device may train a model based on sketch data for the virtual object, the connection relationship between each part, and physical characteristics so that the virtual object can perform a preset motion (e.g., a walking motion). For example, the electronic device may reinforcement train the model to determine the next action of each link and joint based on the current state of each link and joint of the virtual object. In one embodiment, the electronic device may determine the moment of inertia of each part based on the mass of each part.

[0065] In operation (240), the electronic device can control the movement of the virtual object based on the learned model and the user's input. For example, the electronic device can control the virtual object to walk based on the user's controller input.

[0066]

[0067] FIG. 3 is a drawing for explaining an operation of obtaining sketch data according to one embodiment.

[0068] Referring to FIG. 3, the electronic device may draw a first line (325) on an image plane (320) on which a virtual object is projected based on a pen input input from a user to a touch screen of a tablet, and may draw a second line (335) corresponding to the first line (325) on the virtual object projected on the image plane (320). According to one embodiment, the electronic device may determine a reference plane (330) for the virtual object based on a point specified through the pen input, and may draw the second line (335) based on the reference plane (330) and the pen input. The reference plane (330) is a plane arranged in a three-dimensional virtual space on which a sketch line is drawn according to the user's input, and may assist in drawing a sketch line in the three-dimensional virtual space according to the user's intention. Sketch data for the virtual object may be a set of second lines (335) drawn in the virtual space.

[0069] For example, the electronic device may draw a second line (335) corresponding to the first line (325) on the reference plane (330) based on the camera (310) corresponding to the user's viewpoint. For example, the electronic device may determine the points where the line segments from the camera (310) to the first line (325) intersect the reference plane (330) as the second line (335).

[0070] According to one embodiment, the electronic device can draw the first line (325) and the second line (335) based on the angle between the pen and the touchscreen or the pressure applied to the touchscreen. For example, the electronic device can draw the second line (335) thicker as the pressure applied to the touchscreen by the pen increases.

[0071]

[0072] FIG. 4 is a diagram for explaining an operation of acquiring physical characteristics of links and joints according to one embodiment.

[0073] Referring to FIG. 4, a virtual object (410) in the sketch data may include multiple parts (e.g., a torso, legs). Each of the multiple parts may be distinguished by links and joints. A link represents a non-deformable part of the virtual object (e.g., a rigid body), and a joint may be a part (e.g., a joint) that connects multiple links and enables relative movement of the links.

[0074] Each part of the virtual object (410) may have a parent-child part connection relationship established. For example, for a link that performs repetitive movements based on a joint included in the virtual object (410), a parent-child part connection relationship may be established between the joint and the link. In this case, the joint may be referred to as a parent joint, and the link may be referred to as a child link.

[0075] Each of the plurality of parts may have physical characteristics (e.g., mass, inertia). The electronic device may receive input of the physical characteristics of each of the plurality of parts from the user. For example, the electronic device may output a box (425) in which the user can input the physical characteristics of the corresponding part (420) in response to the user touching a part (420) of the virtual object (410) through a tablet connected to the electronic device. The electronic device may obtain the mass of the corresponding part (420) based on the mass value entered in the box (425) for the corresponding part (420).

[0076] In one embodiment, the electronic device can determine other physical characteristics of the part (420) from the input physical characteristics. In one embodiment, the electronic device can obtain the mass of each link and, based on the mass of the link and the sketch data, determine the moment of inertia of the link for each axis on the three-dimensional plane of the virtual object (410).

[0077] Each part of the virtual object (410) may include one or more polylines. The polylines may, for example, correspond to a second line drawn by the user on the virtual object (410). For example, the electronic device may determine the polylines included in each part as shown in Mathematical Expression 1 below.

[0078] [Mathematical Formula 1]

[0079]

[0080] Here, Part is one part of the virtual object (410), L1 to L m can represent polylines included in that part.

[0081] A polyline may contain one or more points in a three-dimensional virtual space. For example, an electronic device may determine the points contained in each polyline as shown in Equation 2 below.

[0082] [Equation 2]

[0083]

[0084] Here, p i,1 Inland p i,n Silver polyline L i It can represent the points included in .

[0085] The segments included in each part can represent lines connecting connected points among the points included in the polylines. For example, the segment s of each part i,j can be expressed as in mathematical formula 3 below.

[0086] [Equation 3]

[0087]

[0088] The electronic device determines the moment of inertia of a part (420) of a virtual object (410) relative to the center point (centroid) of the part (420) and the axis of the part (420). Six moments of inertia components for It can be expressed as an inertia tensor having . The axes of each part can be set as follows.

[0089] 1. X-axis: The direction of the rotation axis of the parent joint of the corresponding part (420) or the upward direction in the virtual space if the corresponding part (420) is not connected to a joint.

[0090] 2. Y-axis: Direction from the center point of the part (420) to the position of the parent joint (orthogonal to the X-axis)

[0091] 3. Z-axis: Direction of the cross product of the X-axis and Y-axis

[0092] For example, the electronic device can determine the axes of each part based on the rotation axis J of the parent joint of each part and the line L connecting the center point of each part and the parent joint position, as shown in the following mathematical expression 4.

[0093] [Equation 4]

[0094]

[0095] The electronic device can determine the inertia moment components of each part as the sum of the inertia moments of the segments included in the part, as shown in the mathematical expression 5 below.

[0096] [Equation 5]

[0097]

[0098] According to one embodiment, the electronic device may determine, for each link, a plurality of points on sketch data included in the link, and, based on the plurality of points, determine the center point of the link. The electronic device may determine the center point of each part by dividing the sum of the position coordinates of the points included in each part by the number of points included in the part. For example, the electronic device may determine the center point of each part as shown in Mathematical Expression 6 below.

[0099] [Equation 6]

[0100]

[0101] The electronic device can determine the moment of inertia of each part about each axis based on the mass of each part and the length of the part. For example, the electronic device can determine the mass of each of one or more segments based on the mass of the link and the overall length of the link, determine the moments of inertia of each of one or more segments about each axis of the link based on the mass of each of the one or more segments, and determine the moment of inertia of the link about each axis by adding the moments of inertia of each of the one or more segments about each axis.

[0102] For example, an electronic device has a mass M of each part as shown in the mathematical expression 7 below. part Based on the length of each segment relative to the total length of the part, the mass of each segment can decide.

[0103] [Equation 7]

[0104]

[0105] The electronic device can determine the moment of inertia components of each segment based on the mass of each segment, as shown in mathematical expression 8 below.

[0106] [Equation 8]

[0107]

[0108] Here, is the center point of the part and segment s i,j can represent the vector connecting them.

[0109] The electronic device can determine the moment of inertia of the corresponding part (420) using the moment of inertia components of each segment determined through mathematical equation 8 and mathematical equation 5. The electronic device can display the determined moment of inertia of the corresponding part (420) in a box (425) output for the corresponding part (420).

[0110]

[0111] Figure 5 is a diagram for explaining the learning and inference process of a model according to one embodiment.

[0112] Referring to FIG. 5, the electronic device trains a transformer model (523) to determine control of joints for a virtual object to perform a walking motion, and can use the trained transformer model (523) to determine control of joints for the virtual object to perform a walking motion based on the current state and commands of links and joints. FIG. 5 exemplarily illustrates the overall architecture of the transformer model (523). In the present specification, for the sake of explanation, the transformer model (523) is described as a graph-based transformer model, but the embodiment is not limited thereto.

[0113] According to one embodiment, the overall architecture for the transformer model (523) may include a gated recurrent unit (GRU) part (510), a morphology-aware part (520), and a decoder part (530). The gated recurrent unit (GRU) part (510), the morphology-aware part (520), and the decoder part (530) may be implemented by one or more components (e.g., a processor) of an electronic device. In this specification, for convenience of description, the GRU part (510) may also be referred to as a GRU network, and the morphology-aware part (520) may also be referred to as a morphology-aware transformer network. In this specification, a model that determines the control of joints for a virtual object to perform a walking motion may include not only the transformer model (523), but also the morphology-aware part (520) and the decoder part (530).

[0114] The GRU part (510) can encapsulate information (511) regarding the current state of a virtual object. Using the GRU (512), the electronic device can efficiently manage not only the current state of the virtual object but also its previous state. In this specification, for convenience of explanation, the information (511) regarding the current state of the virtual object may also be referred to as robot observation information.

[0115] The morphology recognition part (520) can process information (521) about the shape of a virtual object. The information (521) about the shape of the virtual object can include link observation information about links and joint observation information about joints. The morphology recognition part (520) can include one or more encoders (522) and a transformer model (523). The morphology recognition part (520) can process a pair of links and joints as a single token. Information about the shape of the virtual object can be integrated based on a graph structure. For example, absolute positional information can be applied to an input token, and relative positional information can be utilized in the self-attention process of the transformer model (523). Through this, the morphology recognition part (520) can learn the shape of the virtual object by utilizing both the input token and the positional information.

[0116] The input of the transformer model (523) may include link observation information and joint observation information. The electronic device may process the virtual object by generating a one-dimensional sequence of tokens corresponding to a depth-first traversal of a kinematic tree. The link observation information and the joint observation information may be linked as a single pair. The number of joints and the number of links may be set to preset hyperparameters (e.g., max-length) to accommodate various virtual object shapes. Here, joints and links not used in each URDF (Unified Robot Description Format) may be determined as a first value (e.g., "0"). Each link may be linked to its parent joint. For example, the first piece of link observation information may be for the torso, and the corresponding joint observation information may be determined as "0". The electronic device may use the link observation information and the joint observation information to perform absolute position embedding for the shape of the virtual object.

[0117] For example, link observation information and joint observation information may include the components in Table 1 below.

[0118] Information component dimension robot observation information basic rotation 3 angular velocity 3 joint position max-length joint velocity max-length previous action target max-length body height 1 linear velocity 3 height scan 3 6 joint observation information relative distance 3 joint position 1 previous joint position 1 previous joint velocity 1 previous joint velocity 1 previous action target 1 previous action target 1 Pgain 1 Dgain 1 link observation information link mass 1 link inertia 9 relative distance 3

[0119]

[0120] Here, the values ​​shown in Table 1 are also included in the URDF of the virtual object and can be dynamically changed as the state of the virtual object changes.

[0121] The decoder part (530) may include one or more decoders (531). The one or more decoders (531) may determine the next movements (532) of the joints based on the output of the GRU (512), the output of the transformer model (523), and the command information (513).

[0122] Below, each part (510, 520, 530) of the architecture is described in detail through FIGS. 6 to 13.

[0123]

[0124] FIG. 6 is a drawing for explaining a pair of links and joints according to one embodiment.

[0125] Referring to FIG. 6, information (610) about the shape of a virtual object may include link observation information (611) about a link and joint observation information (612) about a joint. The electronic device may determine a link and a parent joint connected to the link as a single pair. The link observation information (611) and joint observation information (612) connected as a single pair may be determined as a single token for model learning. In this specification, for convenience of explanation, information (610) about the shape of a virtual object may also be referred to as link and joint pair observation information.

[0126]

[0127] FIG. 7 and FIG. 8 are drawings for explaining the connection relationship between links and joints according to one embodiment.

[0128] The virtual objects illustrated in FIGS. 7 and 8 are illustrative and the present invention is not limited thereto. For example, an electronic device can determine a graph for model learning based on the connection relationships between links and joints for a virtual object of any shape.

[0129] Referring to FIG. 7, the connection relationship between links and joints in the case where the virtual object (700) is a quadruped walking robot is exemplarily illustrated.

[0130] In the example of FIG. 7, each part of the virtual object (700) can be divided into links (710, 720, 730, 740) and joints (725, 735, 745). In the graph, the links (710, 720, 730, 740) can be represented as nodes, and the joints (725, 735, 745) connecting two of the links can be represented as edges. The electronic device can learn information about the shape of the virtual object (700) through the links (710, 720, 730, 740) and joints (725, 735, 745) represented in the graph.

[0131] An electronic device can perform relative position embedding on the shape of a virtual object (700) based on a graph. The electronic device can encode the shape of the virtual object (700) using a graph representation based on the connection relationship between links and joints. In one embodiment, the electronic device can determine that a pair of tokens representing a link of the virtual object (700) and a parent joint of the link are valid if they are directly connected, and determine that they are invalid if they are not directly connected.

[0132] The electronic device can effectively learn the shape and state of the virtual object (700) based on link observation information and joint observation information by utilizing both absolute position embedding and relative position embedding for the shape of the virtual object (700).

[0133] Referring to FIG. 8, the connection relationship between links and joints in the case where the virtual object (800) is a bipedal walking robot is exemplarily illustrated. In the example of FIG. 8, each part of the virtual object (800) can be divided into links (810, 820, 830, 840, 850, 860) and joints (825, 835, 845, 855, 865).

[0134]

[0135] FIG. 9 is a drawing for explaining an operation in which a pair of links and joints is input into a model according to one embodiment.

[0136] Referring to Fig. 9, a situation in which pairs of links and joints of the virtual object of Fig. 7 are input to the model is exemplarily illustrated.

[0137] The pairs (921) of links and joints determined for the virtual object can be input into the model as input tokens of the model. The number of pairs (910) input into the model can be predetermined. The pairs (921) of links and joints determined for the virtual object can be assigned to some of the pairs (910) input into the model, and the remaining pairs (922) that are not assigned can be determined as predetermined values ​​(e.g., "0").

[0138] According to the connection relationship between the links and joints of the virtual object, pairs (921) of each link and joint can be sequentially assigned to pairs (910) input to the model. In the example of Fig. 9, pair (931) can represent a link representing the torso of the virtual object (e.g., "Torso"). Since there is no parent joint of the link representing the torso of the virtual object, the joint observation information of pair (931) can be determined as a preset value (e.g., "0"). Pair (941) can represent a connection relationship between a first link of the left front leg of the virtual object (e.g., "Hip Link") and a first parent joint (e.g., "HAA Joint"). Pair (942) can represent a connection relationship between a second link of the left front leg of the virtual object (e.g., "Thigh Link") and a second parent joint (e.g., "HFE Joint"). A pair (943) may represent a connection relationship between a third link (e.g., "Shank Link") of the left front leg of a virtual object and a third parent joint (e.g., "KFE Joint"). A pair (951) may represent a connection relationship between a first link of the right front leg of a virtual object and a first parent joint. However, the order and manner in which pairs (921) of links and joints are assigned to pairs (910) input to the model may be determined differently depending on the embodiment.

[0139]

[0140] Figures 10 to 12 are drawings for explaining a transformer model according to one embodiment.

[0141] Referring to FIG. 10, the structure of a transformer model (1020) that determines an output (1030) for determining the behavior of a link and a joint based on an input (1010) for a pair of links and joints is exemplarily illustrated. The transformer model (1020) can perform a multi-head attention operation and a feed forward operation. In one embodiment, the transformer model (1020) can repeatedly perform the multi-head attention operation and the feed forward operation.

[0142] The Transformer model (1020) can learn a policy that effectively generates joint actions and tracks commands based on rewards for joint actions, regardless of the shape of the virtual object. If the reward is designed to favor a specific virtual object shape, the policy may converge to a local minimum that achieves high rewards only for that specific virtual object shape. Furthermore, if there is a significant difference in the reward magnitudes between virtual objects during training, instability may occur, leading to the failure of the training process. To ensure stable training, normalizing the rewards across different virtual object shapes may be required.

[0143] Rewards related to joint dynamics, such as torque, position, velocity, acceleration, and smoothness of a joint, may be affected by the number of joints in a virtual object. In one embodiment, the electronic device may train the transformer model (1020) by determining rewards for the behavior of links and joints based on at least one of the number of links and the number of joints. For example, the electronic device may determine the reward for learning by dividing the reward related to joint dynamics by the number of joints in the virtual object before training. Additionally, the electronic device may determine the reward for learning by dividing the reward related to airtime or foot clearance, which are affected by the number of legs, by the number of legs in the virtual object. Through this, the electronic device may ensure consistent rewards regardless of the shape of the virtual object and assist in robust policy decisions of the transformer model (1020).

[0144] The dynamics of a virtual object containing joints can be expressed by the equations of motion in the following mathematical expressions.

[0145] [Equation 9]

[0146]

[0147] Here, M(q) is the mass matrix, Coriolis and centrifugal forces, is the gravity vector, is the control torque, can represent joint positions, velocities, and accelerations, respectively. In the case of PD (proportional-derivative) control, the control torque can be determined as shown in mathematical formula 10 below.

[0148] [Equation 10]

[0149]

[0150] Here, q d is the desired joint location, is the desired joint velocity, K p is the proportional gain (P-gain), K d can represent the differential gain (D-gain).

[0151] The desired joint positions are provided as outputs of the model, and the desired joint velocities can be set to "0". Control torque To determine the gain, both proportional and differential gains may be required. These gains can be heuristically determined for each virtual object. The electronic device has a unique frequency that controls the response speed. and damping ratio that controls the amount of vibration By using the proportional gain and differential gain, the electronic device can determine the gains systematically, thereby enabling better generalization across various virtual objects. For example, the electronic device can determine the proportional gain (K) for the virtual object as shown in Equations 11 and 12 below. p ) and differential gain (K d ) can be determined.

[0152] [Equation 11]

[0153]

[0154] [Equation 12]

[0155]

[0156] Here, the mass matrix can be used in the nominal position state. To determine the proportional gain, the electronic device can determine M as the sum of the diagonal terms corresponding to the body cross-section of the mass matrix. To determine the differential gain, the electronic device can determine M as the effective mass, as the diagonal terms of the mass matrix corresponding to a specific joint. Through this, the electronic device can select the smallest value from the acquired vector and uniformly apply it to all activated joints to determine the differential gain. Damping ratio can be set as a hyperparameter. Natural frequency can be set to a preset frequency (e.g., 1 rad / s) to enable the virtual object to perform stable walking motions. This allows the electronic device to ensure a consistent and efficient process for determining gain values ​​regardless of the various shapes of the virtual object.

[0157] Referring to FIG. 11, the electronic device can train a model based on absolute position embeddings (1120) and relative position embeddings (1130) determined from vectorized environments (1110) for a virtual object.

[0158] The transformer architecture can facilitate information exchange between tokens. This information exchange can be achieved through self-attention. Each token can determine the contextual significance of each joint and link within a virtual object by considering all other tokens in the sequence. In this process, the vectors of key, query, and value can exchange information through the attention module using Equation 13 below.

[0159] [Equation 13]

[0160]

[0161] Here, Q is the query, K is the key, V is the value, d k can represent the dimension of a token. r can introduce morphology-aware mask padding. The transformer model can better interpret the characteristics of virtual objects by understanding the context in which each token is used.

[0162] When training a model using multiple virtual objects, the number of environments that can be trained simultaneously may be limited due to limited computational resources. In one embodiment, the electronic device can determine the difference between the reward of each virtual object and the overall average reward. The electronic device can apply a softmax function to the data based on this difference to determine the generation probability of each virtual object. The generation probability is used to determine the likelihood of generating a specific virtual object during model training, thereby increasing the generation frequency of virtual objects with relatively low rewards. For example, the softmax function can be determined as shown in Equation 14 below.

[0163] [Equation 14]

[0164]

[0165] Here, P i is the probability of creating a virtual object i, r i is the reward for virtual object i, can represent the overall average reward. The electronic device can balance the learning process across all virtual objects, ensuring appropriate learning for both low-performing and high-performing virtual objects.

[0166] Referring to FIG. 12, examples of absolute position embedding (1120) and relative position embedding (1130) are illustrated. The examples of FIG. 12 may be absolute position embedding (1120) and relative position embedding (1130) determined from the virtual object of FIG. 7.

[0167] In one embodiment, the electronic device can determine the relative position embedding (1130) based on the connection relationship between the links and joints of the virtual object. For example, the electronic device can determine the relative position embedding (1130) by determining that a pair of tokens representing a link of the virtual object and a parent joint of the link are valid if they are directly connected, and invalid if they are not directly connected. For example, the electronic device can determine each value of the relative position embedding (1130) through the following mathematical expression 15. can decide.

[0168] [Equation 15]

[0169]

[0170] Here, can represent the adjacency matrix between part i and part j.

[0171] FIG. 13 is a drawing for explaining an operation of determining control of joints using a model according to one embodiment.

[0172] Referring to FIG. 13, the electronic device can determine the control of joints for the virtual object to perform a walking motion based on the current state of links and joints and a user's command to the virtual object through a model learned about the shape of the virtual object.

[0173] According to one embodiment, the electronic device can determine inputs (1330) to a plurality of decoders (1340) based on information (1320) combining the output (1310) of the learned transformer model and the current states of links and joints and commands for the virtual object. The plurality of decoders (1340) can determine next actions (1350) of the pairs of links and joints for the virtual object to perform a walking motion based on the inputs (1330) for the pairs of links and joints. The next actions (1350) of the pairs can, for example, represent movements of the corresponding joints.

[0174] The plurality of decoders (1340) may include a multi-layer perceptron (MLP) that determines an action based on the token for each pair. For example, the plurality of decoders (1340) may include a set of independent MLPs based on preset hyperparameters (e.g., max-length), and each MLP may determine an action for the pair. The number of joints capable of performing the generated actions (1350) may vary depending on the difference in the URDF used in each environment. Unused actions among the generated actions (1350) may be skipped.

[0175]

[0176] FIG. 14 is a diagram illustrating an operation method of an electronic device in a learning process according to one embodiment.

[0177] In the following embodiments, the operations may be performed sequentially, but are not necessarily performed sequentially. For example, the order of the operations may be changed, and at least two operations may be performed in parallel. Operations (1410) to (1430) may be performed by at least one component (e.g., a processor, etc.) of the electronic device.

[0178] In operation (1410), the electronic device can obtain sketch data regarding links of a three-dimensional virtual object located in a virtual space and joints connecting two of the links. The electronic device can obtain the sketch data by drawing a first line on an image plane on which the virtual object is projected, and drawing a second line corresponding to the first line on the virtual object projected on the image plane, based on a pen input input from a user on a touch screen of a tablet.

[0179] In operation (1420), the electronic device may acquire physical characteristics of links and joints. The electronic device may acquire the mass of each link, and determine the moment of inertia of the link for each axis of the three-dimensional virtual object based on the mass of the link and the sketch data. For each link, the electronic device may determine a plurality of points on the sketch data included in the link, and determine a center point of the link based on the plurality of points. The electronic device may determine one or more segments to which the plurality of points are connected in the link, and determine the mass of each of the one or more segments based on the mass of the link and the overall length of the link, and determine the moments of inertia of each of the one or more segments for each axis of the link based on the mass of each of the one or more segments, and determine the moments of inertia of the link for each axis by adding the moments of inertia of each of the one or more segments for each axis.

[0180] In operation (1430), the electronic device may train a model to determine control of joints for a virtual object to perform a walking motion using sketch data and physical characteristics. The electronic device may train the model to determine connection relationships between links and joints based on the sketch data, determine pairs corresponding to each link and a joint connected to the corresponding link based on the connection relationships, and determine actions of the pairs based on the states of the pairs. The model may be a graph-based transformer model including a plurality of encoders and a plurality of decoders corresponding to each pair. The electronic device may train the model by determining rewards for actions of the links and joints based on at least one of the number of links and the number of joints.

[0181] Since the matters described above through FIGS. 1 to 13 are applied to each operation illustrated in FIG. 14, a more detailed description is omitted.

[0182]

[0183] FIG. 15 is a diagram illustrating an operation method of an electronic device in an inference process according to one embodiment.

[0184] In the following embodiments, the operations may be performed sequentially, but are not necessarily performed sequentially. For example, the order of the operations may be changed, and at least two operations may be performed in parallel. Operations (1510) to (1530) may be performed by at least one component (e.g., a processor, etc.) of the electronic device.

[0185] In operation (1510), the electronic device can obtain sketch data regarding links of a three-dimensional virtual object located in a virtual space and joints connecting two of the links.

[0186] In operation (1520), the electronic device can obtain the current state of links and joints and commands for virtual objects.

[0187] In operation (1530), the electronic device may determine control of joints for the virtual object to perform a walking motion based on the current state and commands through a learned model using sketch data and physical characteristics of links and joints. The model may be learned by acquiring the mass of each link, and determining the moment of inertia of the link for each axis of the virtual object based on the mass of the link and the sketch data. The model may be learned by determining, for each link, a plurality of points on the sketch data included in the link, and determining the center point of the link based on the plurality of points. The model may be learned by determining one or more segments connected to the plurality of points in the link, determining the mass of each of the one or more segments based on the mass of the link and the overall length of the link, determining the moments of inertia of each of the one or more segments for each axis of the link based on the mass of each of the one or more segments, and determining the moment of inertia of the link for each axis by adding up the moments of inertia of each of the one or more segments for each axis.

[0188] Since the matters described above through FIGS. 1 to 14 are applied to each operation illustrated in FIG. 15, a more detailed description is omitted.

[0189]

[0190] The embodiments described above may be implemented using hardware components, software components, and / or a combination of hardware components and software components. For example, the devices, methods, and components described in the embodiments may be implemented using a general-purpose computer or a special-purpose computer, such as, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing instructions and responding to them. The processing device may execute an operating system (OS) and software applications running on the operating system. Furthermore, the processing device may access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, the processing device is sometimes described as being used alone; however, one of ordinary skill in the art will recognize that the processing device may include multiple processing elements and / or multiple types of processing elements. For example, a processing unit may include multiple processors, or a processor and a controller. Other processing configurations, such as parallel processors, are also possible.

[0191] Software may include a computer program, code, instructions, or a combination of one or more of these, and may configure a processing device to perform a desired operation or, independently or collectively, command the processing device. The software and / or data may be stored on any type of machine, component, physical device, virtual equipment, computer storage medium, or device for interpretation by the processing device or for providing instructions or data to the processing device. The software may also be distributed over networked computer systems and stored or executed in a distributed manner. The software and data may be stored on a computer-readable recording medium.

[0192] The method according to the embodiment may be implemented in the form of program commands that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may store program commands, data files, data structures, etc., alone or in combination, and the program commands recorded on the medium may be those specially designed and configured for the embodiment or may be known and available to those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and hardware devices specially configured to store and execute program commands, such as ROMs, RAMs, and flash memories. Examples of program commands include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter, etc.

[0193] The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

[0194] Although the embodiments described above have been described with limited drawings, those skilled in the art will appreciate that various technical modifications and variations can be applied based on the described embodiments. For example, appropriate results can still be achieved even if the described techniques are performed in a different order than described, and / or components of the described systems, structures, devices, circuits, etc. are combined or combined in a different manner than described, or are replaced or substituted with other components or equivalents.

[0195] Therefore, other implementations, other embodiments, and equivalents to the claims also fall within the scope of the claims described below.

Claims

1. In electronic devices, processor; and Memory that stores instructions Including, The above instructions, when executed by the processor, cause the electronic device to: Obtain sketch data about links of a three-dimensional virtual object located in a virtual space and joints connecting two of the links, Obtaining the physical properties of the above links and joints, Using the sketch data and the physical characteristics, a model is trained to determine the control of the joints for the virtual object to perform a walking motion. Electronic devices.

2. In paragraph 1, The above instructions, when executed by the processor, cause the electronic device to: Obtain the mass of each of the above links, Based on the mass of the link and the sketch data, determine the moment of inertia of the link for each axis of the three-dimensional virtual object. Electronic devices.

3. In paragraph 2, The above instructions, when executed by the processor, cause the electronic device to: For each of the above links, determine a plurality of points on the sketch data included in the link, Based on the above multiple points, the centroid of the link is determined. Electronic devices.

4. In paragraph 3, The above instructions, when executed by the processor, cause the electronic device to: Determine one or more segments connected to the plurality of points in the above link, Based on the mass of the link and the total length of the link, the mass of each of the one or more segments is determined, Based on the mass of each of the one or more segments, determine the moments of inertia of each of the one or more segments about each axis of the link, By adding the moments of inertia of each of the one or more segments for each axis, the moment of inertia of the link for each axis is determined. Electronic devices.

5. In paragraph 1, The above instructions, when executed by the processor, cause the electronic device to: Based on the above sketch data, determine the connection relationship between the links and joints, Determine pairs corresponding to each of the links and the joints connected to the links according to the above connection relationship, Based on the state of the pairs, the model is trained to determine the actions of the pairs. Electronic devices.

6. In paragraph 5, The above model A graph-based transformer model including multiple encoders and multiple decoders corresponding to each of the above pairs, Electronic devices.

7. In paragraph 1, The above instructions, when executed by the processor, cause the electronic device to: By determining a reward for the actions of the links and the joints based on at least one of the number of the links and the number of the joints, the model is trained. Electronic devices.

8. In paragraph 1, The above instructions, when executed by the processor, cause the electronic device to: Based on a pen input input from a user to a touch screen of a tablet, the sketch data is obtained by drawing a first line on an image plane on which the virtual object is projected, and drawing a second line corresponding to the first line on the virtual object projected on the image plane. Electronic devices.

9. In electronic devices, processor; and Memory that stores instructions Including, The above instructions, when executed by the processor, cause the electronic device to: Obtain sketch data regarding links of a three-dimensional virtual object located in a virtual space and joints connecting two of the links, Obtain the current status of the above links and joints and commands for the virtual object, Using the learned model using the above sketch data and the physical characteristics of the links and the joints, the virtual object determines the control of the joints to perform a walking motion based on the current state and the command. Electronic devices.

10. In paragraph 9, The above model Obtain the mass of each of the above links, By determining the moment of inertia of the link for each axis of the virtual object based on the mass of the link and the sketch data, Electronic devices.

11. In paragraph 10, The above model For each of the above links, determine a plurality of points on the sketch data included in the link, Based on the above multiple points, learned by determining the center point of the link, Electronic devices.

12. In paragraph 11, The above model Determine one or more segments connected to the plurality of points in the above link, Based on the mass of the link and the total length of the link, the mass of each of the one or more segments is determined, Based on the mass of each of the one or more segments, determine the moments of inertia of each of the one or more segments about each axis of the link, By summing the moments of inertia of each of the one or more segments for each axis, the moment of inertia of the link for each axis is learned. Electronic devices.

13. In the method of operating an electronic device, An operation of obtaining sketch data regarding links of a three-dimensional virtual object located in a virtual space and joints connecting two of the links; An operation of acquiring physical properties of the above links and joints; and An operation of training a model to determine control of the joints for the virtual object to perform a walking motion using the sketch data and the physical characteristics. Including How an electronic device operates.

14. In paragraph 13, The action of acquiring the above physical characteristics is Obtain the mass of each of the above links, Based on the mass of the link and the sketch data, determining the moment of inertia of the link for each axis of the three-dimensional virtual object. How an electronic device operates.

15. In paragraph 14, The action of acquiring the above physical characteristics is For each of the above links, determine a plurality of points on the sketch data included in the link, Based on the above multiple points, determining the center point of the link, How an electronic device operates.

16. In paragraph 15, The action of acquiring the above physical characteristics is Determine one or more segments connected to the plurality of points in the above link, Based on the mass of the link and the total length of the link, the mass of each of the one or more segments is determined, Based on the mass of each of the one or more segments, determine the moments of inertia of each of the one or more segments about each axis of the link, By adding the moments of inertia of each of the one or more segments for each axis, the moment of inertia of the link for each axis is determined. How an electronic device operates.

17. In paragraph 13, The action of training the above model is Based on the above sketch data, determine the connection relationship between the links and joints, Determine pairs corresponding to each of the links and the joints connected to the links according to the above connection relationship, Based on the state of the pairs, the model is trained to determine the actions of the pairs. How an electronic device operates.

18. In paragraph 17, The above model A graph-based transformer model including multiple encoders and multiple decoders corresponding to each of the above pairs, How an electronic device operates.

19. In paragraph 13, The action of training the above model is By determining a reward for the actions of the links and the joints based on at least one of the number of the links and the number of the joints, the model is trained. How an electronic device operates.

20. A computer-readable recording medium storing a computer program for executing any one of the methods of Articles 13 to 19.

Images