Method and system for generating three-dimensional object based on semantic analysis

Abstract

The invention relates to a method and system for generating a three-dimensional object based on semantic analysis. More specifically, the present invention relates to a method and system for automatically generating a three-dimensional object corresponding to the meaning of a sentence by analyzing the sentence in a text form.

Description

Technical Field

[0001] This invention relates to a method and system for generating three-dimensional objects based on semantic analysis. More specifically, this invention relates to a method and system for automatically generating three-dimensional objects corresponding to the meaning of sentences by analyzing sentences in text form. Background Technology

[0002] As the virtual universe market grows, the demand for its core element—3D object generation technology—is also increasing rapidly. Both domestic and international companies view virtual universes, which recreate the real world within a virtual environment, as the next generation of growth engines. Furthermore, 3D modeling technology—the core technology of virtual universes that converts the real world into three dimensions and presents it in the virtual world—is attracting increasing attention.

[0003] In the past, 3D modeling technology was mainly used in the gaming industry, but in recent years, its application trend has been to expand to all industries, including AR / VR, film, animation, broadcasting and other content industries.

[0004] However, despite the high level of attention given to 3D modeling technology, its technological sophistication is lacking. For example, companies aiming to create AI humans focus solely on developing technologies for generating virtual faces using CG or Deep Fake methods, while neglecting the development of technologies that recognize and naturally reproduce human movements.

[0005] Furthermore, with the widespread application of 3D modeling technology in all industries, the market for 3D creation tools is also growing. However, current 3D imaging creation tools only focus on generating static objects (buildings, interior decorations, facilities, etc.), while the technology for converting moving dynamic objects into 3D and representing them in 3D space is still underdeveloped. Summary of the Invention

[0006] The technical problem to be solved The technical problem to be solved by the embodiments of the present invention is to provide a method and system for generating three-dimensional objects by analyzing meaning. Specifically, the present invention provides a method and system for analyzing sentences provided in text form and automatically generating three-dimensional objects corresponding to the meaning of the sentences.

[0007] Therefore, the present invention can provide a method to generate a three-dimensional object that performs the action corresponding to the sentence by applying an action related to the meaning of the specified sentence to a three-dimensional model when a sentence is specified.

[0008] Furthermore, another technical problem to be solved by the embodiments of the present invention is to provide a method and system for generating a three-dimensional object that performs an overall action corresponding to the meaning of a sentence by sequentially combining multiple three-dimensional objects that perform partial actions corresponding to the meaning of words in a sentence.

[0009] Furthermore, the present invention relates to a method and system for generating and applying connecting actions between different actions to a 3D model, so that the 3D model can naturally perform multiple actions related to the meaning of a sentence.

[0010] Technical solutions to solve technical problems To address the aforementioned problems, the three-dimensional object generation method according to the present invention may include the following steps: specifying a sentence based on user input; analyzing the specified sentence and defining multiple actions corresponding to the sentence based on the analysis results; defining connecting actions that connect the multiple actions corresponding to the sentence; and generating a three-dimensional object that performs the multiple actions and the connecting actions using a learned artificial neural network.

[0011] In one example, the connecting action may be a connecting action that connects at least two actions that are adjacent to each other in a time-series flow among the plurality of actions.

[0012] In one example, the motion of the three-dimensional object may be formed by multiple frames corresponding to the first action, including key point information of the three-dimensional model based on the first action of the plurality of actions; multiple frames corresponding to the second action, including key point information of the three-dimensional model based on the second action of the plurality of actions; and multiple frames corresponding to the connecting action, including key point information of the three-dimensional model based on the connecting action that connects the first action and the second action.

[0013] In one example, each of the multiple frames corresponding to the first action, the multiple frames corresponding to the second action, and the multiple frames corresponding to the connecting action may include location information about key points of the 3D model.

[0014] In one example, when the first action and the second action are actions that are adjacent to each other in a time-series flow, the key points of the 3D model corresponding to the multiple frames of the connected action can be determined or defined based on the key point information of the 3D model corresponding to the multiple frames of the first action and the multiple frames of the second action.

[0015] In one example, when the first action precedes the second action according to the time series stream, the location information of the key points of the 3D model contained in the first frame of the plurality of frames corresponding to the connecting action can be related to the location information of the key points of the 3D model contained in the last frame of the plurality of frames corresponding to the first action, and the location information of the key points of the 3D model contained in the last frame of the plurality of frames corresponding to the connecting action can be related to the location information of the key points of the 3D model contained in the first frame of the plurality of frames corresponding to the second action.

[0016] In one example, the positions of the key points of the 3D model contained in the first frame of the connecting action may be the same as or adjacent to the positions of the key points of the 3D model in the last frame of the first action, and the positions of the key points of the 3D model contained in the last frame of the connecting action may be the same as or adjacent to the positions of the key points of the 3D model in the first frame of the second action.

[0017] The three-dimensional object generation system according to the present invention may include: a storage unit including a pre-set database; and a control unit, which analyzes a specified sentence based on user input, defines multiple actions corresponding to the sentence based on the analysis results, defines connecting actions connecting the multiple actions corresponding to the sentence, and uses a learned artificial neural network to generate a three-dimensional object that performs the multiple actions and the connecting actions.

[0018] A program executed by one or more processes and stored in a computer-readable recording medium in an electronic device may include instructions to perform the following steps: specifying a sentence based on user input; analyzing the specified sentence and defining multiple actions corresponding to the sentence based on the analysis results; defining connecting actions that connect the multiple actions corresponding to the sentence; and generating a three-dimensional object of motion that performs the multiple actions and the connecting actions using a learned artificial neural network.

[0019] The three-dimensional object generation method according to the present invention may include: specifying a sentence based on user input; analyzing the specified sentence and defining a plot corresponding to the sentence based on the analysis results; obtaining a three-dimensional model and action information corresponding to the plot using a pre-set database; and generating a three-dimensional object that performs the action corresponding to the sentence by applying the action information to the three-dimensional model.

[0020] Furthermore, the three-dimensional object generation system according to the present invention may include: a storage unit including a pre-set database; and a control unit, which analyzes a specified sentence based on user input, defines a plot corresponding to the sentence based on the analysis results, obtains a three-dimensional model and action information corresponding to the plot using the pre-set database, and generates a three-dimensional object that performs the action corresponding to the sentence by applying the action information to the three-dimensional model.

[0021] Furthermore, the program stored in a computer-readable recording medium according to the present invention is a program executed by one or more processes in an electronic device and stored in a computer-readable recording medium. The program may include instructions to perform the following steps: specifying a sentence based on user input; analyzing the specified sentence and defining a plot corresponding to the sentence based on the analysis results; obtaining a three-dimensional model and action information corresponding to the plot using a pre-set database; and generating a three-dimensional object that performs the action corresponding to the sentence by applying the action information to the three-dimensional model.

[0022] Invention Effects According to the three-dimensional object generation method and system of the present invention, by defining actions corresponding to the meaning of sentences and generating three-dimensional objects that perform the defined actions, users can generate moving three-dimensional objects simply by inputting sentences, without requiring users to define the actions of the three-dimensional objects one by one.

[0023] Furthermore, according to the three-dimensional object generation method and system of the present invention, information on various elements (such as actions, expressions, appearances, ages, clothing, genders, physical conditions, backgrounds, locations, and scenarios) of a three-dimensional model corresponding to a three-dimensional object can be obtained based on keywords specified in a sentence.

[0024] Furthermore, according to various embodiments of the present invention, the three-dimensional object generation method and system can naturally connect different actions performed by a three-dimensional object by generating connection action information that connects multiple action information related to the sequence of actions. Attached Figure Description

[0025] Figure 1 A three-dimensional object generation system according to the present invention is shown.

[0026] Figure 2 An embodiment of a database included in a three-dimensional object generation system according to the present invention is shown.

[0027] Figure 3 and Figure 4 This is a flowchart illustrating a three-dimensional object generation method according to the present invention.

[0028] Figure 5 An example is shown based on a sentence specified by user input.

[0029] Figures 6 to 9 Examples of actions that define the corresponding sentences are shown.

[0030] Figures 10 to 14 An example of applying motion information to a 3D model is shown. Detailed Implementation

[0031] The embodiments disclosed in this specification will now be described in detail with reference to the accompanying drawings. Regardless of the drawing numbers, identical or similar components are given the same reference numerals, and repeated descriptions are omitted. The suffixes "module" and "unit" used for components in the following description are designated or used interchangeably only for convenience of writing the specification and do not inherently have different meanings or functions. Furthermore, in describing the embodiments disclosed in this specification, detailed descriptions of relevant well-known technologies will be omitted if it is determined that such detailed descriptions might obscure the essence of the embodiments disclosed in this specification. Moreover, the accompanying drawings are only for facilitating understanding of the embodiments disclosed in this specification. The technical concepts disclosed in this specification are not limited by the drawings, and the drawings should be understood to include all modifications, equivalents, and substitutions within the scope and technical range of the invention.

[0032] Terms containing ordinal numbers (such as "first," "second," etc.) can be used to describe various components, but components are not limited by these terms. These terms are only used to distinguish one component from another.

[0033] When a component is described as "connected" or "accessed" to another component, it should be understood that the component may be directly connected to or accessed by that other component, and there may be other components in between. Conversely, when a component is described as "directly connected" or "directly accessed" to another component, it should be understood that there are no other components in between.

[0034] Unless the context clearly indicates otherwise, singular expressions include plural expressions.

[0035] In this application, it should be understood that terms such as “comprising” or “having” are intended to specify the presence of features, numbers, steps, operations, components, parts or combinations thereof described in the specification, but do not preclude the possibility of the presence or addition of one or more other features, numbers, steps, operations, components, parts or combinations thereof.

[0036] Figure 1 A three-dimensional object generation system according to the present invention is shown. Figure 2 An embodiment of a database included in a three-dimensional object generation system according to the present invention is shown.

[0037] Reference Figure 1According to the present invention, the 3D object generation system 100 can analyze a specified sentence based on user input 10 and define (or determine, generate, or extract) a continuity corresponding to the sentence. The 3D object generation system 100 can define a continuity corresponding to the sentence, or define actions, or define a 3D model, based on learning described later. In this invention, a continuity refers to information defining various aspects such as the actions a 3D model needs to perform, the appearance, size, expression, and gender of the 3D model to present the meaning or situation corresponding to the sentence; it can also be understood as a "storyboard." In this invention, "definition" can be understood as determining, generating, or extracting information or data.

[0038] The 3D object generation system 100 can, based on a defined scenario, retrieve a 3D model 1 and action information corresponding to a sentence from a pre-set database 131, and generate a 3D object 3 that performs actions based on the sentence by applying the action information to the 3D model 1. In this invention, the 3D model is the subject (e.g., a person, animal, object, etc.) that performs actions corresponding to the sentence, and the dynamic object that is moved by the 3D model and performs actions is called a 3D object. The 3D objects described in this invention can be modified and named as 3D dynamic objects, 3D graphic objects, 3D graphic dynamic objects, 3D modeling objects, 3D animation objects, or objects, dynamic objects, graphic objects, graphic dynamic objects, modeling objects, animation objects, etc. In addition, in the following description, the executing subject of this invention is referred to as the 3D object generation system 100, but it can also be described as a "control unit".

[0039] In this invention, the plot may include information defining at least one of the characters and actions of the 3D model. That is, the 3D object generation system according to this invention can define, generate, or extract information about at least one of the characters and actions of the 3D model from sentences based on a learned artificial intelligence model.

[0040] At this point, the character in the 3D model can include information that defines the visual appearance, personality, and other aspects of the 3D model.

[0041] The visual appearance of a 3D model can include at least some of the information that defines the visual appearance of the 3D model, such as its age, gender, height, weight, hairstyle, face shape, skin color, and clothing. Furthermore, a character can include at least some of the information that influences the behavioral characteristics of the 3D model, such as its occupation and personality. Based on a specified sentence, a 3D model can define people, animals, or objects in various ways.

[0042] In addition, the 3D object generation system 100 can define a plot based on a specified sentence input and other sentences (or other information, such as information contained in the paragraph containing the specified sentence, or sentences (or information) input before or after the specified sentence) associated with the specified sentence.

[0043] Furthermore, character information used to define the 3D model can be defined based on individual user input or settings, rather than extracted from sentences. In this case, the 3D object generation system according to the present invention can be configured with a separate process for generating or defining 3D models that will perform sentence-based actions. In this case, the 3D object generation system 100 can specify at least one action to be performed by the 3D model from a sentence and generate a 3D object in which the 3D model performs the specified action.

[0044] In addition, in this invention, it can also be understood that at least one “action” to be performed by the 3D model is defined from the sentence, rather than “defining the plot” from the sentence.

[0045] The 3D object generation system 100 can receive various forms of user input 10, such as voice signals, images, and text, and specify the sentence to be applied to the 3D model 1 based on the received user input 10.

[0046] For example, when a sentence such as "a person walks" is specified based on user input 10, the 3D object generation system 100 can define the action of "walking" based on the analysis results of the sentence and obtain action information related to "walking" from the database 131. Thus, the 3D object generation system 100 can generate and output a 3D object 3 performing the walking action by applying the obtained action information to the 3D model 1. At this time, when the 3D model 1 is defined as a person, the 3D object generation system 100 can generate and output a 3D object 3 performing the walking action by applying the obtained action information to the 3D model 1 presented in the shape of a person. In the following description, for ease of explanation, the case of a 3D model being a "person" is used as an example, but the 3D model in this invention is not necessarily limited to a person.

[0047] Additionally, in this specification, sentences can be specified based on user input 10, and sentences can be specified in the form of text describing at least one of the actions and appearance of the three-dimensional object 3.

[0048] Furthermore, 3D objects can be modeled and generated in 3D space and can be generated in various shapes such as people, animals, and objects.

[0049] For example, a 3D object 3 may include at least one of the following: multiple key points, a line connecting two different key points, a mesh representing the shape (or volume) of the 3D object 3, and a texture (or image) representing the texture and appearance of the 3D object 3.

[0050] At this point, the 3D model 1 can be configured to constitute the 3D object 3; that is, the 3D model 1 can form the skeleton (or framework) of the 3D object 3. For this purpose, the 3D model 1 can be configured to correspond to at least one of a plurality of key points in the 3D object 3, a line connecting two distinct key points, and a mesh representing the shape (or volume) of the 3D object 3. According to an embodiment, the 3D object 3 can be generated by overlaying textures onto the 3D model 1.

[0051] In addition, multiple key points can be positioned to correspond to joints in 3D model 1 (or 3D object 3).

[0052] Furthermore, the 3D model 1 may also include the following information as label data for each keypoint: information to distinguish each keypoint, the available range of movement for the position of each keypoint, and the available range of speeds for changes with respect to each keypoint. In this way, the 3D model 1 can be configured to achieve natural motion.

[0053] Furthermore, the plot can include not only the information used to define the actions of the 3D model and the information used to define the characters, as described above, but also information of various categories (or categories), such as information about the context in which the 3D model performs the actions and information about the background of the 3D model's actions. For example, the plot can include information (or definitions) of various categories (or categories), such as the actions of the 3D model, its appearance, age, clothing, gender, physical condition, and the background, location, and context in which the 3D model performs the actions. In other words, the plot can include (or define) information related to the shape of the 3D model to be obtained corresponding to the specified sentence, the actions to be applied to the 3D model, and the location, background, and context in which the 3D model will be placed.

[0054] For example, corresponding to the sentence "the cat jumps over the wall," the 3D object generation system 100 can define a scenario where a 3D model with a shape corresponding to a cat performs the action of jumping over a wall at a location where a wall is set up. Specifically, "cat" can be included in the scenario as a category related to the appearance of the 3D model, "wall" can be included in the scenario as a category related to the background where the 3D model is set up, and "jump" can be included in the scenario as a category related to the action to be applied to the 3D model.

[0055] Additionally, the motion information may include information generated to cause the 3D model 1 to move according to sentence-based actions. Specifically, the motion information may include information related to the motion of each of one or more keypoints included in the 3D model 1. That is, the motion information may include multiple poses or multiple frames (or times), and each pose or each frame may include position information of each of the multiple keypoints included in the 3D model 1. In this case, the position of each of the multiple keypoints may be specified in the motion information based on its location in 3D space.

[0056] For example, motion information may include information related to the positional changes and rate of change of each of the multiple key points included in the 3D model 1.

[0057] For example, motion information can be generated such that, for a series of time intervals corresponding to the motion information, position information of each of the multiple key points in the 3D model 1 is specified at a preset time period (e.g., every 1 / 60 of a second).

[0058] Furthermore, the motion information may also include facial expression information corresponding to an expression category generated to cause the facial features of the 3D model 1 to move according to a sentence-based expression. In this case, the facial expression information may include information related to the motion of each of the keypoints belonging to the facial features of the 3D model 1, and therefore can be set to be included in the motion information.

[0059] Additionally, refer to Figure 2 Database 131 can store multiple distinct 3D models 1 and multiple action information 20. Here, the multiple distinct 3D models 1 can refer to multiple 3D models 1 configured with different appearances. For example, the multiple distinct 3D models 1 can include objects with various appearances, such as cats, people, boys, girls, and dogs. In this case, the multiple distinct 3D models 1 can be configured with different numbers and positions of key points, mesh shapes and textures, etc., based on the appearance of each object.

[0060] Database 131 can also store tag information 60 (or tag, keyword) that matches each of the multiple 3D models 1 and multiple motion information 20.

[0061] The tagging information 60 can be set to correspond to the analysis results of the sentence. That is, the tagging information 60 is a keyword used to define the plot based on the sentence, and can represent elements such as plot-based actions and the appearance of the 3D model. In other words, the 3D object generation system 100 can define the plot corresponding to the sentence by obtaining keywords from the sentence. At this time, the 3D object generation system 100 can also classify the keywords according to plot-based categories.

[0062] For example, the tagging information 60 may include words related to the action of walking, such as "person" and "walking". As another example, the tagging information 60 may further include objects, complements, and adverbs in association with the action based on the subject and predicate contained in the sentence.

[0063] In this way, the 3D object generation system 100 can utilize plots (e.g., keywords or words) defined based on the analysis results of sentences to obtain a 3D model 1 and action information corresponding to the sentence from the database 131. In particular, the 3D object generation system 100, through morpheme analysis, considers not only the subject and predicate contained in the sentence but also various sentence components such as objects, complements, and adverbs, thereby obtaining a 3D model 1 with a more accurate and diverse appearance and action-related action information 20.

[0064] In this regard, database 131 can also store an artificial neural network model that, when a plot defined by a sentence is input, outputs a 3D model 1 and action information corresponding to the sentence. In this case, the 3D object generation system 100 according to the present invention can input a plot defined based on the analysis results of the sentence into the artificial neural network model set in database 131, and obtain the 3D model 1 and action information corresponding to the sentence from the artificial neural network model.

[0065] Therefore, an artificial neural network model can be trained using a learning scenario, a correct 3D model labeled on the learning scenario as tag data, and correct action information. The 3D model and action information can include the various types of information described above. Thus, the artificial neural network model can learn using learning scenarios and correct action information corresponding to the various categories described above, such that when a scenario is input, the output is a 3D model 1 and action information corresponding to the input scenario.

[0066] In this regard, the overall operation of the three-dimensional object generation system 100 according to the present invention will be described below by searching for three-dimensional models and motion information corresponding to the plot in database 131. However, it should be understood that the three-dimensional models and motion information in the present invention can also be generated by artificial neural network models as described above.

[0067] Furthermore, database 131 can store data categorized according to the aforementioned categories. In this case, the 3D object generation system 100 can match the plot corresponding to the sentence by category and retrieve the data corresponding to the matched category from database 131.

[0068] According to the embodiment, database 131 can also store multiple artificial neural network models for the various categories mentioned above, and can obtain data corresponding to each category.

[0069] Therefore, the three-dimensional object generation system 100 according to the present invention may include an input unit 110, a storage unit 130, a control unit 150, and an output unit 170. Additionally, Figure 1 Only components relevant to embodiments of the present invention are shown. Therefore, those skilled in the art should understand that, in addition to… Figure 1 In addition to the components shown, other general-purpose components may also be included.

[0070] Input unit 110 can receive user input 10 through a pre-configured input device. For example, the input device can refer to various devices such as a keyboard, mouse, touchpad, microphone (MIC), and line-in. Therefore, input unit 110 can receive various types of user input 10, such as voice and text. In this way, control unit 150 can acquire the sentence corresponding to user input 10.

[0071] Storage unit 130 can store information and instructions required for the operation of the three-dimensional object generation system 100 according to the present invention. Therefore, storage unit 130 can load one or more programs to perform methods / operations according to various embodiments of the present invention. An example of storage unit 130 may be RAM, but is not limited thereto.

[0072] Additionally, storage unit 130 can store a database containing multiple motion information entries. Furthermore, storage unit 130 (or database) can store a 3D model 1 to which motion information is to be applied.

[0073] The control unit 150 can control the overall operation of the three-dimensional object generation system 100 according to the present invention. For this purpose, the control unit 150 can be configured to include at least one of a central processing unit (CPU), a microprocessor (MPU), a microcontroller (MCU), a graphics processing unit (GPU), or any other type of processor known in the art. Furthermore, the control unit 150 can perform at least one application or program-related calculations for performing methods / operations according to various embodiments of the present invention. The three-dimensional object generation system 100 may also be configured with one or more control units 150.

[0074] In addition, the control unit 150 can analyze the sentence corresponding to the user input 10 to define the plot corresponding to the sentence from the database 131, obtain the three-dimensional model and action information based on the plot, and generate the three-dimensional object 3 by applying the action information to the three-dimensional model 1.

[0075] The output unit 170 can output the three-dimensional object 3 generated by the control unit 150. At this time, the output unit 170 can output the three-dimensional object 3 so that the user can visually confirm it through a display device such as a monitor or television (TV).

[0076] Based on the structure of the 3D object generation system 100 as described above, the 3D object generation method will be described in more detail below.

[0077] Figure 3 and Figure 4 This is a flowchart illustrating a three-dimensional object generation method according to the present invention. Figure 5 An example is shown based on a sentence specified by user input. Figures 6 to 9 Examples of actions that define the corresponding sentences are shown. Figures 10 to 14 An example of applying motion information to a 3D model is shown.

[0078] Reference Figure 3 According to the present invention, the three-dimensional object generation system 100 can analyze the specified sentence based on the user input 10 (S100), and define the plot corresponding to the sentence based on the analysis result (S200).

[0079] Specifically, when the data input by user input 10 is text, the 3D object generation system 100 can specify a sentence based on the input text. Furthermore, when the data input by user input 10 is in a form other than text, the 3D object generation system 100 can specify the sentence by analyzing the input data and converting it into text, and defining the actions to be applied to the 3D model 1 based on the converted text.

[0080] Reference Figure 5 For example, when the user inputs text 10a, the 3D object generation system 100 can designate the input text 10a as a sentence.

[0081] For example, when user input 10 inputs data containing multiple texts, the 3D object generation system 100 can analyze the input data to extract multiple texts, segment the extracted multiple texts, and assign each segmented text as a sentence for generating the 3D object 3 in the order it appears in the data.

[0082] To this end, the 3D object generation system 100 can segment multiple texts based on the text in the data that corresponds to preset punctuation marks (e.g., periods). Thus, the 3D object generation system 100 can sequentially analyze the segmented texts according to the order they appear in the data and define the plot corresponding to each sentence.

[0083] For example, when a user inputs a voice signal 10b, the 3D object generation system 100 can generate text (or sentence) 11a corresponding to the user input 10 by analyzing the voice signal 10b.

[0084] At this time, the 3D object generation system 100 can utilize a technique for converting the speech signal 10b corresponding to the user input 10 into text 11a (e.g., speech-to-text), and for this purpose may include a speech model or an artificial neural network, etc.

[0085] For example, when an image is input based on user input 10, the 3D object generation system 100 can generate text corresponding to user input 10 by analyzing the image.

[0086] At this time, the 3D object generation system 100 can utilize techniques for converting the image corresponding to the user input 10 into text (e.g., optical character recognition (OCR), and for this purpose may include character detection models, character recognition models, and artificial neural networks, etc.).

[0087] For example, when data generated based on user input 10 according to a preset format (e.g., script format), the 3D object generation system 100 extracts one or more texts from a region that is pre-defined to represent the appearance, background, and actions of the object in accordance with the format, and sequentially designates the extracted one or more texts as sentences for generating the 3D object 3.

[0088] At this point, if the preset format is a script, the 3D object generation system 100 can extract one or more texts from the areas in the script corresponding to the instructions (actions) (the parenthetical content of 11b, 11c, 11d, and 11e). Therefore, the 3D object generation system 100 can sequentially assign each text as a sentence for generating the 3D object 3 according to the order contained in the data.

[0089] Furthermore, the 3D object generation system 100 can segment a sentence into multiple words based on morphemes, confirm the meanings corresponding to the segmented words, and generate (or extract) at least one keyword corresponding to the meaning of the specified sentence based on the confirmed meanings. At this point, the at least one keyword generated by the 3D object generation system 100 that corresponds to the meaning of the sentence can define the plot corresponding to the sentence.

[0090] For example, the 3D object generation system 100 can use artificial neural networks (e.g., recurrent neural networks (RNNs) and long short-term memory networks (LSTMs)) to define plots corresponding to sentences. These artificial neural networks are learned to output keywords related to the plots corresponding to the input words when multiple words in a sentence are input sequentially.

[0091] In this case, the 3D object generation system 100 can input the specified sentence into a pre-learned artificial neural network to obtain keywords related to the plot corresponding to the sentence, and generate a 3D object that performs the action corresponding to the keyword.

[0092] For example, the 3D object generation system 100 can identify the part of speech of each word in a sentence and extract words with parts of speech related to the object's actions as keywords.

[0093] At this point, the 3D object generation system 100 can utilize an artificial neural network that has been pre-learned to identify the part of speech of words, or it can identify the part of speech of words through a pre-set language model.

[0094] With the configuration described above, the three-dimensional object generation system 100 according to the present invention can generate plot-related keywords from a sentence specified by user input 10.

[0095] For example, refer to Figure 6 When a sentence such as "He sat down on the ground and then stood up" is specified, the 3D object generation system 100 can obtain "he," "sit," "stand up," and "ground" as plot-related keywords by analyzing the morphemes of the specified sentence. At this time, the 3D object generation system 100 can define the appearance of the 3D model corresponding to "he" based on the sentence structure of the obtained keywords, define the action of sitting on the floor corresponding to "sit" and "floor," and define the action of standing up from the floor corresponding to "he," "stand up," and "floor."

[0096] For example, when a sentence such as “sit down cross-legged and then stand up” is specified, the 3D object generation system 100 can define the actions of sitting down and standing up as plots corresponding to the analysis results of the specified sentence, and accordingly, can generate keywords related to “sitting down” and “standing up”.

[0097] Furthermore, the 3D object generation system 100 can confirm the correlation between multiple words that are different from each other based on the meaning of each of the multiple words segmented from the sentence, and determine at least one of the action level and action sequence of at least one keyword related to the sentence based on the confirmed correlation.

[0098] Here, action level can refer to the level of a part (e.g., arm, leg, person) that performs multiple different actions at the same point in time for a 3D model 1.

[0099] Furthermore, action sequence can represent the order in which multiple distinct actions are performed on a 3D model 1. In other words, action sequence can represent the temporal order in which actions are performed on a 3D object.

[0100] For example, the 3D object generation system 100 can generate objects for the sentence "sit cross-legged (…)". Generates such as "legs ( )","plate( ")" and "sit ( The keyword is “)”. At this point, the 3D object generation system 100 can generate an object based on the postposition “leg”. The object being referred to as "disc" is confirmed to be "leg". Therefore, the 3D object generation system 100 can determine that "leg" and "disc" have the same level of motion.

[0101] Furthermore, the 3D object generation system 100 can be based on the conjunction "disk". "It is confirmed that 'sitting' and 'crossing over' are performed at the same time, and based on the fact that the action of 'sitting' is performed by the object, it can be determined that the action of 'crossing over' regarding 'legs' has a lower action level than 'sitting'."

[0102] For example, the 3D object generation system 100 can generate objects based on the sentence "sit down, then stand up (…)". Generates such as "sit ()" ")" and "Stand up ( The keyword is “sit”. At this point, the 3D object generation system 100 can use the auxiliary verb “sit”. It is confirmed that "standing up" is performed after "sitting". Therefore, the 3D object generation system 100 can determine that "sitting" has an action sequence that precedes "standing up".

[0103] Furthermore, when the plot defined above includes various elements related to three-dimensional objects, the three-dimensional object generation system 100 can match each of the multiple keywords corresponding to the plot to one of the multiple categories preset based on various elements (e.g., action, expression, appearance, age, clothing, gender, physical condition, background, location, scene, etc.), and retrieve the information corresponding to each keyword from the database based on the matched category.

[0104] For example, the 3D object generation system 100 can match at least one keyword from a plurality of keywords corresponding to a plot to a category related to facial expression, so as to apply the facial expression to the face of the 3D model 1. At this time, the 3D object generation system 100 can match keywords related to words corresponding to emotions (or feelings) from a plurality of words contained in a sentence to a category related to the facial expression corresponding to the sentence.

[0105] In one embodiment, the 3D object generation system 100 can generate keywords such as "happy" and "run" for the sentence "run happily". In this case, the 3D object generation system 100 can match the keyword "happy" corresponding to the emotion to a category related to facial expression, and can match the keyword "run" corresponding to the action to a category related to the action.

[0106] Furthermore, when a sentence contains multiple words including keywords related to the appearance of the 3D object 3 (or 3D model 1), the 3D object generation system 100 can also match these keywords to appearance-related categories. In this case, the 3D object generation system 100 can match keywords related to the appearance of the 3D object 3 to keywords related to actions. Thus, the 3D object generation system 100 can retrieve information corresponding to each category from the database (e.g., 3D model and action information), enabling the 3D object 3, which has an appearance corresponding to the sentence, to perform actions corresponding to the sentence.

[0107] For example, the 3D object generation system 100 can generate the keyword "person" as a keyword related to the appearance of the 3D model 1 for the sentence "a person is running". At this time, the 3D object generation system 100 can match the keyword related to the action "running" to the keyword "person" related to the appearance of the 3D model 1.

[0108] For example, the 3D object generation system 100 can generate the keyword "cat" as a keyword related to the appearance of the 3D model 1 for the sentence "the cat jumps over the wall". At this time, the 3D object generation system 100 can match the action-related keywords "wall" and "jump" (or "flip" and "jump") to the keyword "cat" related to the appearance of the 3D model 1.

[0109] For example, the 3D object generation system 100 can generate the keywords "boy" and "girl" respectively for the sentence "a boy and a girl are standing under an umbrella" as keywords related to the appearance of the 3D model 1. At this time, the 3D object generation system 100 can match the action-related keywords "umbrella," "under," and "stand" to the appearance-related keywords "boy" and "girl."

[0110] In this way, the 3D object generation system 100 can determine the actions and appearance of 3D objects based on keywords generated through semantic analysis of sentences, and can also generate 3D objects based on various keywords related to the age, clothing, gender, physical condition (e.g., weight, height, etc.) of 3D objects.

[0111] Furthermore, when generating keywords about the space from which the three-dimensional object will be generated (e.g., location, background, appearance of the object placed in the space, context, etc.) through semantic analysis of the sentence, the three-dimensional object generation system 100 can also generate other three-dimensional objects adjacent to the three-dimensional object to correspond to the keyword.

[0112] Therefore, data corresponding to various categories such as actions, appearance, expressions, age, clothing, gender, and physical condition can be stored in the database included in the 3D object generation system 100.

[0113] With the above configuration, the three-dimensional object generation system 100 according to the present invention can define at least one action that best suits the sentence by analyzing the specified sentence based on user input 10.

[0114] Furthermore, as a result of the analysis of the sentences specified above, the 3D object generation system 100 can also define multiple scenarios (or multiple plots or multiple actions). In this case, each of the multiple scenarios can be realized in different spaces at the same time, or it can be realized in the same space at different times.

[0115] Therefore, the 3D object generation system 100 can use a database to obtain 3D models and motion information corresponding to each situation, and generate 3D objects that perform actions corresponding to each situation by applying the motion information to the 3D model.

[0116] According to an embodiment, when multiple different situations are applied to a single three-dimensional model, the three-dimensional object generation system 100 can generate a three-dimensional object, such that the three-dimensional object performs the action corresponding to the previous situation in the order of the sentence analysis results based on the multiple situations, and then performs the action corresponding to the next situation after a preset time interval.

[0117] Furthermore, the 3D object generation system 100 can also define conditional scenarios for specified situations based on the analysis results of the aforementioned sentences. These conditional scenarios can be set such that when a condition set based on the sentence analysis results (or user input) is met, the 3D object performs an action corresponding to the conditional scenario.

[0118] For example, a conditional plot can be defined as performing a specified action (e.g., stretching) at a specified time (e.g., 6 a.m.). In this case, a 3D object generated based on the conditional plot can be implemented to perform the specified action at the specified time according to the scenario.

[0119] For example, a conditional plot can be defined as performing a specified action (e.g., a greeting) in response to a pre-defined user input (e.g., a click on a 3D object). In this case, the 3D object generated based on the conditional plot can be configured to perform the specified action in response to the user input.

[0120] To this end, the 3D object generation system 100 retrieves the 3D model and action information corresponding to the conditional plot from the database, and generates a 3D object by applying the action information to the 3D model. Here, an event flag can be set for the action information applied to the 3D model to represent the condition corresponding to the conditional plot.

[0121] Therefore, the 3D object generation system 100 can also generate 3D objects that correspond to actions performed under specified conditions, and can also generate single 3D objects that apply multiple conditional plots based on different conditions.

[0122] Furthermore, the 3D object generation system 100 can also directly define at least one action as the analysis result of the specified sentence. In this case, the 3D object generation system 100 can also retrieve action information corresponding to the at least one action defined above from the database and apply the action information to a pre-set 3D model to generate a 3D object that performs the action corresponding to the sentence.

[0123] In other words, the 3D object generation system 100 can also generate at least one keyword based on a sentence, obtain at least one of the 3D model and motion information corresponding to the above-mentioned at least one keyword from the database, and generate a 3D object by applying the motion information to the 3D model.

[0124] Refer again Figure 3 According to the present invention, the three-dimensional object generation system 100 can use a pre-set database 131 to obtain three-dimensional models and motion information corresponding to the plot (S300).

[0125] Specifically, the 3D object generation system 100 can search the database 131 for 3D models and motion information corresponding to at least one keyword generated above. Thus, the 3D object generation system 100 can obtain the searched 3D models and motion information.

[0126] For example, when generating keywords such as "running", the 3D object generation system 100 can obtain the action information corresponding to the running action by searching the database 131 for the action information corresponding to "running".

[0127] Reference Figure 7 As another example, for the sentence "He sat down on the ground and then stood up," the 3D object generation system 100 can define the appearance of the 3D model related to "him" as a plot and search for the 3D model corresponding to "him" in the database 131. Furthermore, the 3D object generation system 100 can define a first action related to "sitting" and "ground" and a second action related to "him," "standing up," and "ground," and search for action information corresponding to the first and second actions.

[0128] Thus, the 3D object generation system 100 can obtain from the database 131 a 3D model related to a person, motion information 21 related to the action of sitting on the floor, and motion information 22 related to the action of standing up from the floor.

[0129] Reference Figure 8 As another example, when generating keywords such as "legs," "cross-legged," and "sit," the 3D object generation system 100 can search for action information corresponding to "sit" in the database 131 based on the action level set for the keyword. At this time, the 3D object generation system 100 can confirm the action information of various sitting postures corresponding to "sit" from the database 131, such as the action 21a of sitting cross-legged, the action 21b of squatting, and the action 21c of sitting with legs together.

[0130] Therefore, the 3D object generation system 100 can identify the corresponding action information for "leg" and "cross-legged" based on the action level among the multiple action information searched for "sitting," and obtain the identified action information. Thus, when generating keywords divided into multiple action levels, the 3D object generation system 100 can consider searching the database 131 for action levels to obtain the action information most suitable for the multiple keywords.

[0131] Reference Figure 8 As another example, when generating keywords such as "sit" and "stand up," the 3D object generation system 100 can search for action information 21 corresponding to "sit" in the database 131 based on the action sequence set for the keywords, and obtain the searched action information 21. Then, the 3D object generation system 100 can search for action information 22 corresponding to "stand up" in the database 131 based on the action sequence, and obtain the searched action information 22.

[0132] Therefore, the 3D object generation system 100 can obtain multiple action information corresponding to a sentence by sequentially listing action information 21 related to "sitting" and action information 22 related to "standing up".

[0133] For example, when generating keywords such as "happy" and "run", the 3D object generation system 100 can search and obtain the expression information corresponding to "happy" from the database 131, and can search and obtain the action information corresponding to "run".

[0134] Furthermore, when generating keywords related to the appearance of the 3D model 1 in relation to a sentence, the 3D object generation system 100 can obtain the 3D model 1 corresponding to the aforementioned appearance-related keywords from the database 131.

[0135] For example, when generating the keywords "person" and "run", the 3D object generation system 100 can obtain the 3D model corresponding to the keyword "person" and the action information corresponding to the keyword "run" from the database 131.

[0136] For example, when generating keywords such as "cat", "wall", and "jump" (or "jump" and "jump"), the 3D object generation system 100 can obtain the 3D model corresponding to the keyword "cat" and the action information corresponding to the keyword "wall" and "jump" (or "jump" and "jump") from the database 131.

[0137] For example, when generating keywords "boy," "girl," "umbrella," "down," and "stand," the 3D object generation system 100 can retrieve the 3D models corresponding to the keywords "boy" and "girl" from the database 131, and obtain the action information corresponding to the keywords "umbrella," "down," and "stand." In this case, the 3D object generation system 100 can also be implemented by retrieving the action information corresponding to the keywords "umbrella," "down," and "stand" from the action information related to "boy" and "girl" in the database 131.

[0138] Furthermore, when a plot related to multiple actions is defined for a sentence, the 3D object generation system 100 can obtain action information corresponding to each of the multiple actions from the database 131, and sequentially connect the obtained action information according to the action order specified based on the plot.

[0139] Reference Figure 4 When the 3D object generation system 100 obtains multiple action information from the database 131 (S310), it can sequentially connect the multiple action information obtained above according to the action order specified based on the analysis results of the sentence (S320).

[0140] Therefore, the 3D object generation system 100 can connect the last frame of the earlier action information with the first frame of the next action information to connect multiple different action information in sequence, for two action information with adjacent action order among multiple action information obtained from the database 131.

[0141] At this time, the 3D object generation system 100 can generate connection action information for two action information with adjacent action sequences, so that actions based on multiple action information obtained from the database 131 are naturally connected (S330).

[0142] Connecting action information is a type of action information that can include information related to actions performed between preceding and subsequent actions to allow two different actions to be connected naturally.

[0143] To this end, the 3D object generation system 100 can generate connection motion information such that, among the two motion information obtained from the database 131, the final pose of the first motion information with the action sequence preceding the final pose becomes the initial pose (or the first frame) of the second motion information with the next action sequence.

[0144] Here, the final pose of the first motion information can be the pose of the last frame among the multiple frames (or times) contained in the first motion information, and the initial pose of the second motion information can be the pose of the first frame among the multiple frames (or times) contained in the second motion information.

[0145] Reference Figure 10 For example, the 3D object generation system 100 can generate connection motion information 50, so that the positions of multiple key points 40a in the final pose 31 of the first motion information 21 are moved to the positions of multiple key points 40b in the initial pose 32 of the second motion information 22.

[0146] At this time, the 3D object generation system 100 can generate connection motion information 50, so that the position of each of the multiple key points 40a corresponding to the final pose of the first motion information becomes the position of each of the multiple key points 40b corresponding to the initial pose of the second motion information.

[0147] As another example, the 3D object generation system 100 can generate multiple first key points for a first pose to be included in the connected motion information by using the center points of the positions of multiple key points corresponding to the final pose of the first motion information and the positions of multiple key points corresponding to the initial pose of the second motion information; generate multiple second key points for a second pose to be included in the connected motion information by using the positions of the multiple key points of the first motion information and the center points of the positions of the multiple first key points; and generate multiple third key points for a third pose to be included in the connected motion information by using the positions of the multiple first key points and the center points of the positions of the multiple key points of the second motion information.

[0148] As described above, the 3D object generation system 100 can repeatedly generate multiple key points for multiple poses to be included in the connected motion information by utilizing the center point between key points included in different actions.

[0149] Therefore, when multiple poses to be included in the connection action information are generated, the 3D object generation system 100 can determine the order of the multiple poses to be included in the connection action information based on the order in which the positions of multiple key points corresponding to the multiple poses to be included in the connection action information are closest to the positions of multiple key points corresponding to the final pose of the first action information, and generate the connection action information in the determined order.

[0150] Alternatively, when multiple poses to be included in the connection action information are generated, the 3D object generation system 100 can determine the order of the multiple poses to be included in the connection action information based on the order of the positions of multiple key points corresponding to the multiple poses to be included in the connection action information being farther from the positions of multiple key points corresponding to the initial pose of the second action information, and generate the connection action information in the determined order.

[0151] For example, the 3D object generation system 100 can generate a connecting motion line by connecting the positions of multiple key points corresponding to the final pose of the first motion information and the positions of multiple key points corresponding to the initial pose of the second motion information. Furthermore, it can generate multiple key points on the generated connecting motion line from multiple key points of the first motion information to multiple key points of the second motion information, according to a preset interval (or preset number).

[0152] Therefore, the 3D object generation system 100 can determine the order of multiple poses to be included in the connected motion information based on the order of their proximity to multiple key points of the first motion information, and generate the connected motion information in the determined order.

[0153] Alternatively, the 3D object generation system 100 can determine the order of multiple poses to be included in the connected motion information based on the order of their distance from multiple key points of the second motion information, and generate the connected motion information in the determined order.

[0154] Furthermore, in the process of generating connection action information, the 3D object generation system 100 can determine the action speed of the connection action information by considering the first action speed of the first action information (i.e., the prior action of the connection action information) and the second action speed of the second action information (i.e., the subsequent action of the connection action information).

[0155] For example, refer to Figure 11 The 3D object generation system 100 can determine a first motion speed based on the distance difference 41 between the positions of multiple key points 40a in the final pose 31 of the first motion information and the positions of multiple key points 40c in the pose 33 immediately preceding the final pose 31, and can determine a second motion speed based on the distance difference 42 between the positions of multiple key points 40b in the initial pose 32 of the second motion information and the positions of multiple key points 40d in the pose immediately following the initial pose 32.

[0156] Therefore, the 3D object generation system 100 can determine the motion speed of the connected motion information so that the posture that is closer to the final posture 31 among the multiple postures contained in the connected motion information has a speed that is closer to the first motion speed, and can determine the motion speed of the connected motion information so that the posture that is closer to the initial posture 32 has a speed that is closer to the second motion speed.

[0157] For example, the 3D object generation system 100 can calculate the center point between the positions of multiple key points corresponding to the final pose 31 and the positions of multiple key points corresponding to the initial pose 32 by applying the ratio of the first motion speed and the second motion speed to calculate the center point of the key points in the final pose 31 and the key points in the initial pose 32 by applying the motion speed.

[0158] Therefore, the 3D object generation system 100 can generate multiple poses to be included in the connection motion information by repeatedly generating at least one pose close to the final pose 31 and at least one pose close to the initial pose 32 based on the center point of the applied motion speed. In this way, the 3D object generation system 100 can generate one or more poses with greater distances between keypoints for poses with faster motion speeds that are closer to the final pose 31 and the initial pose 32, and generate one or more poses with closer distances between keypoints for poses with slower motion speeds that are closer to the final pose 31 and the initial pose 32. Therefore, the 3D object generation system 100 can sequentially list the multiple poses generated through the above process and generate connection motion information.

[0159] For example, the 3D object generation system 100 generates a first pose to be included in the connecting motion information by calculating the center point between the positions of multiple key points corresponding to the final pose 31 and the positions of multiple key points corresponding to the initial pose 32. It can also determine the number of poses to be generated between the final pose 31 and the first pose, and the number of poses to be generated between the initial pose 32 and the second pose, based on the ratio of the first motion speed and the second motion speed.

[0160] For example, the 3D object generation system 100 can generate connecting motion lines by connecting the positions of multiple key points corresponding to the final pose 31 and the positions of multiple key points corresponding to the initial pose 32. On the generated connecting motion lines, as the final pose 31 approaches, poses to be included in the connecting motion information are generated at intervals with a small difference from the distance difference 41 corresponding to the first motion speed. As the initial pose 32 approaches, poses to be included in the connecting motion information are generated at intervals with a small difference from the distance difference 42 corresponding to the second motion speed.

[0161] Furthermore, refer to again Figure 4 The 3D object generation system 100 can sequentially connect multiple action information and insert connecting action information between each action information to make connections (S340).

[0162] For example, referring to 12, the 3D object generation system 100 can, corresponding to the sentence "sit down, then stand up," acquire first action information 21 about the action of sitting and second action information 22 about the action of standing up, and generate connecting action information 50 for the first action information 21 and the second action information 22. Thus, the 3D object generation system 100 can connect the final posture of the first action information 21 with the initial posture of the connecting action information 50, and connect the final posture of the connecting action information 50 with the initial posture of the second action information 22, thereby connecting the first action information, the connecting action information, and the second action information.

[0163] With the configuration described above, the three-dimensional object generation system 100 according to the present invention can obtain at least one action information from the database 131 based on at least one keyword corresponding to the sentence.

[0164] Furthermore, the three-dimensional object generation system 100 according to the present invention can also obtain information on the appearance and expression of the three-dimensional object 3 based on at least one keyword corresponding to the sentence generated.

[0165] Furthermore, the three-dimensional object generation system 100 according to the present invention can naturally connect different actions by generating connection action information for the continuous action sequence of multiple action information.

[0166] Refer again Figure 3 According to the present invention, the three-dimensional object generation system 100 can generate a three-dimensional object 3 that performs an action corresponding to a sentence by applying action information to a three-dimensional model 1 (S400).

[0167] Specifically, the 3D object generation system 100 can apply the position of at least one key point specified in the first frame, which is the foremost of the multiple frames (or times) contained in the motion information, to at least one key point corresponding to the aforementioned at least one key point among the multiple key points contained in the pre-set (or obtained from the database 131) 3D model 1.

[0168] Then, the 3D object generation system 100 can apply the position of at least one key point specified in the second frame immediately following the first frame to at least one key point among the multiple key points contained in the 3D model 1 that corresponds to the at least one key point, and repeat the above process for all frames specified in the motion information, thereby applying the motion information to the 3D model 1.

[0169] For example, refer to Figure 13 and Figure 14 The 3D object generation system 100 can apply the positions of multiple keypoints 40 corresponding to each pose 35 to multiple keypoints contained in the 3D model 1. At this time, the 3D object generation system 100 can generate a 3D object 3 that performs the action corresponding to the sentence by setting the multiple keypoints contained in the 3D model 1 to move sequentially according to the poses of the action information connected above.

[0170] With the configuration described above, the 3D object generation system 100 according to the present invention can generate 3D objects 3 that perform actions corresponding to sentences. Furthermore, by generating and applying connecting actions between different actions performed by the 3D model 1, the 3D object generation system 100 is able to generate 3D objects 3 that move more naturally.

[0171] Additionally, the 3D object generation system 100 can also generate 3D objects from 2D images. For example, when a 2D image is input, the 3D object generation system 100 can detect objects in the 2D image, then identify the parts corresponding to the objects through object segmentation, and extract pose-related data of the objects from each frame of the 2D image. Thus, the 3D object generation system 100 can collect and merge the pose data extracted from each frame of the 2D image by angle, and generate 3D objects with volume and texture based on the merged results.

[0172] As one embodiment, a three-dimensional object can be generated based on a three-dimensional volume metric. The three-dimensional volume metric can be implemented by setting the initial position of the object in a two-dimensional image as a reference point, then determining the position information of the corresponding position when the object moves to another position in the two-dimensional image, thereby estimating the size of the object, and then generating the volume in the form of a three-dimensional mesh.

[0173] More specifically, the 3D object generation system 100 can receive a 2D image as input and output a 3D object in the form of a mesh based on the volume and texture of the object in the 2D image. In this case, the input 2D image can include multiple frames representing the shape of the object at various angles and / or positions.

[0174] The 3D object generation system 100 can analyze key points and lines representing objects from a 2D image and generate a 3D point cloud as the shape information of the object.

[0175] The 3D object generation system 100 can perform 3D mesh modeling and texture mapping based on the generated 3D point cloud to generate 3D mesh objects.

[0176] When a two-dimensional image is input, the three-dimensional object generation system 100 can track key points of objects in the two-dimensional image 41 and reconstruct a three-dimensional model based on these key points. Simultaneously, the three-dimensional object generation system 100 can detect and fit lines of objects in the two-dimensional image, thereby reconstructing a line-based three-dimensional model. The three-dimensional object generation system 100 can utilize pre-determined reconstruction parameters during the reconstruction of the three-dimensional model.

[0177] In addition, the 3D object generation system 100 can generate a 3D point cloud (PC) using a 3D model reconstructed based on key points and / or lines, which estimates the 3D shape of an object in a 2D image.

[0178] The 3D object generation system 100 can perform 3D mesh modeling based on a 3D point cloud (PC). The result of the 3D mesh modeling can be output as mesh information (MI) representing the volume of the 3D object. The 3D object generation system 100 can perform 3D mesh modeling using pre-defined modeling parameters. Simultaneously, the 3D object generation system 100 can perform texture mapping based on a color image of a 2D image. The result of the texture mapping can be output as texture information (TI) representing the texture of the 3D object.

[0179] Furthermore, the 3D object generation system 100 can utilize mesh information (MI) and texture information (TI) to construct data related to a 3D mesh, and then generate a 3D object based on this.

[0180] Therefore, the 3D object generation system 100 can merge previously generated motion information into the 3D object, thereby enabling the 3D object to perform 3D motion based on the motion information.

[0181] At this point, the 3D object can be a 3D mesh object in the form of a 3D grid, or a pre-stored 3D character object. Alternatively, if the 3D object generation system 100 uses a 3D character object that simulates a celebrity or virtual character instead of a 3D mesh object, by incorporating motion information into the 3D character object, a 3D object can be generated that reproduces the motion of the object in the 2D image.

[0182] In this regard, since action information is data generated based on sentences or two-dimensional images, while three-dimensional objects are objects generated independently of sentences or two-dimensional images, inconsistencies may arise between action information and three-dimensional objects. Therefore, the three-dimensional object generation system 100 can perform a re-targeting operation to adjust the action information to fit the three-dimensional object in order to resolve this inconsistency.

[0183] For example, the 3D object generation system 100 can align the joint positions of the motion information with the joint positions of the 3D object based on data related to key points of the motion information. Here, aligning the joint positions of the motion information with the joint positions of the 3D object can, for example, mean adjusting the joint positions of the motion information so that they match the joint positions of the 3D object.

[0184] For example, when motion information is applied to a 3D object, the body features of the model on which the motion information is based may differ from the body features of the 3D object. Therefore, in order to eliminate the inconsistency caused by the differences in body features between objects, the joint positions (i.e., keypoints) of the motion information can be adjusted to match the joint positions (i.e., keypoints) of the 3D object.

[0185] At this point, the lines of the motion information can also be adjusted based on the adjusted key points of the motion information. For example, suppose that the motion information extracted from the database contains key points K1 (1,1,1), K2 (2,2,2) and line L1 connecting key points K1 and K2. When key points K1 and K2 are adjusted to K1 (1,2,2) and K2 (2,3,3) respectively through redirection, line L1 is also adjusted to be the line connecting (1,2,2) and (2,3,3).

[0186] Furthermore, the 3D object generation system 100 can realize the motion of the 3D object as the motion of the simulated motion information based on the joint positions of the aligned motion information.

[0187] Therefore, the 3D object generation system 100 can adjust the motion of the motion information based on the adjusted joint positions (i.e., key points) and lines, such as the position, movement distance, and range of motion of each key point and line. Moreover, by incorporating the motion of the adjusted motion information into the 3D object, the motion of the 3D object is achieved.

[0188] The 3D object generation system 100 can output and store the motion of the 3D object implemented above.

[0189] Furthermore, when organizing and explaining the present invention from an operational perspective, the three-dimensional object generation system 100 according to the present invention can analyze a specified sentence based on user input. Moreover, the three-dimensional object generation system 100 can define multiple actions corresponding to the sentence based on the analysis results, and define connecting actions for linking the multiple actions corresponding to the sentence. Further, the three-dimensional object generation system 100 can utilize a learned artificial neural network to generate a three-dimensional object that performs multiple actions and connecting actions.

[0190] Here, the connecting action can be, as described above, a connecting action that connects at least two actions that are adjacent to each other in a time-series flow.

[0191] Furthermore, the motion of a 3D object can be formed by the following multiple frames: multiple frames corresponding to the first action, based on key point information of the 3D model of the first action among multiple actions; multiple frames corresponding to the second action, based on key point information of the 3D model of the second action among multiple actions; and multiple frames corresponding to the connecting action, based on key point information of the 3D model of the connecting action that connects the first action and the second action.

[0192] In other words, a three-dimensional object can be understood as having the following shape: by playing multiple frames corresponding to the first action, the second action, and the connecting action, the three-dimensional model moves.

[0193] As previously mentioned, each of the multiple frames corresponding to each of the multiple actions and connection actions (e.g., multiple frames corresponding to the first action, multiple frames corresponding to the second action, and multiple frames corresponding to the connection action) may include location information of key points of the 3D model.

[0194] Furthermore, when actions that are adjacent in time sequence flow in multiple frames (e.g., the first action and the second action) are adjacent to each other in time sequence flow, key points of the 3D model corresponding to the connected actions can be defined based on key point information of the 3D model corresponding to the first action and the second action.

[0195] At this point, according to the time sequence, if the first action precedes the second action, then the positional information of the keypoints of the 3D model contained in the first frame of the multiple frames corresponding to the connected action can be correlated with the positional information of the keypoints of the 3D model contained in the last frame of the multiple frames corresponding to the first action, and the positional information of the keypoints of the 3D model contained in the last frame of the multiple frames corresponding to the connected action can be correlated with the positional information of the keypoints of the 3D model contained in the first frame of the multiple frames corresponding to the second action. Here, correlation of keypoint positional information can mean that the positional information of the keypoints corresponds to the same or adjacent positions.

[0196] For example, the key points of the 3D model contained in the first frame of the connecting action may be the same as or adjacent to the key points of the 3D model contained in the last frame of the first action, and the key points of the 3D model contained in the last frame of the connecting action may be the same as or adjacent to the key points of the 3D model contained in the first frame of the second action.

[0197] As described above, the three-dimensional object generation method and system according to the present invention can define actions corresponding to the meaning of sentences and generate three-dimensional objects that perform the defined actions, thereby generating moving three-dimensional objects simply by inputting sentences, without requiring the user to define the actions of the three-dimensional objects one by one.

[0198] Furthermore, the three-dimensional object generation method and system according to the present invention can also obtain information on various elements such as actions, expressions, appearance, age, clothing, gender, physical condition, background, location, and context based on keywords corresponding to sentences, which are set to realize a three-dimensional model corresponding to the three-dimensional object.

[0199] Furthermore, according to various embodiments of the present invention, the three-dimensional object generation method and system can naturally connect different movements performed by a three-dimensional object by generating connection action information for multiple action information with consecutive action sequences.

[0200] Furthermore, the present invention described above can be implemented as computer-readable code or instructions on a medium containing a program. That is, the various control methods according to the present invention can be provided centrally or separately in the form of programs.

[0201] In addition, computer-readable media include all types of recording devices that store data that can be read by a computer system. Examples of computer-readable media include hard disk drives (HDDs), solid-state drives (SSDs), silicon disk drives (SDDs), ROM, RAM, CD-ROMs, magnetic tapes, floppy disks, optical data storage devices, etc.

[0202] Furthermore, the computer-readable medium may include storage space, which may be a server or cloud storage space that an electronic device can access via communication. In this case, the computer can download the program according to the invention from the server or cloud storage space via wired or wireless communication.

[0203] Furthermore, in this invention, the aforementioned computer is an electronic device equipped with a processor, i.e., a central processing unit (CPU), and there are no particular restrictions on its type.

[0204] Furthermore, the specific embodiments described above should not be construed as limiting in all respects, but should be considered exemplary. The scope of the invention should be determined by a reasonable interpretation of the appended claims, and all modifications within the equivalent scope of the invention should be covered within the scope of the invention.

Claims

1. A three-dimensional object generation method, the method comprising the steps of: specifying a sentence based on a user input; analyzing the specified sentence and defining a plurality of actions corresponding to the sentence based on the analysis result; defining a connection action connecting the plurality of actions corresponding to the sentence; and generating a three-dimensional object of a motion performing the plurality of actions and the connection action using a learned artificial neural network.

2. The three-dimensional object generation method according to claim 1, wherein the connection action is a connection action connecting at least two actions of the plurality of actions that are adjacent to each other in a time series flow.

3. The three-dimensional object generation method according to claim 2, wherein the motion of the three-dimensional object is formed by a plurality of frames including: a plurality of frames corresponding to a first action of the plurality of actions including key point information of a three-dimensional model based on the first action, a plurality of frames corresponding to a second action of the plurality of actions including key point information of the three-dimensional model based on the second action, and a plurality of frames corresponding to the connection action including key point information of the three-dimensional model based on the connection action connecting the first action and the second action.

4. The three-dimensional object generation method according to claim 3, wherein each of the plurality of frames corresponding to the first action, the plurality of frames corresponding to the second action, and the plurality of frames corresponding to the connection action includes position information on a key point of the three-dimensional model.

5. The three-dimensional object generation method according to claim 4, wherein when the first action and the second action are actions that are adjacent to each other in a time series flow, the key point of the three-dimensional model of the plurality of frames corresponding to the connection action is defined based on the key point information of the three-dimensional model of the plurality of frames corresponding to the first action and the plurality of frames corresponding to the second action.

6. The three-dimensional object generation method according to claim 5, wherein when the first action precedes the second action in a time series flow, position information of a key point of the three-dimensional model included in an initial one of the plurality of frames corresponding to the connection action is related to position information of a key point of the three-dimensional model included in a final one of the plurality of frames corresponding to the first action, and position information of a key point of the three-dimensional model included in a final one of the plurality of frames corresponding to the connection action is related to position information of a key point of the three-dimensional model included in an initial one of the plurality of frames corresponding to the second action.

7. The three-dimensional object generation method according to claim 6, wherein a position of a key point of the three-dimensional model included in the initial one of the plurality of frames of the connection action is the same as or adjacent to a position of a key point of the three-dimensional model of the final one of the first action, and a position of a key point of the three-dimensional model included in the final one of the plurality of frames of the connection action is the same as or adjacent to a position of a key point of the three-dimensional model of the initial one of the second action.

8. A three-dimensional object generation system comprising: a storage unit including a database set in advance; and a control unit that specifies a sentence based on a user input, analyzes the specified sentence, defines a plurality of actions corresponding to the sentence based on an analysis result, defines a connection action that connects the plurality of actions corresponding to the sentence, and generates a three-dimensional object of a motion that performs the plurality of actions and the connection action using a learned artificial neural network. 9.A program stored in a computer-readable recording medium, the program being executed by one or more processes in an electronic device, the program comprising instructions to perform: specifying a sentence based on a user input; analyzing the specified sentence and defining a plurality of actions corresponding to the sentence based on an analysis result; defining a connection action that connects the plurality of actions corresponding to the sentence; and generating a three-dimensional object of a motion that performs the plurality of actions and the connection action using a learned artificial neural network. 10.A three-dimensional object generation method comprising: specifying a sentence based on a user input; analyzing the specified sentence and defining a scenario corresponding to the sentence based on an analysis result; acquiring a three-dimensional model and action information corresponding to the scenario using a pre-set database; and generating a three-dimensional object that performs an action corresponding to the sentence by applying the action information to the three-dimensional model. The acquiring of the three-dimensional model and action information comprises:

11. The three-dimensional object generation method according to claim 10, wherein when a scenario related to a plurality of actions is defined corresponding to the sentence, acquiring action information corresponding to each of the plurality of actions from the database and sequentially connecting the acquired action information in an action order specified based on the scenario. The sequentially connecting of the action information comprises:

12. The three-dimensional object generation method according to claim 11, wherein, generating connection action information such that a final pose of first action information of an action in the plurality of actions that has a preceding action order becomes an initial pose of second action information having a next action order. The generating of the connection action information comprises:

13. The three-dimensional object generation method according to claim 12, wherein, determining an action speed of the connection action information by considering a first action speed of the first action information and a second action speed of the second action information in the plurality of actions. The acquiring of the three-dimensional model and action information comprises:

14. The three-dimensional object generation method according to claim 10, wherein when the scenario includes a plurality of elements related to the three-dimensional object, matching each of a plurality of keywords corresponding to the scenario with one of a plurality of categories pre-set based on the plurality of elements; acquiring information corresponding to each keyword from a database based on the matched category. The specifying of the sentence comprises:

15. The three-dimensional object generation method according to claim 10, wherein data is input based on a user input; the data is analyzed and converted into text; and the sentence is specified such that a scenario is defined based on the converted text. The defining of the scenario comprises:

16. The three-dimensional object generation method according to claim 10, wherein dividing the specified sentence into a plurality of words in morphemes; confirming meanings corresponding to the divided plurality of words; and generating at least one keyword corresponding to a meaning of the specified sentence based on the confirmed meanings. The defining of the scenario comprises:

17. The three-dimensional object generation method of claim 16, wherein, confirming a relevance between a plurality of words that are different from each other based on meanings of each of the plurality of words divided from the sentence; and determining at least one of an action level and an action order of the at least one keyword based on the confirmed relevance. ​ 18. A three-dimensional object generation system, comprising: Storage unit, including a pre-configured database; as well as The control unit analyzes a specified sentence based on user input, defines a plot corresponding to the sentence based on the analysis results, obtains a three-dimensional model and action information corresponding to the plot using a pre-set database, and generates a three-dimensional object that performs the action corresponding to the sentence by applying the action information to the three-dimensional model.

19. A program stored in a computer-readable recording medium, the program being executed in an electronic device by one or more processes, the program including instructions to perform the following steps: Based on the user's input of a specified sentence; Analyze the specified sentence and define the plot corresponding to the sentence based on the analysis results; Using a pre-set database, obtain the 3D model and motion information corresponding to the plot; and By applying the action information to the three-dimensional model, a three-dimensional object that performs the action corresponding to the sentence is generated.

Images