Method and system for generating three-dimensional object on basis of semantic analysis

The system analyzes textual descriptions to generate dynamic 3D objects that perform actions, addressing the limitations of current 3D modeling technologies by enabling the creation of moving and expressive 3D models for diverse industrial applications.

WO2024228430A1PCT designated stage expired Publication Date: 2024-11-07NATIONA INC
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
PCT/KR2023/013837
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-05-04
Filing Date
2023-09-14
Publication Date
2024-11-07

AI Technical Summary

Technical Problem

Current 3D modeling technologies are limited in creating dynamic and moving objects, focusing mainly on static objects, and lack the capability to naturally perform operations based on textual descriptions, which is essential for expanding applications in various industries like AR/VR, movies, and animation.

Method used

A method and system that analyzes textual descriptions to generate 3D objects by defining operations and connecting them using a learned artificial neural network, allowing for the creation of moving 3D objects that perform actions corresponding to the meaning of the text, including motion, expression, and appearance.

Benefits of technology

Enables the automatic generation of dynamic 3D objects that can perform complex actions and movements, enhancing the scope of 3D modeling applications beyond static objects, facilitating more realistic and diverse content creation in various industries.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure KR2023013837_07112024_PF_FP_ABST
    Figure KR2023013837_07112024_PF_FP_ABST
Patent Text Reader

Abstract

The present invention relates to a method and a system for generating a three-dimensional object on the basis of semantic analysis. More specifically, the present invention relates to a method and a system for analyzing a sentence in a text format to automatically generate a three-dimensional object corresponding to the meaning of the sentence.
Need to check novelty before this filing date? Find Prior Art

Description

Method and system for generating 3D objects based on semantic analysis

[0001] The present invention relates to a method and system for generating a three-dimensional object based on semantic analysis. More specifically, the present invention relates to a method and system for automatically generating a three-dimensional object corresponding to the meaning of a textual sentence by analyzing the sentence.

[0002] As the metaverse market grows, demand for 3D object creation technology, a key element, is rapidly increasing. Domestic and international companies are focusing on the metaverse, which replicates the real world in the virtual world, as a next-generation growth engine. Interest in 3D modeling technology, a core technology of the metaverse that transforms the real world into three dimensions and expresses it in the virtual world, is also growing significantly.

[0003] In the past, 3D modeling technology was mainly used in the gaming industry, but recently, it has been expanding its application to all industrial fields, expanding its scope of application to various content industries such as AR / VR, movies, animation, and broadcasting.

[0004] However, despite the high interest in 3D modeling technology, its technical maturity remains inadequate. For example, companies developing AI humans are focusing solely on creating virtual faces using CG or deep fake methods, and are lacking in technology that recognizes human motion and reproduces it naturally.

[0005] In addition, as 3D modeling technology is being applied widely across all industrial fields, the 3D authoring tool market is also continuously growing. However, current 3D imaging authoring tools are focused only on creating static objects (architecture, interior design, facilities, etc.), and there is a lack of research and development on technologies for converting moving dynamic objects into 3D and expressing them in 3D space.

[0006] The technical challenge to be addressed by embodiments of the present invention is to provide a method and system for generating a three-dimensional object by analyzing meaning. Specifically, the present invention provides a method and system for analyzing sentences provided in text format and automatically generating a three-dimensional object corresponding to the meaning of the sentences.

[0007] To this end, the present invention can be provided to generate a three-dimensional object that performs an action corresponding to a sentence by applying an action related to the meaning of the specified sentence to a three-dimensional model when the sentence is specified.

[0008] In addition, another technical problem to be solved through embodiments of the present invention is to provide a method and system for generating a three-dimensional object that performs an entire operation corresponding to the meaning of a sentence by sequentially combining a plurality of three-dimensional objects that perform partial operations corresponding to the meaning of words in a sentence.

[0009] Furthermore, the present invention relates to a method and system for generating and applying to a three-dimensional model a connection motion between different motions so that the three-dimensional model can naturally perform a plurality of motions in relation to the meaning of a sentence.

[0010] In order to solve the problem discussed above, a method for generating a 3D object according to the present invention may include a step of specifying a sentence based on a user input, a step of analyzing the specified sentence and defining a plurality of actions corresponding to the sentence based on the analysis result, a step of defining a connection action connecting the plurality of actions corresponding to the sentence, and a step of generating a moving 3D object that performs the plurality of actions and the connection action using a learned artificial neural network.

[0011] In one example, the above-mentioned connection operation may be a connection operation that connects at least two adjacent operations according to a time series flow among the plurality of operations.

[0012] In one example, the movement of the three-dimensional object may be formed by a plurality of frames corresponding to a first operation among the plurality of operations, which include key point information of a three-dimensional model according to the first operation, a plurality of frames corresponding to a second operation among the plurality of operations, which include key point information of the three-dimensional model according to the second operation, and a plurality of frames corresponding to a connection operation, which includes key point information of the three-dimensional model according to the connection operation connecting the first operation and the second operation.

[0013] In one example, each of the plurality of frames corresponding to the first operation, the plurality of frames corresponding to the second operation, and the plurality of frames corresponding to the connection operation may include position information for key points of the three-dimensional model.

[0014] In one example, when the first action and the second action are adjacent actions according to a time series flow, the key point of the three-dimensional model of the plurality of frames corresponding to the connection action may be determined or defined based on the key point information of the three-dimensional model of the plurality of frames corresponding to the first action and the plurality of frames corresponding to the second action.

[0015] In one example, based on the time series flow, if the first operation is an operation preceding the second operation, the position information of the key point of the 3D model included in the first frame among the plurality of frames corresponding to the connection operation may be related to the position information of the key point of the 3D model included in the last frame among the plurality of frames corresponding to the first operation, and the position information of the key point of the 3D model included in the last frame among the plurality of frames corresponding to the connection operation may be related to the position information of the key point of the 3D model included in the first frame among the plurality of frames corresponding to the second operation.

[0016] In one example, the position of the key point of the 3D model included in the first frame of the connection operation may be the same as or adjacent to the position of the key point of the 3D model included in the last frame of the first operation, and the position of the key point of the 3D model included in the last frame of the connection operation may be the same as or adjacent to the position of the key point of the 3D model included in the first frame of the second operation.

[0017] A three-dimensional object generation system according to the present invention may include a storage unit including a pre-provisioned database; and a control unit that specifies a sentence based on a user input, analyzes the specified sentence, defines a plurality of actions corresponding to the sentence based on the analysis result, defines a connection action connecting the plurality of actions corresponding to the sentence, and generates a moving three-dimensional object that performs the plurality of actions and the connection action using a learned artificial neural network.

[0018] A program stored in a computer-readable recording medium, which is executed by one or more processes in an electronic device, may include instructions for performing the steps of: specifying a sentence based on a user input; analyzing the specified sentence and defining a plurality of actions corresponding to the sentence based on a result of the analysis; defining a connection action connecting the plurality of actions corresponding to the sentence; and generating a moving three-dimensional object that performs the plurality of actions and the connection action using a learned artificial neural network.

[0019] A method for generating a three-dimensional object according to the present invention may include: a step of specifying a sentence based on a user input; a step of analyzing the specific sentence and defining a storyboard corresponding to the sentence based on the analysis result; a step of obtaining a three-dimensional model and motion information corresponding to the storyboard using a pre-prepared database; and a step of applying the motion information to the three-dimensional model to generate a three-dimensional object that performs a motion corresponding to the sentence.

[0020] In addition, a three-dimensional object generation system according to the present invention may include a storage unit including a pre-prepared database; and a control unit that specifies a sentence based on a user input, analyzes the specific sentence, defines a storyboard corresponding to the sentence based on the analysis result, obtains a three-dimensional model and motion information corresponding to the storyboard using the pre-prepared database, and applies the motion information to the three-dimensional model to generate a three-dimensional object that performs a motion corresponding to the sentence.

[0021] In addition, a program stored in a computer-readable recording medium according to the present invention is a program stored in a computer-readable recording medium, which is executed by one or more processes in an electronic device, and the program may include commands to perform the steps of: specifying a sentence based on a user input; analyzing the specific sentence and defining a conti corresponding to the sentence based on the analysis result; obtaining a three-dimensional model and motion information corresponding to the conti using a pre-prepared database; and applying the motion information to the three-dimensional model to create a three-dimensional object that performs a motion corresponding to the sentence.

[0022] The method and system for generating a 3D object according to the present invention defines actions corresponding to the meaning of a sentence and generates a 3D object that performs the defined actions, thereby making it possible to generate a 3D object that moves by simply inputting a sentence, without the user having to define the actions of the 3D object one by one.

[0023] Furthermore, the method and system for generating a 3D object according to the present invention can further acquire information designed to implement various elements such as movement, expression, appearance, age, clothing, gender, physical condition, background, location, and situation of a 3D model corresponding to a 3D object based on keywords specified in response to a sentence.

[0024] In addition, according to various embodiments of the present invention, the method and system for generating a three-dimensional object can naturally connect different motions performed by a three-dimensional object by generating connection motion information for a plurality of motion information pieces in which the motion sequence is continuous.

[0025] Figure 1 illustrates a three-dimensional object generation system according to the present invention.

[0026] FIG. 2 illustrates an embodiment of a database included in a three-dimensional object generation system according to the present invention.

[0027] Figures 3 and 4 are flowcharts showing a method for creating a three-dimensional object according to the present invention.

[0028] Figure 5 illustrates an embodiment of specifying a sentence based on user input.

[0029] Figures 6 to 9 illustrate an embodiment of defining an action corresponding to a sentence.

[0030] Figures 10 to 14 illustrate an embodiment of applying motion information to a three-dimensional model.

[0031] Hereinafter, embodiments disclosed in this specification will be described in detail with reference to the attached drawings. Regardless of the drawing numbers, identical or similar components will be given the same reference numbers, and redundant descriptions thereof will be omitted. The suffixes "module" and "part" used for components in the following description are assigned or used interchangeably only for the convenience of writing the specification, and do not in themselves have distinct meanings or roles. In addition, when describing the embodiments disclosed in this specification, if it is determined that a specific description of a related known technology may obscure the gist of the embodiments disclosed in this specification, a detailed description thereof will be omitted. In addition, the attached drawings are only intended to facilitate easy understanding of the embodiments disclosed in this specification, and the technical ideas disclosed in this specification are not limited by the attached drawings, and should be understood to include all modifications, equivalents, and substitutes included in the spirit and technical scope of the present invention.

[0032] Terms that include ordinal numbers, such as first, second, etc., may be used to describe various components, but the components are not limited by these terms. These terms are used solely to distinguish one component from another.

[0033] When a component is referred to as being "connected" or "connected" to another component, it should be understood that it may be directly connected or connected to that other component, but that there may be other components intervening. Conversely, when a component is referred to as being "directly connected" or "connected" to another component, it should be understood that there are no other components intervening.

[0034] Singular expressions include plural expressions unless the context clearly indicates otherwise.

[0035] In this application, terms such as “include” or “have” are intended to specify the presence of a feature, number, step, operation, component, part or combination thereof described in the specification, but should be understood not to exclude in advance the possibility of the presence or addition of one or more other features, numbers, steps, operations, components, parts or combinations thereof.

[0036] Fig. 1 illustrates a three-dimensional object generation system according to the present invention. Fig. 2 illustrates one embodiment of a database included in the three-dimensional object generation system according to the present invention.

[0037] Referring to FIG. 1, a three-dimensional object generation system (100) according to the present invention can specify a sentence based on a user input (10), analyze the specified sentence, and define (or determine, create, or extract) a continuity corresponding to the sentence. The three-dimensional object generation system (100) can define a continuity corresponding to a sentence, define an action, or define a three-dimensional model based on learning described later. In the present invention, a continuity is information that defines various matters such as an action that a three-dimensional model must perform, the appearance, size, expression, and gender of the three-dimensional model in order to express the meaning or situation corresponding to a sentence, and can also be understood as a "storyboard." In the present invention, "definition" can be understood to determine, create, or extract information or data.

[0038] The 3D object generation system (100) can obtain a 3D model (1) and motion information corresponding to a sentence from a pre-prepared database (131) based on a defined storyboard, and apply the motion information to the 3D model (1) to generate a 3D object (3) that performs a motion according to the sentence. In the present invention, the 3D model is a subject (e.g., a person, an animal, an object, etc.) that performs a motion corresponding to a sentence, and a dynamic object in which the 3D model moves and performs a motion is called a 3D object. The 3D object described in the present invention may be modified and named as a 3D dynamic object, a 3D graphic object, a 3D graphic dynamic object, a 3D modeling object, a 3D animation object, or an object, a dynamic object, a graphic object, a graphic dynamic object, a modeling object, an animation object, etc. Meanwhile, the performing subject of the present invention in the following description is called a 3D object generation system (100), but this may also be described as a "control unit" instead.

[0039] In the present invention, the storyboard may include information defining at least one of the persona of the 3D model and the movement of the 3D model. That is, the 3D object generation system according to the present invention can define, generate, or extract information regarding at least one of the persona of the 3D model and the movement of the 3D model from a sentence based on a learned artificial intelligence model.

[0040] At this time, the persona of the 3D model may include information defining the visual appearance, personality, etc. of the 3D model.

[0041] The visual appearance of a 3D model may include at least some of various information that can define the visual appearance of the 3D model, such as the 3D model's age, gender, height, weight, hairstyle, face shape, skin color, and clothing. Furthermore, the persona may include at least some of various information that can influence the 3D model's behavioral characteristics, such as the 3D model's occupation and personality. The 3D model may be defined in various ways, such as a person, animal, or object, depending on a specific sentence.

[0042] Meanwhile, the 3D object creation system (100) can define the conti based on a specific sentence input and other sentences related to the specific sentence (or other information, for example, information included in a paragraph including the specific sentence, sentences (or information) input before or after the specific sentence is input).

[0043] Meanwhile, persona information for defining a 3D model may be defined based on separate user input or settings, rather than being extracted from a sentence. In this case, the 3D object generation system according to the present invention may be equipped with a separate process for generating or defining a 3D model that performs an action according to the sentence. In this case, the 3D object generation system (100) may specify at least one action to be performed by the 3D model from the sentence, and generate a 3D object in which the 3D model performs the specified action.

[0044] Meanwhile, in the present invention, it may also be understood that, rather than "defining a storyboard" from a sentence, at least one "action" to be performed by a 3D model is defined from a sentence.

[0045] A 3D object creation system (100) receives various forms of user input (10), such as voice signals, images, and texts, and can specify a sentence to be applied to a 3D model (1) based on the received user input (10).

[0046] For example, if a sentence such as 'a person walks' is specified based on a user input (10), the 3D object generation system (100) can define the action of 'walking' based on the analysis result of the sentence, and obtain action information related to 'walking' from the database (131). Accordingly, the 3D object generation system (100) can generate and output a 3D object (3) that performs a walking action by applying the acquired action information to the 3D model (1). At this time, when the 3D model (1) is defined as a person, the 3D object generation system (100) can generate and output a 3D object (3) in the form of a person that performs a walking action by applying the acquired action information to the 3D model (1) provided in the shape of a person. Hereinafter, for the convenience of explanation, the 3D model will be described as an example of a "person"; however, the 3D model in the present invention is not necessarily limited to a person.

[0047] Meanwhile, in the present specification, a sentence can be specified based on a user input (10), and the sentence can be specified in the form of text describing at least one of an action and an appearance of a three-dimensional object (3).

[0048] Additionally, the three-dimensional object (3) may be created to be modeled in three-dimensional space, and may be created to have various shapes such as people, animals, and objects.

[0049] For example, a 3D object (3) may include at least one of a plurality of key points, a line connecting two different key points, a mesh representing the shape (or volume) of the 3D object (3), and a texture (or image) representing the texture and appearance of the 3D object (3).

[0050] At this time, the three-dimensional model (1) may be arranged to constitute a three-dimensional object (3), that is, the three-dimensional model (1) may form a skeleton (or, skeleton) of the three-dimensional object (3). To this end, the three-dimensional model (1) may be configured to correspond to at least one of a plurality of key points, a line connecting two different key points, and a mesh representing the shape (or, volume) of the three-dimensional object (3) among the three-dimensional object (3). According to an embodiment, the three-dimensional object (3) may be generated by overlaying a texture onto the three-dimensional model (1).

[0051] Meanwhile, multiple key points can be positioned corresponding to joint parts of a 3D model (1) (or a 3D object (3)).

[0052] Furthermore, the 3D model (1) may further include label data for each key point, such as information for distinguishing each key point, an available range of movement for the location of each key point, and an available range of speed for changes in each key point. Through this, the 3D model (1) can be arranged to implement natural movements.

[0053] Meanwhile, the storyboard may include information for various categories (or categories), such as information for defining the movements of the 3D model discussed above, information for defining the persona, information for defining the situation in which the 3D model performs the movements, and information for defining the background in which the 3D model performs the movements. For example, the storyboard may include information (or definitions) for various categories (or categories), such as the movements of the 3D model, the appearance, age, clothing, gender, physical condition of the 3D model, the background, place, and situation in which the 3D model performs the movements. In other words, the storyboard may include (or define) information related to the shape of the 3D model to be acquired in response to the sentence specified above, the movements to be applied to the 3D model, and the place, background, and situation in which the 3D model will be placed.

[0054] For example, in response to the sentence 'The cat jumps over the fence', a 3D object generation system (100) may define a storyboard so that a 3D model having a shape corresponding to a cat performs an action of jumping over a fence in a place where the fence is placed. Specifically, 'cat' may be included in the storyboard as a definition for a category related to the appearance of the 3D model, 'fence' may be included in the storyboard as a definition for a category related to a background on which the 3D model will be placed, and 'jump' may be included in the storyboard as a definition for a category related to an action to be applied to the 3D model.

[0055] Meanwhile, the motion information may include information generated so that the 3D model (1) moves according to a motion based on a sentence. Specifically, the motion information may include information related to the movement of each of one or more key points included in the 3D model (1). That is, the motion information may include a plurality of postures or a plurality of frames (or times), and each posture or each frame may include position information of each of the plurality of key points included in the 3D model (1). At this time, the motion information may specify the positions of each of the plurality of key points based on a point in the 3D space.

[0056] For example, the motion information may include information related to position changes and change speeds for each of a plurality of key points included in the three-dimensional model (1).

[0057] As another example, the motion information may be generated such that position information for each of a plurality of key points included in the three-dimensional model (1) is specified at a preset time period (e.g., 1 / 60 second) for a series of time intervals corresponding to the motion information.

[0058] Additionally, the motion information may include expression information corresponding to an expression category generated so that the facial area of ​​the 3D model (1) moves according to an expression based on a sentence. At this time, the expression information may include information related to the movement of each key point belonging to the facial area of ​​the 3D model (1), and thus may be arranged to be included in the motion information.

[0059] Also, referring to FIG. 2, the database (131) may store a plurality of different three-dimensional models (1) and a plurality of motion information (20). Here, the plurality of different three-dimensional models (1) may refer to a plurality of three-dimensional models (1) prepared to have different appearances. For example, the plurality of different three-dimensional models (1) may include objects having various appearances, such as a cat, a person, a boy, a girl, and a dog. At this time, the plurality of different three-dimensional models (1) may have the number and positions of the plurality of key points, the shape and texture of the mesh, etc., set differently according to the appearance of each object.

[0060] The database (131) may further store labeling information (60) (or tag, keyword) matching each of a plurality of three-dimensional models (1) and a plurality of motion information (20).

[0061] Labeling information (60) can be set to correspond to the analysis results for a sentence. That is, the labeling information (60) is a keyword for a storyboard defined according to a sentence, and can indicate elements such as movement according to the storyboard and the appearance of a 3D model. That is, the 3D object generation system (100) can define a storyboard corresponding to a sentence by obtaining keywords from the sentence. At this time, the 3D object generation system (100) can also classify keywords by category according to the storyboard.

[0062] For example, the labeling information (60) may include words such as "person" and "walk" in relation to the motion of a person walking. As another example, the labeling information (60) may further include objects, complements, and adverbs in relation to motions according to the subject and predicate included in the sentence.

[0063] Through this, the 3D object generation system (100) can obtain a 3D model (1) and motion information corresponding to a sentence from a database (131) using a conti (e.g., a keyword or word) defined based on the analysis result for the sentence. In particular, the 3D object generation system (100) can obtain a 3D model (1) with a more accurate and diverse appearance and motion information (20) related to motion by considering various sentence components such as objects, complements, and adverbs in addition to subjects and predicates included in the sentence through morphological analysis.

[0064] In this regard, when a conti defined from a sentence is input, the database (131) may store a three-dimensional model (1) corresponding to the sentence and an artificial neural network model that outputs motion information. In this case, the three-dimensional object generation system (100) according to the present invention inputs a conti defined based on the analysis result for the sentence into the artificial neural network model provided in the database (131), and can obtain a three-dimensional model (1) corresponding to the sentence and motion information from the artificial neural network model.

[0065] To this end, the artificial neural network model can be trained using a learning conti and a correct 3D model and correct motion information labeled as label data in the learning conti, wherein the 3D model and the motion information may include various types of information described above. Accordingly, the artificial neural network model can be trained using the learning conti corresponding to the various categories described above and the correct motion information so that when a conti is input, a 3D model (1) and motion information corresponding to the input conti are output.

[0066] In this regard, the overall operation of the 3D object generation system (100) according to the present invention will be described below through a method of searching for a 3D model and motion information corresponding to a conti in a database (131). However, it should be considered that the 3D model and motion information in the present invention may also be generated through an artificial neural network model as described above.

[0067] Additionally, the database (131) may store data classified by the various categories mentioned above. In this case, the 3D object creation system (100) can match the conti corresponding to the sentences by category and obtain data corresponding to the matched categories from the database (131).

[0068] According to an embodiment, the database (131) may store multiple artificial neural network models for each of the above-described various categories, and may acquire data corresponding to each category.

[0069] To this end, the three-dimensional object generation system (100) according to the present invention may include an input unit (110), a storage unit (130), a control unit (150), and an output unit (170). Meanwhile, FIG. 1 only illustrates components related to an embodiment of the present invention. Therefore, a person skilled in the art to which the present invention pertains will understand that other general components may be included in addition to the components illustrated in FIG. 1.

[0070] The input unit (110) can receive user input (10) through a pre-provided input device. For example, the input device may refer to various devices such as a keyboard, a mouse, a touch pad, a microphone (MIC), and a line input (Line-IN). Accordingly, the input unit (110) can receive user input (10) corresponding to various types such as voice and text. Through this, the control unit (150) can obtain a sentence corresponding to the user input (10).

[0071] The storage unit (130) may store information and commands necessary for the operation of the three-dimensional object generation system (100) according to the present invention. Accordingly, the storage unit (130) may load one or more programs to execute methods / operations according to various embodiments of the present invention. An example of the storage unit (130) may be RAM, but is not limited thereto.

[0072] Meanwhile, the storage unit (130) may store a database containing a plurality of pieces of motion information. In addition, the storage unit (130) (or database) may store a three-dimensional model (1) to which motion information is applied.

[0073] The control unit (150) can control the overall operation of the three-dimensional object generation system (100) according to the present invention. To this end, the control unit (150) can be configured to include at least one of a CPU (Central Processing Unit), an MPU (Micro Processor Unit), an MCU (Micro Controller Unit), a GPU (Graphics Processing Unit), or any other type of processor well known in the technical field of the present invention. In addition, the control unit (150) can perform operations for at least one application or program for executing methods / operations according to various embodiments of the present invention. The three-dimensional object generation system (100) may be equipped with one or more control units (150).

[0074] Meanwhile, the control unit (150) can analyze a sentence corresponding to a user input (10) to define a conti corresponding to the sentence from a database (131), obtain a 3D model and motion information based on the conti, and apply the motion information to the 3D model (1) to create a 3D object (3).

[0075] The output unit (170) can output a three-dimensional object (3) generated by the control unit (150). At this time, the output unit (170) can output the three-dimensional object (3) so that a user can visually confirm it through a display device such as a monitor or television.

[0076] Based on the configuration of the 3D object creation system (100) discussed above, the 3D object creation method will be described in more detail below.

[0077] Figures 3 and 4 are flowcharts illustrating a method for creating a three-dimensional object according to the present invention. Figure 5 illustrates an embodiment of specifying a sentence based on user input. Figures 6 to 9 illustrate an embodiment of defining an action corresponding to a sentence. Figures 10 to 14 illustrate an embodiment of applying action information to a three-dimensional model.

[0078] Referring to FIG. 3, the 3D object creation system (100) according to the present invention can specify a sentence based on user input (10) (S100), analyze the specific sentence, and define a conti corresponding to the sentence based on the analysis result (S200).

[0079] Specifically, the 3D object generation system (100) can specify a sentence according to the input text when the data input according to the user input (10) is text. In addition, the 3D object generation system (100) can specify a sentence so that, when the data input according to the user input (10) is in a form other than text, the data is analyzed and converted into text, and an operation to be applied to the 3D model (1) is defined based on the converted text.

[0080] Referring to FIG. 5, for example, when text (10a) is input based on user input (10), the 3D object generation system (100) can specify the input text (10a) as a sentence.

[0081] As another example, when data including a plurality of texts is input based on a user input (10), the 3D object generation system (100) performs an analysis on the input data to extract a plurality of texts, segments the extracted plurality of texts, and sequentially specifies each of the segmented texts according to the order included in the data as a sentence for generating a 3D object (3).

[0082] To this end, the 3D object generation system (100) can segment a plurality of texts based on texts corresponding to preset punctuation marks (e.g., periods) among the texts included in the data. Accordingly, the 3D object generation system (100) can sequentially analyze the segmented plurality of texts according to the order included in the data and define a storyboard corresponding to each sentence.

[0083] As another example, when a voice signal (10b) is input based on a user input (10), the 3D object generation system (100) can generate text (or sentence) (11a) corresponding to the user input (10) through analysis of the voice signal (10b).

[0084] At this time, the 3D object generation system (100) may use a technique (e.g., Speech-to-Text) to convert a voice signal (10b) corresponding to a user input (10) into text (11a), and may include a voice model or an artificial neural network for this purpose.

[0085] As another example, when an image is input based on a user input (10), the 3D object generation system (100) can generate text corresponding to the user input (10) through analysis of the image.

[0086] At this time, the 3D object generation system (100) may use a technique (e.g., optical character recognition (OCR)) to convert an image corresponding to a user input (10) into text, and may include a character detection model, a character recognition model, and an artificial neural network for this purpose.

[0087] As another example, when data generated according to a preset format (e.g., a script format) based on user input (10) is input, the 3D object generation system (100) extracts one or more texts from a preset area to indicate the appearance, background, and movement of the object in response to the format, and sequentially specifies the one or more extracted texts as sentences for generating a 3D object (3).

[0088] At this time, the 3D object generation system (100) can extract one or more texts from the area (11b, 11c parentheses, 11d parentheses, 11e parentheses) corresponding to the fingerprint (Action) in the script if the preset form is a script (or script) form. Accordingly, the 3D object generation system (100) can sequentially specify each text according to the order included in the data as a sentence for generating a 3D object (3).

[0089] Furthermore, the 3D object generation system (100) can divide a sentence into multiple words in morpheme units, confirm the meaning corresponding to the divided multiple words, and generate (or extract) at least one keyword corresponding to the meaning of the sentence specified above based on the confirmed meaning. In this case, the 3D object generation system (100) generating at least one keyword corresponding to the meaning of the sentence may be defining a storyboard corresponding to the sentence.

[0090] For example, the 3D object generation system (100) can define a conti corresponding to a sentence by using an artificial neural network (e.g., RNN (Recurrent Neural Network) and LSTM (Long Short Term Memory)) trained to output keywords related to the conti corresponding to the plurality of input words when a plurality of words included in a sentence are sequentially input.

[0091] In this case, the 3D object generation system (100) can input the sentence specified above into a pre-learned artificial neural network, obtain a keyword related to the conti corresponding to the sentence, and generate a 3D object that performs an action corresponding to the keyword.

[0092] As another example, the 3D object generation system (100) can extract words corresponding to parts of speech related to the object's motion as keywords by identifying the part of speech of each word included in a sentence.

[0093] At this time, the 3D object generation system (100) may use an artificial neural network that has been trained in advance to identify the part of speech of a word, or may identify the part of speech of a word through a language model prepared in advance.

[0094] Through the above configurations, the 3D object generation system (100) according to the present invention can generate keywords related to a conti in a specific sentence based on a user input (10).

[0095] For example, referring to FIG. 6, if a sentence such as 'He sat on the floor and then got up' is specified, the 3D object generation system (100) can obtain 'that', 'sit', 'get up', and 'floor' as keywords related to the storyboard through morphological analysis of the specified sentence. At this time, the 3D object generation system (100) can define the appearance of a 3D model corresponding to 'that', define the action of sitting on the floor corresponding to 'sit' and 'floor', and define the action of getting up from the floor corresponding to 'that', 'get up', and 'floor', according to the sentence structure of the obtained keywords.

[0096] For another example, if a sentence such as 'sit down with your legs crossed and then stand up' is specified, the 3D object generation system (100) can define the sitting and standing up actions as a conti corresponding to the sentence based on the analysis result for the specified sentence, and accordingly, can generate keywords related to 'sit down' and 'stand up'.

[0097] Furthermore, the 3D object generation system (100) can confirm the correlation between different words based on the meaning of each of the words distinguished from the sentence, and determine at least one of the action level and the action order for at least one keyword related to the sentence based on the confirmed correlation.

[0098] Here, the action level can represent the level for a part (e.g., an arm, a leg, a person) that performs multiple different actions at the same time for one three-dimensional model (1).

[0099] Additionally, the action sequence can indicate the order in which multiple different actions are performed for a single 3D model (1). In other words, the action sequence can indicate the temporal order in which a single 3D object performs actions.

[0100] For example, the 3D object generation system (100) can generate keywords such as 'bridge', 'cross', and 'sit' in response to the sentence 'sit with legs crossed'. At this time, the 3D object generation system (100) can confirm that the object of 'cross' is 'bridge' based on the case particle '-reul' for 'bridge'. Therefore, the 3D object generation system (100) can determine that 'bridge' and 'cross' have the same level of motion.

[0101] In addition, the 3D object generation system (100) can confirm that 'sit' and 'kkoda' are performed at the same point in time based on the conjunction particle '-go' for 'kkoda', and can determine that 'kkoda', which is an action for 'legs', has a lower action level than 'sit' based on the action of 'sit' being performed by an object.

[0102] For another example, the 3D object generation system (100) can generate keywords such as "sit" and "stand up" in response to the sentence "sit down and then stand up." At this time, the 3D object generation system (100) can determine that "stand up" is executed after "sit down" based on the auxiliary verb "-daga" of "sit down." Accordingly, the 3D object generation system (100) can determine that "sit down" has a precedent action order over "stand up."

[0103] Furthermore, when the 3D object creation system (100) includes various elements related to the 3D object in the conti defined above, it can match each of the plurality of keywords corresponding to the conti to one of the plurality of categories (e.g., movement, expression, appearance, age, clothing, gender, physical condition, background, place, situation, etc.) preset according to the various elements, and obtain information corresponding to each keyword from the database based on the matched category.

[0104] For example, the 3D object generation system (100) can match at least one of a plurality of keywords corresponding to a storyboard to a category related to an expression so that an expression is applied to the facial area of ​​the 3D model (1). At this time, the 3D object generation system (100) can match a keyword related to a word corresponding to an emotion (or sentiment) among a plurality of words included in a sentence to a category related to an expression corresponding to the sentence.

[0105] In one embodiment, the 3D object generation system (100) may generate keywords such as "fun" and "run" in response to the sentence "run happily." At this time, the 3D object generation system (100) may match the keyword "fun," which corresponds to an emotion, to a category related to facial expressions, and may match the keyword "run," which corresponds to an action, to a category related to the action.

[0106] Furthermore, the 3D object generation system (100) may match a keyword related to the appearance of a 3D object (3) (or a 3D model (1)) among a plurality of words included in a sentence to a category related to the appearance. At this time, the 3D object generation system (100) may match a keyword related to the appearance of the 3D object (3) with a keyword related to an action. Through this, the 3D object generation system (100) may obtain information (e.g., 3D model and action information) corresponding to each category from a database so that a 3D object (3) having an appearance corresponding to the sentence performs an action corresponding to the sentence.

[0107] For example, the 3D object generation system (100) can generate the keyword 'person' as a keyword related to the appearance of the 3D model (1) in response to the sentence 'person is running'. At this time, the 3D object generation system (100) can match the keyword related to the action 'run' with the keyword 'person', which is a keyword related to the appearance of the 3D model (1).

[0108] As another example, the 3D object generation system (100) can generate the keyword 'cat' as a keyword related to the appearance of the 3D model (1) in response to the sentence 'the cat jumps over the fence.' At this time, the 3D object generation system (100) can match keywords related to the actions 'fence' and 'jump over' (or 'run' and 'jump over') to the keyword 'cat', which is a keyword related to the appearance of the 3D model (1).

[0109] As another example, the 3D object generation system (100) can generate keywords such as 'boy' and 'girl', which are related to the appearance of the 3D model (1), in response to the sentence 'A boy and a girl are standing under an umbrella.' At this time, the 3D object generation system (100) can match keywords related to the actions of 'umbrella,' 'under,' and 'stand,' to the keywords 'boy' and 'girl,' which are related to the appearance of the 3D model (1).

[0110] In this way, the 3D object generation system (100) can determine the movement and appearance of a 3D object in response to keywords generated through semantic analysis of a sentence, and further, can generate a 3D object in response to various keywords related to age, clothing, gender, physical condition (e.g., weight, height, etc.) of the 3D object.

[0111] Furthermore, if the 3D object creation system (100) generates keywords for the space where a 3D object is to be created (e.g., location, background, appearance of an object placed in the space, situation, etc.) through semantic analysis of a sentence, it can also generate other 3D objects adjacent to the 3D object to correspond to the keywords.

[0112] To this end, data corresponding to various categories such as movement, appearance, expression, age, clothing, gender, and physical condition can be stored in the database included in the 3D object creation system (100).

[0113] Through the above configurations, the 3D object creation system (100) according to the present invention can define at least one action most suitable for the sentence by analyzing a specific sentence based on user input (10).

[0114] Meanwhile, the 3D object generation system (100) may define multiple situations (or multiple storyboards or multiple actions) as a result of analyzing the sentence specified above. In this case, each of the multiple situations may be implemented in different spaces at the same time, or may be implemented in the same space at different times.

[0115] Accordingly, the 3D object creation system (100) can use a database to obtain 3D models and motion information corresponding to each situation, and create a 3D object to perform motions corresponding to each situation by applying the motion information to the 3D model.

[0116] According to an embodiment, when multiple different situations are applied to one 3D model, the 3D object generation system (100) may generate a 3D object so that the 3D object performs an action corresponding to a previous situation in the order of the multiple situations according to the sentence analysis results, and then performs an action corresponding to the next situation after a preset time interval.

[0117] Additionally, the 3D object generation system (100) may define a conditional conti for a specific situation as a result of analysis of the sentence specified above. The conditional conti may be set so that the 3D object performs an action corresponding to the conditional conti when a condition set based on the sentence analysis result (or user input) is satisfied.

[0118] For example, a conditional storyboard may be defined to perform a specific action (e.g., stretching) at a specific time (e.g., 6:00 AM). In this case, a 3D object generated based on the conditional storyboard may be implemented to perform the specific action at the specific time depending on the situation.

[0119] For another example, a conditional storyboard may be defined to perform a specific action (e.g., a greeting action) in response to a preset user input (e.g., a click on a 3D object). In such a case, a 3D object generated based on the conditional storyboard may be implemented to perform the specific action in response to the user input.

[0120] To this end, the 3D object creation system (100) obtains a 3D model and motion information corresponding to a conditional conti from a database, and creates a 3D object by applying the motion information to the 3D model, and can set an event flag for a condition corresponding to the conditional conti for the motion information applied to the 3D model.

[0121] Through this, the 3D object generation system (100) can generate a 3D object that performs an action in response to a specific condition, and can also generate a single 3D object to which multiple conditional conti are applied according to different conditions.

[0122] In addition, the 3D object generation system (100) may directly define at least one action as a result of the analysis of the sentence specified above. In this case, the 3D object generation system (100) may obtain action information corresponding to at least one action defined above from a database and apply the action information to a pre-prepared 3D model to generate a 3D object that performs an action corresponding to the sentence.

[0123] That is, the 3D object creation system (100) can create a 3D object by creating at least one keyword based on a sentence, obtaining at least one of a 3D model and motion information corresponding to the at least one keyword from a database, and applying the motion information to the 3D model.

[0124] Referring again to FIG. 3, the 3D object creation system (100) according to the present invention can obtain a 3D model and motion information corresponding to the conti using a pre-prepared database (131) (S300).

[0125] Specifically, the 3D object generation system (100) can search for 3D models and motion information corresponding to at least one of the keywords generated above in the database (131). Accordingly, the 3D object generation system (100) can obtain the searched 3D models and motion information.

[0126] For example, when a keyword such as 'run' is generated, the 3D object generation system (100) can obtain motion information corresponding to the running motion by searching for motion information corresponding to 'run' in the database (131).

[0127] Referring to FIG. 7, as another example, the 3D object generation system (100) can search for a 3D model corresponding to 'that' in the database (131) by defining the appearance of a 3D model related to 'that' as a conti for the sentence 'He sat on the floor and then got up.' In addition, the 3D object generation system (100) can search for motion information corresponding to the first motion and the second motion by defining a first motion related to 'sit' and 'the floor' and a second motion related to 'that', 'get up', and 'the floor'.

[0128] Accordingly, the 3D object creation system (100) can obtain a 3D model related to a person, motion information (21) related to a motion of sitting on the floor, and motion information (22) related to a motion of getting up on the floor, from the database (131).

[0129] Referring to FIG. 8, as another example, the 3D object generation system (100) can search for motion information corresponding to 'sit' in the database (131) based on the motion levels set for the keywords when keywords such as 'leg', 'cross', and 'sit' are generated. At this time, the 3D object generation system (100) can check motion information for various types of sitting motions, such as a motion of sitting with legs crossed (21a), a motion of sitting with legs crossed (21b), and a motion of sitting with legs together (21c), corresponding to 'sit' from the database (131).

[0130] Accordingly, the 3D object generation system (100) can identify the motion information corresponding to 'leg' and 'cross' based on the motion level among the plurality of motion information searched in response to 'sit', and can obtain the identified motion information. In this way, when keywords divided into a plurality of motion levels are created, the 3D object generation system (100) can perform a search on the database (131) considering the motion level, thereby obtaining the motion information most suitable for the plurality of keywords.

[0131] As another example with reference to FIG. 9, when keywords such as 'sit' and 'stand up' are generated, the 3D object generation system (100) can search for motion information (21) corresponding to 'sit' in the database (131) based on the motion order set for the keywords and obtain the searched motion information (21). Thereafter, the 3D object generation system (100) can search for motion information (22) corresponding to 'stand up' in the database (131) based on the motion order and obtain the searched motion information (22).

[0132] Accordingly, the 3D object creation system (100) can obtain multiple pieces of motion information corresponding to a sentence by sequentially listing motion information (21) related to 'sit down' and motion information (22) related to 'stand up'.

[0133] For another example, when keywords such as 'fun' and 'run' are generated, the 3D object generation system (100) can search and acquire facial expression information corresponding to 'fun' from the database (131) and can search and acquire motion information corresponding to 'run'.

[0134] \*129Furthermore, when a keyword related to the appearance of a 3D model (1) is generated in response to a sentence, the 3D object generation system (100) can obtain a 3D model (1) corresponding to the keyword related to the appearance from the database (131).

[0135] For example, when keywords such as 'person' and 'run' are generated, the 3D object generation system (100) can obtain a 3D model corresponding to the keyword related to 'person' and motion information corresponding to the keyword 'run' from the database (131).

[0136] For another example, when keywords such as 'cat', 'fence', and 'jump over' (or 'jump' and 'jump over') are generated, the 3D object generation system (100) can obtain a 3D model corresponding to the keyword related to 'cat' and motion information corresponding to the keywords 'fence' and 'jump over' (or 'jump' and 'jump over') from the database (131).

[0137] As another example, when keywords such as 'boy', 'girl', 'umbrella', 'below', and 'stand' are generated, the 3D object generation system (100) can obtain a 3D model corresponding to the keywords related to 'boy' and 'girl' from the database (131), and obtain motion information corresponding to the keywords 'umbrella', 'below', and 'stand'. At this time, the 3D object generation system (100) can also be implemented to obtain motion information corresponding to the keywords 'umbrella', 'below', and 'stand' among the motion information related to 'boy' and 'girl' from the database (131).

[0138] Furthermore, when a conti related to a plurality of actions is defined in response to a sentence, the 3D object creation system (100) can obtain action information corresponding to each of the plurality of actions from the database (131) and sequentially connect the obtained action information according to the action order specified based on the conti.

[0139] Referring to FIG. 4, when the 3D object creation system (100) obtains multiple motion information from the database (131) (S310), it can sequentially connect the multiple motion information obtained above according to a motion order specified based on the analysis result for the sentence (S320).

[0140] To this end, the 3D object creation system (100) can sequentially connect multiple different pieces of motion information by connecting the last frame of the preceding motion information and the first (very first) frame of the next motion information for two pieces of motion information that are adjacent in motion order among the multiple pieces of motion information obtained from the database (131).

[0141] At this time, the 3D object creation system (100) can create connection motion information for two pieces of motion information with adjacent motion sequences so that motions according to multiple pieces of motion information obtained from the database (131) are naturally connected (S330).

[0142] Linked motion information is a type of motion information, and may include information related to motions performed between preceding and succeeding motions so that two different motions can be naturally connected.

[0143] To this end, the 3D object generation system (100) can generate connection motion information so that, among the two motion information acquired from the database (131), the final posture of the first motion information, which is earlier in the motion order, changes to the initial posture (or first frame) of the second motion information, which is the next motion order.

[0144] Here, the final posture of the first motion information may be the posture in the last frame among the plurality of frames (or times) included in the first motion information, and the initial posture of the second motion information may be the posture in the first frame among the plurality of frames (or times) included in the second motion information.

[0145] Referring to FIG. 10, for example, the 3D object generation system (100) can generate connection motion information (50) so that the positions of a plurality of key points (40a) in the final posture (31) of the first motion information (21) are moved to the positions of a plurality of key points (40b) in the initial posture (32) of the second motion information (22).

[0146] At this time, the 3D object generation system (100) can generate connection motion information (50) so that each of the plurality of key points (40a) corresponding to the final posture of the first motion information changes its position to a corresponding key point among the plurality of key points (40b) corresponding to the initial posture of the second motion information.

[0147] As another example, the 3D object generation system (100) may generate a plurality of first key points for a first posture to be included in the linked motion information by using the positions of a plurality of key points corresponding to a final posture of the first motion information and the center points of the positions of a plurality of key points corresponding to an initial posture of the second motion information, generate a plurality of second key points for a second posture to be included in the linked motion information by using the positions of a plurality of key points of the first motion information and the center points of the positions of the plurality of first key points, and generate a plurality of third key points for a third posture to be included in the linked motion information by using the positions of a plurality of first key points and the center points of the positions of a plurality of key points of the second motion information.

[0148] As described above, the 3D object generation system (100) can repeat the process of generating multiple key points for multiple poses to be included in the connection motion information by using the center point between key points included in different motions.

[0149] Accordingly, when a plurality of postures to be included in the connection motion information are generated, the 3D object generation system (100) determines the order of the plurality of postures to be included in the connection motion information based on the order in which the positions of the plurality of key points corresponding to the final posture of the first motion information and the positions of the plurality of key points corresponding to the plurality of postures to be included in the connection motion information are close to each other, and can generate the connection motion information according to the determined order.

[0150] Alternatively, when a plurality of postures to be included in the connection motion information are generated, the 3D object generation system (100) can determine the order of the plurality of postures to be included in the connection motion information based on the order of the positions of the plurality of key points corresponding to the initial posture of the second motion information and the positions of the plurality of key points corresponding to the plurality of postures to be included in the connection motion information, and can generate the connection motion information according to the determined order.

[0151] As another example, the 3D object generation system (100) can generate a connection motion line by connecting the positions of a plurality of key points corresponding to the final posture of the first motion information and the positions of a plurality of key points corresponding to the initial posture of the second motion information, and can generate a plurality of key points based on a preset interval (or preset number) from the plurality of key points of the first motion information to the plurality of key points of the second motion information on the generated connection motion line.

[0152] Accordingly, the 3D object generation system (100) can determine the order of a plurality of postures to be included in the connection motion information based on the order in which the positions of a plurality of key points of the first motion information are close, and generate the connection motion information according to the determined order.

[0153] Alternatively, the 3D object generation system (100) can determine the order of a plurality of postures to be included in the linked motion information based on the order of distance between the plurality of key points of the second motion information and the positions thereof, and generate the linked motion information according to the determined order.

[0154] Furthermore, in the process of generating connection motion information, the 3D object generation system (100) can determine the motion speed of the connection motion information by considering the first motion speed of the first motion information, which is a preceding motion for the connection motion information, and the second motion speed of the second motion information, which is a subsequent motion for the connection motion information.

[0155] Referring to FIG. 11, for example, the 3D object generation system (100) can determine a first motion speed based on a distance difference (41) between the positions of a plurality of key points (40a) in the final posture (31) of the first motion information and the positions of a plurality of key points (40c) in the posture (33) immediately before the final posture (31), and can determine a second motion speed based on a distance difference (42) between the positions of a plurality of key points (40b) in the initial posture (32) of the second motion information and the positions of a plurality of key points (40d) in the posture immediately after the initial posture (32).

[0156] Accordingly, the 3D object creation system (100) can determine the operation speed of the connection motion information so that among the multiple postures included in the connection motion information, the posture closer to the final posture (31) becomes closer to the first operation speed, and the operation speed of the connection motion information can be determined so that the posture closer to the initial posture (32) becomes closer to the second operation speed.

[0157] As another example, the 3D object generation system (100) can calculate center points for the positions of a plurality of key points corresponding to the final pose (31) and the positions of a plurality of key points corresponding to the initial pose (32) by applying a ratio of the first motion speed and the second motion speed, thereby calculating center points to which motion speeds are applied for the key points in the final pose (31) and the key points in the initial pose (32).

[0158] Accordingly, the 3D object generation system (100) can generate a plurality of postures to be included in the linked motion information by repeating the process of generating at least one posture close to the final posture (31) and at least one posture close to the initial posture (32) based on the center point to which the motion speed is applied. Through this, the 3D object generation system (100) can generate one or more postures in which the distance between key points is far as the motion speed is close to a fast posture among the final posture (31) and the initial posture (32), and can generate one or more postures in which the distance between key points is close as the motion speed is close to a slow posture. Accordingly, the 3D object generation system (100) can sequentially list the plurality of postures generated through the above process and generate linked motion information.

[0159] As another example, the 3D object generation system (100) generates a first posture to be included in the connection motion information by calculating the center points for the positions of a plurality of key points corresponding to the final posture (31) and the positions of a plurality of key points corresponding to the initial posture (32), and can determine the number of postures to be generated between the final posture (31) and the first posture and the number of postures to be generated between the initial posture (32) and the second posture based on the ratio of the first motion speed and the second motion speed, respectively.

[0160] As another example, the 3D object generation system (100) can generate a connection motion line by connecting the positions of a plurality of key points corresponding to the final posture (31) and the positions of a plurality of key points corresponding to the initial posture (32), and on the generated connection motion line, generate a pose to be included in the connection motion information at an interval with a smaller difference from the distance difference (41) corresponding to the first motion speed as it gets closer to the final posture (31), and generate a pose to be included in the connection motion information at an interval with a smaller difference from the distance difference (42) corresponding to the second motion speed as it gets closer to the initial posture (32).

[0161] Furthermore, referring again to FIG. 4, the 3D object creation system (100) can sequentially connect multiple pieces of motion information and insert connection motion information between each piece of motion information to connect them (S340).

[0162] Referring to FIG. 12, for example, the 3D object generation system (100) can obtain first motion information (21) for a sitting motion and second motion information (22) for a standing motion in response to the sentence 'sit down and then stand up', and can generate linked motion information (50) for the first motion information (21) and the second motion information (22). Accordingly, the 3D object generation system (100) can link the final posture of the first motion information (21) and the initial posture of the linked motion information (50), and link the final posture of the linked motion information (50) and the initial posture of the second motion information (22), thereby linking the first motion information, the linked motion information, and the second motion information.

[0163] Through the above configurations, the 3D object generation system (100) according to the present invention can obtain at least one piece of motion information from the database (131) based on at least one keyword generated in response to a sentence.

[0164] \*159In addition, the 3D object generation system (100) according to the present invention can further acquire information designed to configure the appearance of the 3D object (3) and the expression of the 3D object (3) based on at least one keyword generated in response to a sentence.

[0165] Furthermore, the 3D object creation system (100) according to the present invention can naturally connect different actions by creating connection action information for a plurality of action information items in a continuous action sequence.

[0166] Referring again to FIG. 3, the 3D object generation system (100) according to the present invention can generate a 3D object (3) that performs a movement corresponding to a sentence by applying movement information to a 3D model (1) (S400).

[0167] Specifically, the 3D object creation system (100) can apply the position of at least one key point specified in the first frame, which is the most advanced among a plurality of frames (or times) included in the motion information, to at least one key point corresponding to the at least one key point among a plurality of key points included in a 3D model (1) prepared in advance (or acquired from a database (131)).

[0168] Thereafter, the 3D object creation system (100) applies the position of at least one key point specified in the second frame immediately after the first frame to at least one key point corresponding to the at least one key point among a plurality of key points included in the 3D model (1), and repeats the above process for all frames specified in the motion information, thereby applying motion information to the 3D model (1).

[0169] Referring to FIGS. 13 and 14, for example, the 3D object generation system (100) can apply the positions of a plurality of key points (40) corresponding to each posture (35) to a plurality of key points included in a 3D model (1). At this time, the 3D object generation system (100) can generate a 3D object (3) that performs an action corresponding to a sentence by setting the plurality of key points included in the 3D model (1) to move sequentially according to the posture of the motion information connected above.

[0170] Through the above-described configurations, the 3D object generation system (100) according to the present invention can generate a 3D object (3) that performs an action corresponding to a sentence. In addition, the 3D object generation system (100) can generate a 3D object (3) that moves more naturally by generating and applying a connection action between different actions performed by the 3D model (1).

[0171] Meanwhile, the 3D object generation system (100) can also generate a 3D object from a 2D image. For example, when a 2D image is input, the 3D object generation system (100) can detect an object from the 2D image, identify a part corresponding to the object through object segmentation, and extract data related to the pose of the object for each frame of the 2D image. Accordingly, the 3D object generation system (100) can collect and merge pose data extracted from each frame of the 2D image by angle, and generate a 3D object with expressed volume and texture based on the merged result.

[0172] As an example, the generation of a 3D object may be performed based on a 3D volume metric. The 3D volume metric may be implemented by setting the initial location of an object within a 2D image as a reference point, and then estimating the size of the object by determining the location information of the corresponding location when the object moves to a different location within the 2D image, and then generating a volume in the form of a 3D mesh.

[0173] More specifically, the 3D object generation system (100) can input a 2D image and output a 3D object in a mesh form based on the volume and texture of the object in the 2D image. At this time, the input 2D image can include multiple frames in which the shape of the object is expressed at various angles and / or positions.

[0174] A 3D object generation system (100) can analyze key points and lines representing an object from a 2D image and generate a 3D point cloud as shape information of the object based on this.

[0175] The 3D object creation system (100) can create a 3D mesh object by performing 3D mesh modeling and texture mapping based on the created 3D point cloud.

[0176] When a 2D image is input, the 3D object generation system (100) can track key points of an object in the 2D image (41) and reconstruct a 3D model based on key points. In addition, the 3D object generation system (100) can detect and fit lines of an object in the 2D image to reconstruct a line-based 3D model. The 3D object generation system (100) can utilize predetermined reconstruction parameters in the process of reconstructing a 3D model.

[0177] In addition, the 3D object generation system (100) can generate a 3D point cloud (PC) that estimates the 3D shape of an object in a 2D image using a 3D model reconstructed based on key points and / or lines.

[0178] A 3D object generation system (100) can perform 3D mesh modeling based on a 3D point cloud (PC). The result of performing the 3D mesh modeling can be output as mesh information (MI, Mesh Information) representing the volume of the 3D object. The 3D object generation system (100) can use predetermined modeling parameters (Modeling Parameters) for the 3D mesh modeling. In addition, the 3D object generation system (100) performs texture mapping based on a color image of a 2D image. The result of performing the texture mapping can be output as texture information (TI, Texture Information) representing the texture of the 3D object.

[0179] In addition, the 3D object creation system (100) can create data related to a 3D mesh using mesh information (MI) and texture information (TI), and then create a 3D object based on this.

[0180] Accordingly, the 3D object generation system (100) can be implemented so that the 3D object performs a 3D motion based on the motion information by merging the motion information generated in advance with the 3D object.

[0181] At this time, the 3D object may be a 3D mesh object in the form of a 3D mesh, or may be a 3D character object that has been previously saved. Alternatively, if the 3D object generation system (100) uses a 3D character object modeled after a celebrity or a virtual character instead of a 3D mesh object, by merging motion information into the 3D character object, a 3D object can be generated so that the 3D character object reproduces the movement of the object in the 2D image.

[0182] In this regard, since motion information is data created based on sentences or two-dimensional images, and three-dimensional objects are objects created independently of sentences or two-dimensional images, a dissonance may occur between the motion information and the three-dimensional object. Accordingly, the three-dimensional object creation system (100) may perform a retargeting operation to adjust the motion information to fit the three-dimensional object in order to resolve this dissonance.

[0183] For example, the 3D object generation system (100) can align the joint positions of the motion information with the joint positions of the 3D object based on data related to key points of the motion information. Here, aligning the joint positions of the motion information with the joint positions of the 3D object may mean, for example, adjusting the joint positions of the motion information so that the joint positions of the motion information match the joint positions of the 3D object.

[0184] For example, when applying motion information to a 3D object, the physical characteristics of the model that formed the basis of the motion information may differ from those of the 3D object. Therefore, to eliminate dissonance resulting from differences in physical characteristics between objects, the joint locations (i.e., key points) of the motion information can be adjusted to match those of the 3D object.

[0185] At this time, the line of the motion information can also be adjusted together based on the adjusted key points of the motion information. For example, assuming that in the motion information extracted from the database, there are key points K1(1,1,1), K2(2,2,2), and a line L1 connecting the key points K1 and K2, the key points K1 and K2 can be adjusted to K1(1,2,2) and K2(2,3,3) respectively through retargeting, and accordingly, the line L1 can also be adjusted to a line connecting (1,2,2) and (2,3,3).

[0186] Furthermore, the 3D object creation system (100) can implement the movement of the 3D object to simulate the movement of the motion information based on the joint positions of the aligned motion information.

[0187] To this end, the 3D object generation system (100) can adjust the movement of the motion information, for example, the position of each key point and line, the movement distance, the motion range, etc., based on the adjusted joint positions (i.e., key points) and lines of the motion information. Then, by merging the movement of the adjusted motion information into the 3D object, the movement of the 3D object can be implemented.

[0188] The 3D object creation system (100) can output and store the movement of the 3D object implemented above.

[0189] Meanwhile, when explaining the present invention from an operational perspective, the 3D object generation system (100) according to the present invention can specify a sentence based on a user input and analyze the specified sentence. Then, the 3D object generation system (100) can define a plurality of actions corresponding to the sentence based on the analysis result, and can define a connection action that connects the plurality of actions corresponding to the sentence. Furthermore, the 3D object generation system (100) can generate a moving 3D object that performs the plurality of actions and the connection action using a learned artificial neural network.

[0190] Here, the connection action may be a connection action that connects at least two actions that are adjacent to each other according to the time series flow among the plurality of actions, as described above.

[0191] And, the movement of the three-dimensional object may be formed by a plurality of frames corresponding to a first operation among the plurality of operations, which include key point information of a three-dimensional model according to the first operation, a plurality of frames corresponding to a second operation among the plurality of operations, which include key point information of the three-dimensional model according to the second operation, and a plurality of frames corresponding to a connection operation, which includes key point information of the three-dimensional model according to the connection operation connecting the first operation and the second operation.

[0192] That is, the 3D object can be understood as a shape in which the 3D model moves while a plurality of frames corresponding to each of the first motion, the second motion, and the connection motion are played.

[0193] As previously discussed, each of the plurality of frames corresponding to each of the plurality of motions and each of the connection motions, for example, each of the plurality of frames corresponding to the first motion, each of the plurality of frames corresponding to the second motion, and each of the plurality of frames corresponding to the connection motion, may include location information for key points of the 3D model.

[0194] And, when the motions among the plurality of frames are adjacent to each other according to the time series flow, for example, the first motion and the second motion are adjacent to each other according to the time series flow, the key point of the three-dimensional model of the plurality of frames corresponding to the connection motion can be defined based on the key point information of the three-dimensional model of the plurality of frames corresponding to the first motion and the plurality of frames corresponding to the second motion.

[0195] At this time, based on the time series flow, if the first action is an action that precedes the second action, the position information of the key point of the 3D model included in the first frame among the plurality of frames corresponding to the connection action may be related to the position information of the key point of the 3D model included in the last frame among the plurality of frames corresponding to the first action, and the position information of the key point of the 3D model included in the last frame among the plurality of frames corresponding to the connection action may be related to the position information of the key point of the 3D model included in the first frame among the plurality of frames corresponding to the second action. Here, the fact that the position information of the key points is related to each other may mean that the position information of the key points is the same as or is position information corresponding to a close position.

[0196] For example, the position of the key point of the 3D model included in the first frame of the connection operation may be the same as or adjacent to the position of the key point of the 3D model included in the last frame of the first operation, and the position of the key point of the 3D model included in the last frame of the connection operation may be the same as or adjacent to the position of the key point of the 3D model included in the first frame of the second operation.

[0197] As described above, the method and system for generating a 3D object according to the present invention defines actions corresponding to the meaning of a sentence and generates a 3D object that performs the defined actions, thereby making it possible to generate a 3D object that moves by simply inputting a sentence, without the user having to define the actions of the 3D object one by one.

[0198] Furthermore, the method and system for generating a 3D object according to the present invention can further acquire information designed to implement various elements such as movement, expression, appearance, age, clothing, gender, physical condition, background, location, and situation of a 3D model corresponding to a 3D object based on keywords specified in response to a sentence.

[0199] In addition, according to various embodiments of the present invention, the method and system for generating a three-dimensional object can naturally connect different motions performed by a three-dimensional object by generating connection motion information for a plurality of motion information pieces in which the motion sequence is continuous.

[0200] Furthermore, the present invention discussed above can be implemented as computer-readable code or instructions on a program-recorded medium. That is, the various control methods according to the present invention can be provided in the form of programs, either integrated or individually.

[0201] Meanwhile, computer-readable media include all types of recording devices that store data that can be read by a computer system. Examples of computer-readable media include hard disk drives (HDDs), solid-state disk drives (SSDs), silicon disk drives (SDDs), ROMs, RAMs, CD-ROMs, magnetic tapes, floppy disks, and optical data storage devices.

[0202] Furthermore, the computer-readable medium may include a storage device and may be a server or cloud storage device accessible via communication. In this case, the computer may download the program according to the present invention from the server or cloud storage device via wired or wireless communication.

[0203] Furthermore, in the present invention, the computer described above is an electronic device equipped with a processor, i.e., a CPU (Central Processing Unit), and there is no particular limitation on its type.

[0204] Meanwhile, the detailed description above should not be construed as limiting in any respect and should be considered illustrative. The scope of the present invention should be determined by a reasonable interpretation of the appended claims, and all modifications within the equivalent scope of the present invention are intended to be included within the scope of the present invention.

Claims

1. A step of specifying a sentence based on user input; A step of analyzing the above-mentioned specific sentence and defining a plurality of actions corresponding to the sentence based on the analysis result; A step of defining a connecting action that connects the plurality of actions corresponding to the above sentence; and A method for generating a three-dimensional object, characterized in that it comprises a step of generating a moving three-dimensional object that performs the plurality of movements and the connection movements using a learned artificial neural network.

2. In paragraph 1, A method for generating a 3D object, characterized in that the above-mentioned connection operation is a connection operation that connects at least two adjacent operations according to a time series flow among the plurality of operations.

3. In paragraph 2, The movement of the above 3D object is, A plurality of frames corresponding to the first action including key point information of a three-dimensional model according to the first action among the plurality of actions, A plurality of frames corresponding to the second motion, which include key point information of the three-dimensional model according to the second motion among the plurality of motions, and A method for generating a 3D object, characterized in that the 3D object is formed by a plurality of frames corresponding to the connection operation including key point information of the 3D model according to the connection operation connecting the first operation and the second operation.

4. In paragraph 3, A method for generating a 3D object, characterized in that each of a plurality of frames corresponding to the first operation, a plurality of frames corresponding to the second operation, and a plurality of frames corresponding to the connection operation includes location information for key points of the 3D model.

5. In paragraph 4, If the first and second actions are adjacent actions according to the time series flow, the key points of the three-dimensional model of the plurality of frames corresponding to the connection actions are A method for generating a 3D object, characterized in that the 3D model is defined based on key point information of a plurality of frames corresponding to the first action and a plurality of frames corresponding to the second action.

6. In paragraph 5, Based on the time series flow, if the first action is an action that precedes the second action, The location information of the key point of the 3D model included in the first frame among the plurality of frames corresponding to the above connection operation is, Relating to the location information of the key point of the 3D model included in the last frame among the plurality of frames corresponding to the first movement, The location information of the key point of the 3D model included in the last frame among the plurality of frames corresponding to the above connection operation is, A method for generating a 3D object, characterized in that it is related to the location information of the key point of the 3D model included in the first frame among a plurality of frames corresponding to the second movement.

7. In paragraph 6, The position of the key point of the 3D model included in the first frame of the above connection operation is identical to or adjacent to the position of the key point of the 3D model of the last frame of the above first operation, A method for generating a 3D object, characterized in that the position of the key point of the 3D model included in the last frame of the above-mentioned connection operation is identical to or adjacent to the position of the key point of the 3D model of the first frame of the above-mentioned second operation.

8. A storage unit containing a pre-prepared database; and A three-dimensional object generation system comprising a control unit that specifies a sentence based on user input, analyzes the specified sentence, defines a plurality of actions corresponding to the sentence based on the analysis result, defines a connection action connecting the plurality of actions corresponding to the sentence, and generates a moving three-dimensional object that performs the plurality of actions and the connection action using a learned artificial neural network.

9. A program that is executed by one or more processes in an electronic device and stored in a computer-readable recording medium, The above program is, A step of specifying a sentence based on user input; A step of analyzing the above-mentioned specific sentence and defining a plurality of actions corresponding to the sentence based on the analysis result; A step of defining a connecting action that connects the plurality of actions corresponding to the above sentence; and A program stored on a computer-readable recording medium, characterized in that it includes commands for performing a step of generating a moving three-dimensional object that performs the plurality of movements and the connection movements using a learned artificial neural network.

10. Step of specifying a sentence based on user input; A step of analyzing the specific sentence and defining a conti corresponding to the sentence based on the analysis result; A step of obtaining a 3D model and motion information corresponding to the conti using a pre-prepared database; and A method for generating a 3D object, comprising the step of applying the motion information to the 3D model to generate a 3D object that performs a motion corresponding to the sentence.

11. In the 10th paragraph, the step of obtaining the 3D model and motion information, A method for generating a 3D object, comprising the steps of: when a conti related to a plurality of actions is defined in response to the above sentence, obtaining action information corresponding to each of the plurality of actions from the database, and sequentially connecting the obtained action information according to an action order specified based on the conti.

12. In the 11th paragraph, the step of sequentially connecting the motion information is: A method for generating a 3D object, comprising a step of generating connection motion information such that, among the plurality of motions, the final posture of the first motion information in the preceding motion order changes to the initial posture of the second motion information in the next motion order.

13. In the 12th paragraph, the step of generating the connection operation information comprises: A method for generating a three-dimensional object, comprising a step of determining the operation speed of the connection operation information by considering the first operation speed of the first operation information and the second operation speed of the second operation information among the plurality of operations.

14. In the 10th paragraph, the step of acquiring the 3D model and motion information, In the case where the conti includes a plurality of elements related to the three-dimensional object, a step of matching each of the plurality of keywords corresponding to the conti to one of the plurality of categories preset according to the plurality of elements; A method for generating a three-dimensional object, comprising a step of obtaining information corresponding to each keyword from a database based on the above-mentioned matched category.

15. In paragraph 10, the step of specifying the sentence comprises: A step in which data is entered based on user input; A step of analyzing the above data and converting it into text; and A method for generating a three-dimensional object, comprising a step of specifying the sentence so that a conti is defined based on the converted text.

16. In the 10th paragraph, the step of defining the conti is: A step of dividing the above-mentioned specific sentence into multiple words in morpheme units; A step of confirming the meaning corresponding to the above-mentioned plurality of words; and A method for generating a three-dimensional object, comprising the step of generating at least one keyword corresponding to the meaning of the specified sentence based on the confirmed meaning.

17. In paragraph 16, the step of defining the conti is: A step of confirming the correlation between different multiple words based on the meaning of each of the multiple words distinguished from the above sentence; and A method for generating a three-dimensional object, comprising a step of determining at least one of an action level and an action order for at least one keyword based on the above-determined correlation.

18. A storage unit including a pre-established database; and A 3D object generation system comprising a control unit that specifies a sentence based on user input, analyzes the specific sentence, defines a conti corresponding to the sentence based on the analysis result, obtains a 3D model and motion information corresponding to the conti using a pre-prepared database, and applies the motion information to the 3D model to generate a 3D object that performs a motion corresponding to the sentence.

19. A program executed by one or more processes in an electronic device and stored in a computer-readable recording medium, The above program is, A step of specifying a sentence based on user input; A step of analyzing the specific sentence and defining a conti corresponding to the sentence based on the analysis result; A step of obtaining a 3D model and motion information corresponding to the conti using a pre-prepared database; and A program stored on a computer-readable recording medium, characterized in that it includes commands for performing a step of applying the motion information to the three-dimensional model to create a three-dimensional object that performs a motion corresponding to the sentence.

Citation Information

Patent Citations

  • Cartoon image automatic generation device based on input sentence and method

    JP2022041990A

  • Apparatus, system and method for controlling virtual reality image and simulator

    KR1020170114371A

  • Optical laminate, optical member comprising the same and optical display apparatus comprising the same

    KR1020220121648A

  • Container

    KR1020230030774A

  • KR20220038945A