system

Abstract

Provide a system. A means for a user to upload image data from a terminal; means for analyzing the image data received by the server and generating a three-dimensional model from the two-dimensional image; means for animating the server-generated three-dimensional model; A means for a user to set a position where the three-dimensional model is to be displayed as augmented reality in the real world using a terminal; a means for the server to acquire the set location information and display the three-dimensional model at the specified location; A system including:

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Currently, it is difficult for ordinary users, especially children, to easily create and share augmented reality (AR) content with others. This limits the use of AR as a creative activity, a new communication tool, or an advertising and marketing method. Therefore, there is a need for a tool that allows anyone to easily convert their own illustrations into three-dimensional models, move them, display them in AR at a specified location, and share them with others. [Means for solving the problem]

[0005] The present invention solves the above problems by the following means.

[0006] The system includes a means for users to upload image data from their devices. The server has a means for analyzing the received image data and generating a 3D model from the 2D image. The server also includes a means for animating the generated 3D model and provides a means for users to set a location using their device to display the 3D model as augmented reality in the real world. The server also has a means for acquiring the set location information and displaying the 3D model at the specified location. The server also has a means for inspecting the quality of the analyzed image data and notifying the user of quality improvement suggestions. The server also provides a means for adding animation to the 3D model based on instructions from the user and notifying the user of the results. This makes it easy for anyone to create and animate their own illustrations, display them in AR at a specified location, and share them.

[0007] A "user" is a person or organization that uses the system to upload image data, operate three-dimensional models, set positions, and so on.

[0008] A "terminal" is a device used by a user to perform operations such as uploading image data, inputting chat instructions, and setting up augmented reality display.

[0009] "Image data" refers to digital files of two-dimensional illustrations or photographs that users upload from their devices.

[0010] A "server" is a computer system that receives, stores, and analyzes data sent from terminals via a network, and processes information for generating 3D models, adding animations, and displaying augmented reality.

[0011] "Analysis" refers to the computational process used to recognize and extract features and structures from received image data.

[0012] A "two-dimensional image" is an illustration or photograph drawn on a two-dimensional plane of width and height.

[0013] A "3D model" is a three-dimensional digital object expressed in three-dimensional space based on a two-dimensional image.

[0014] "Animation" is a technique for adding movement to a three-dimensional model and displaying how it changes over time.

[0015] "Augmented reality" is a technology that overlays digital information and objects onto the real world.

[0016] "Location information" is data relating to a specific location in the real world that is set by the user in order to display a three-dimensional model as augmented reality.

[0017] "Quality Check" is the process used to determine the visual quality and technical suitability of received image data.

[0018] "Quality improvement proposals" is a system that notifies users of recommendations for correcting or optimizing image data based on the results of quality checks. [Brief explanation of the drawings]

[0019] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7]FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0020] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0021] First, the terms used in the following description will be explained.

[0022] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0023] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0024] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0025] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0026] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0027] [First embodiment]

[0028] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0029] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0030] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0031] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0032] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0033] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0034] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0035] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0036] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0037] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0038] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0039] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0040] This invention relates to a system that allows anyone to easily convert two-dimensional images into three-dimensional models, add animation, and display them in augmented reality for sharing. This system has the function of allowing users to upload image data from their devices, and a server analyzes the data to generate a three-dimensional model and add animation. It also has the function of allowing users to set the augmented reality display position of the three-dimensional model using their devices, and the server acquires the location information and displays it at the specified location.

[0041] A natural language description of the program's operation

[0042] 1. Ability for users to upload illustrations

[0043] The user opens the app on their device, selects the illustration they want to upload from the gallery, and presses the "Upload" button to send the selected illustration to the server.

[0044] The server receives the uploaded illustration data and stores it in a temporary storage area. It checks the integrity of the file and, if there are no problems, proceeds to the next step.

[0045] 2. 3D conversion function for illustrations

[0046] The server analyzes the received illustration data, extracts key features and contours, and creates initial data for generating a 3D model from the 2D image based on the analysis results.

[0047] The server calls the generative AI model to convert the 2D illustration into a 3D model, which is then saved in a database and a notification is sent to the user when the conversion is complete.

[0048] 3. 3D model animation function

[0049] The user uses the chat box on the terminal to input actions (e.g., "walk," "jump," etc.) for the three-dimensional model and sends them to the server.

[0050] The server analyzes the user's instructions, generates the corresponding animation using Text2Motion technology, and adds it to the 3D model. The generated animation data is also saved in the database. Once completed, the server notifies the user of the results.

[0051] 4. AR display position setting function

[0052] The user uses the device's camera function to determine where in the real world the 3D model should be displayed, specifically by dragging and positioning the 3D model on the device screen.

[0053] The terminal transmits the set position information to the server, and the server analyzes the information to determine the display position of the three-dimensional model in the real world.

[0054] The server sends data for displaying the 3D model at the specified location to the terminal, and the terminal displays the 3D model at the specified location as augmented reality based on the received data.

[0055] Specific examples

[0056] When a child uploads an illustration of a character they've drawn and commands "run" via chat, the character is converted into a three-dimensional model and a running animation is added. The user can then use their device's camera to position the character in their living room. This positioning information is sent to the server, and the character is displayed in augmented reality, running at the specified location in the living room. The user can view this display or share it with others.

[0057] As described above, the present invention relates to a system that allows users to easily convert two-dimensional illustrations into three-dimensional models, animate them, and display them as augmented reality. This system can provide creative activities and new communication methods.

[0058] The processing flow will be explained below.

[0059] Specific processing flow of the program

[0060] 1. Ability for users to upload illustrations

[0061] Step 1:

[0062] The user opens the app on their device and selects the illustration they want to upload from the gallery.

[0063] When the user presses the "Upload" button, the terminal sends the selected illustration to the server.

[0064] Step 2:

[0065] The server receives the transmitted illustration data and stores it in a temporary storage area.

[0066] The server performs a file integrity check to ensure the file was uploaded successfully.

[0067] Step 3:

[0068] The server analyzes the quality of the uploaded illustrations and, if necessary, notifies the user of suggestions for improving the quality.

[0069] 2. 3D conversion function for illustrations

[0070] Step 1:

[0071] The server analyzes the received illustration data and extracts key features and contours, allowing each part of the illustration to be recognized.

[0072] Step 2:

[0073] Based on the analysis results, the server creates initial data for generating a three-dimensional model from the two-dimensional image.

[0074] Step 3:

[0075] The server invokes a generative AI model (e.g., generative AI model technology) to convert the 2D illustration into a 3D model.

[0076] The server stores the generated 3D model in a database.

[0077] Step 4:

[0078] The server notifies the user when the 3D rendering is complete and provides a link or preview for confirmation.

[0079] 3. 3D model animation function

[0080] Step 1:

[0081] The user opens a chat box through the device app and inputs actions (e.g., "run," "jump," etc.) for the three-dimensional model.

[0082] When the user sends an instruction, the terminal sends the instruction to the server.

[0083] Step 2:

[0084] The server receives instructions from the user and analyzes the content of the instructions. For example, if the instruction to "run" is sent, the server analyzes the action.

[0085] Step 3:

[0086] The server uses Text2Motion technology to generate animation data for adding movement to the three-dimensional model based on the analyzed instructions.

[0087] Step 4:

[0088] The server saves the generated animation data in the 3D model database and notifies the user when it is complete.

[0089] 4. AR display position setting function

[0090] Step 1:

[0091] The user uses the device's camera to determine where in the real world the 3D model should be displayed, and then drags and positions the 3D model on the device screen.

[0092] Step 2:

[0093] The terminal transmits the set position information of the augmented reality display to the server.

[0094] Step 3:

[0095] The server analyzes the received location information and generates data for displaying the three-dimensional model at the specified location.

[0096] Step 4:

[0097] The server transmits this data to the terminal, and the terminal displays the three-dimensional model as augmented reality at the specified position based on the received data.

[0098] For example, a child can upload an illustration of a character they have drawn and use chat to say "run." As a result, the character is generated as a 3D model, a running animation is added, and it is displayed as augmented reality on the living room floor of the user's home. The user can then use their device's camera to watch the character running around the living room.

[0099] Example 1

[0100] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0101] With conventional technologies, converting 2D images into 3D models, adding movement, and displaying them in augmented reality is extremely difficult and requires specialized knowledge and time. Furthermore, there is a lack of easy ways for users to set their desired actions and location information, creating a demand for intuitive and easy operation. This creates a need for an efficient system for creating and sharing creative content.

[0102] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0103] In this invention, the server includes means for a user to upload image data from a terminal, means for analyzing the image data received by the server and generating a 3D model from a 2D image, means for converting the 2D image into a 3D model using a generative AI model, means for adding animation to the 3D model based on an action specified by the user, means for the user to use the terminal to set a position where the 3D model is to be displayed as augmented reality in the real world, and means for the server to acquire the set position information and display the 3D model at the specified position. This allows users to easily convert 2D illustrations into 3D models, add animation, and display and share them as augmented reality.

[0104] A "user" is any individual or organization who wishes to use the system to convert two-dimensional images into three-dimensional models and display them as augmented reality.

[0105] "Terminal" refers to a device operated by a user to upload image data, place 3D models, and specify actions. Specifically, this includes smartphones, tablets, and PCs.

[0106] A "server" is a computer or network system that receives and analyzes data sent from a terminal, generates a three-dimensional model from a two-dimensional image, adds actions, and performs calculations to display it as augmented reality.

[0107] "Image data" is digital data containing visual information such as two-dimensional illustrations and photographs that are uploaded by users from their terminals.

[0108] "Analysis" is the process by which the server understands the structure and characteristics of the image data it receives and extracts the necessary information.

[0109] A "two-dimensional image" is flat image data that has two dimensions, vertical and horizontal.

[0110] A "three-dimensional model" is digital data that has three dimensions - length, width, and depth - and can be displayed three-dimensionally in a virtual space.

[0111] "Generative AI model" refers to an algorithm or technique that uses artificial intelligence to convert two-dimensional images into three-dimensional models.

[0112] "Animation" is digital data that adds movement to a three-dimensional model and makes it operate continuously.

[0113] "Text2Motion technology" is a technology that analyzes text instructions entered by the user and adds corresponding movements to a three-dimensional model.

[0114] "Augmented reality (AR)" is a technology that displays computer-generated visual information overlaid on a real-world environment.

[0115] "Location information" is digital data relating to a location designated by a user to display a three-dimensional model in the real world.

[0116] A "database" is a storage device or system for storing and managing three-dimensional models and animation data.

[0117] The present invention relates to a system that allows users to easily convert two-dimensional images into three-dimensional models, add motion, and display and share them as augmented reality. The system has several main functions, each of which is realized by specific hardware and software.

[0118] The system's hardware includes terminal devices such as smartphones, tablets, and PCs, as well as a server, while the software uses the generative AI model "DeepArt3D" and Text2Motion technology.

[0119] Ability for users to upload illustrations

[0120] Users launch the application using a terminal device such as a smartphone or PC and select the 2D illustration they want to upload from local storage or a gallery. The selected illustration is sent from the device to the server. The server receives the uploaded illustration data and stores it in a temporary storage area. Once the integrity of the file is confirmed, the process proceeds to the next step.

[0121] 3D conversion function for illustrations

[0122] The server analyzes the received illustration data and uses the DeepArt3D model to generate a 3D model from the 2D image. First, it extracts the main features and contours of the illustration and creates initial data for the 3D model based on them. Next, the DeepArt3D model uses this initial data to generate the 3D model. The generated 3D model is saved in a database, and the user is notified when the conversion is complete.

[0123] 3D model animation function

[0124] The user opens the chat box on their device and inputs actions for the 3D model (such as "walk" or "jump") in text format. The input instructions are sent to the server, which uses Text2Motion technology to generate the corresponding animation and add it to the 3D model. The generated animation data is saved in a database, and the user is notified when the animation has been added.

[0125] For example, if a user inputs "Make this character run," the server will analyze this prompt and generate and add a "running" animation using Text2Motion technology. This method allows users to easily implement their desired movements in a 3D model.

[0126] AR display position setting function

[0127] The user activates the device's camera and decides where in the real world to display the 3D model. Specifically, they drag and drop the 3D model onto the device screen to place it. The device then sends this location information to the server, which analyzes it and determines where to display the 3D model. The server then sends the necessary data to the device, and the device displays the 3D model at the specified location as augmented reality.

[0128] For example, if a child uploads an illustration of a character and commands "run" via chat, the character is converted into a three-dimensional model and a running animation is added. Furthermore, the user can use the device's camera function to position the character in their living room. This positioning information is sent to the server, and the character is displayed in augmented reality as if running at the specified location in the living room.

[0129] As described above, the present invention relates to a system that allows users to intuitively operate it, quickly and easily convert two-dimensional illustrations into three-dimensional models, and then use them as augmented reality. This system provides new ways of creative activities and communication, and greatly enhances user convenience.

[0130] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0131] Step 1:

[0132] The user launches the application using their device. The user selects the 2D illustration they want to upload from local storage or the gallery. The selected illustration data is sent from the device to the server by pressing the "Upload" button.

[0133] Input: 2D illustration data selected from local storage or gallery

[0134] Output: Illustration data sent to the server

[0135] Step 2:

[0136] The server receives the uploaded illustration data and stores it in a temporary storage area. The server then checks the integrity of the file and, if there are no problems, begins analysis.

[0137] Input: Received illustration data

[0138] Output: Illustration data with consistency confirmed

[0139] Step 3:

[0140] The server analyzes the received illustration data, extracting the main features and contours of the image and creating initial data for generating a 3D model from the 2D image.

[0141] Input: Illustration data that has passed the consistency check

[0142] Output: Initial data including key feature and contour extraction results

[0143] Step 4:

[0144] The server calls the generative AI model "DeepArt3D" and converts the 2D illustration into a 3D model using the initial data. At this time, the server constructs the model and adds texture.

[0145] Input: Initial data

[0146] Output: Generated 3D model

[0147] Step 5:

[0148] The server stores the generated 3D model in a database and also notifies the user that the conversion of the 3D model is complete.

[0149] Input: Generated 3D model

[0150] Output: 3D model stored in database, notification to user

[0151] Step 6:

[0152] The user opens a chat box on the terminal and inputs text for actions to be performed on the 3D model (for example, "walk," "jump," etc.). The input instructions are sent from the terminal to the server.

[0153] Input: User instruction text

[0154] Output: Instruction text sent to the server

[0155] Step 7:

[0156] The server analyzes the user's text instructions and generates the corresponding animation using Text2Motion technology, a process that adds movement to the 3D model based on the user's instructions.

[0157] Input: Instruction text

[0158] Output: An animated 3D model

[0159] Step 8:

[0160] The server stores the animated 3D model in a database and also notifies the user that the animation has been completed.

[0161] Input: An animated 3D model

[0162] Output: Animated 3D model stored in database, notification to user

[0163] Step 9:

[0164] The user activates the device's camera function and sets the location in the real world where they want to display the 3D model on the device screen. Specifically, the user drags and drops the 3D model onto the device screen to place it. The set location information is sent to the server.

[0165] Input: Set location information

[0166] Output: Location information sent to the server

[0167] Step 10:

[0168] The server analyzes the received location information and determines the display position of the 3D model in the real world. It then transmits the necessary data to the device and displays the 3D model in the specified position as augmented reality.

[0169] Input: Location

[0170] Output: Display data of a 3D model of the real world generated from engine data

[0171] Step 11:

[0172] The device uses the received display data to display the 3D model as augmented reality at a specified location in the real world, allowing the user to view and share the display with others.

[0173] Input: Display data

[0174] Output: Augmented reality display of 3D model in the real world

[0175] (Application example 1)

[0176] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0177] Conventional systems that convert 2D images into 3D models and add animation have had the problem of being complex to operate, making them difficult for users to operate intuitively. Furthermore, when displaying the generated 3D model in the real world, it is difficult for users to set the exact display position, which is time-consuming. Furthermore, in advertising and promotions, there is a lack of user-participation interactive elements, and there is a lack of a way to easily create visually appealing content.

[0178] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0179] In this invention, the server includes: means for a user to upload image data from a terminal; means for the server to analyze the received image data and generate a three-dimensional model from a two-dimensional image; means for the server to animate the generated three-dimensional model; means for the user to set a position in the real world where the three-dimensional model will be displayed as augmented reality using the terminal; means for the server to acquire the set position information and display the three-dimensional model at the specified position; and means for analyzing the user's action instructions regarding the three-dimensional model and generating corresponding animations. This allows a user to generate a three-dimensional model from a two-dimensional image with simple operations, add animation to the model, and display it in augmented reality. Furthermore, incorporating interactive elements makes it easy to create visually appealing advertisements and promotional content.

[0180] "User" refers to the entity that uses the system to upload image data, generate three-dimensional models, add animations, and set display positions as augmented reality.

[0181] A "terminal" is a device used by a user, and refers to an electronic device such as a smartphone, smart glasses, or a head-mounted display.

[0182] "Image data" refers to two-dimensional visual information uploaded by a user through a terminal.

[0183] A "server" refers to a computing device that receives image data from a terminal, analyzes it, generates a three-dimensional model from the two-dimensional image, adds animation, and notifies the user of the results.

[0184] A "two-dimensional image" refers to image information that is composed of two dimensions, vertical and horizontal.

[0185] A "three-dimensional model" refers to a three-dimensional model expressed in three dimensions: length, width, and height.

[0186] "Animation" refers to a technique of expression that gives the visual impression of movement by adding movement to a three-dimensional model.

[0187] "Augmented reality" refers to the technology of overlaying digital information (in this case, three-dimensional models) onto a real-world environment.

[0188] "Location information" refers to data that allows a user to use a device to set where in the real world a three-dimensional model is to be displayed.

[0189] An "action instruction" refers to input information instructing a three-dimensional model to perform a specific action or movement.

[0190] The present invention provides a system that allows a user to convert a two-dimensional image into a three-dimensional model, add movement to the model, and display it in augmented reality. The following description will explain in detail how to specifically implement the present invention.

[0191] 1. System Configuration

[0192] The system for implementing the present invention mainly uses the following hardware and software.

[0193] Hardware:

[0194] User devices: smartphones (ANDROID (registered trademark) or iOS), smart glasses, head-mounted displays

[0195] Server: A cloud-based computing device (such as AWS® or Google® Cloud)

[0196] software:

[0197] Smartphone applications: Kotlin for Android and Swift for iOS

[0198] Server software: Python is used, with TENSORFLOW (registered trademark) for generative AI models and OpenCV for image analysis.

[0199] Database: PostgreSQL

[0200] 2. Program Processing Overview

[0201] The program on the server performs the following process.

[0202] 1. Ability for users to upload illustrations:

[0203] The user launches the app on their smartphone, selects the 2D image they want to upload from their gallery, and presses the "Upload" button.

[0204] The server receives the uploaded image data and stores it in a temporary storage area. It checks the integrity of the data and, if there are no problems, proceeds to the next step.

[0205] 2. 3D conversion function for illustrations:

[0206] The server analyzes the received image data, extracts key features and contours, and creates initial data for generating a 3D model from the 2D image based on the analysis results.

[0207] The server invokes the generative AI model to convert the 2D image into a 3D model, which is then stored in a database and a notification is sent to the user when the conversion is complete.

[0208] 3. 3D model animation features:

[0209] The user uses the chat box on their smartphone to instruct the 3D model to take actions (e.g., "spin," "jump," "glow," etc.) and sends the instructions to the server.

[0210] The server analyzes the user's instructions, generates the corresponding animation using Text2Motion technology, and adds it to the 3D model. The generated animation data is also saved in a database, and the user is notified of the results once the process is complete.

[0211] 4. AR display position setting function:

[0212] The user uses the smartphone's camera to decide where in the real world the 3D model should be displayed, by dragging and positioning the 3D model on the device screen.

[0213] The terminal transmits the set position information to the server, and the server analyzes the information to determine the display position of the three-dimensional model in the real world.

[0214] The server transmits data for displaying a three-dimensional model at a specified position to the terminal, and the terminal displays the three-dimensional model at the specified position as augmented reality based on the received data.

[0215] Specific examples

[0216] Suppose a company wants to run an advertising campaign. A company representative opens a smartphone app and uploads a promotional company logo. Then, they type "jump" into the chat box. The server analyzes the logo, converts it into a 3D model, and adds a jumping animation to the model. The representative then places a promotional character on a cafe table, which is displayed in the real world as augmented reality. Examples of prompts used in this process include "upload an image file selected by the user to the server," "extract key features from the 2D logo and convert it into a 3D model," and "add a rotation animation to the 3D model."

[0217] Through the above process, users can intuitively and simply convert two-dimensional images into three-dimensional models, add movement to the models, and display them in the real world as augmented reality, making it possible to generate visually appealing advertisements and promotional content.

[0218] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0219] Program processing steps

[0220] Step 1:

[0221] The user launches the smartphone app, selects a two-dimensional image (e.g., a company logo) from the gallery, and presses the "Upload" button.

[0222] Input: 2D image data from the user's terminal

[0223] Operation: The terminal sends the selected image data to the server.

[0224] Output: 2D image data uploaded to the server

[0225] Step 2:

[0226] The server analyzes the received image data and stores it in a temporary storage area. It checks the data for consistency and, if there are no problems, it proceeds to the next step.

[0227] Input: Received image data

[0228] Action: Check the data integrity and save it to a temporary storage area.

[0229] Output: Image data with integrity confirmed

[0230] Step 3:

[0231] The server extracts the main features and contours of the received image data, and based on the analysis results, creates initial data for generating a 3D model from the 2D image.

[0232] Input: Image data whose integrity has been verified

[0233] How it works: Feature extraction is performed using OpenCV.

[0234] Output: Key features and contours

[0235] Step 4:

[0236] The server invokes a generative AI model (TensorFlow) to convert the 2D image into a 3D model, which is then saved in a database and a notification is sent to the user when the conversion is complete.

[0237] Input: feature and contour data

[0238] How it works: Generates a 3D model using a TensorFlow model and stores it in a database

[0239] Output: 3D model data, conversion completion notification

[0240] Step 5:

[0241] The user uses the chat box on their smartphone to instruct the 3D model to take an action (e.g., "spin," "jump," "glow") and send it to the server.

[0242] Input: User instruction for action

[0243] Action: Enter instructions in the chat box and send them to the server

[0244] Output: Action instructions sent to the server

[0245] Step 6:

[0246] The server analyzes the user's instructions, generates the corresponding animation using Text2Motion technology, and adds it to the 3D model. This animation data is also stored in the database, and the result is notified to the user after completion.

[0247] Input: User instruction for action

[0248] Action: Animation is generated using Text2Motion technology and saved in the database

[0249] Output: 3D model data with animation, notification of completion of generation

[0250] Step 7:

[0251] Users use the smartphone's camera to decide where in the real world they want to display the 3D model, then drag and place the model on the device screen.

[0252] Input: User's display position setting information

[0253] Operation: Set the display position using the camera function and send the position information to the server

[0254] Output: Display position information sent to the server

[0255] Step 8:

[0256] The server analyzes the location information set by the user and determines the display position of the 3D model in the real world. The server then sends data to the device to display the 3D model at the specified position.

[0257] Input: Display location information

[0258] Operation: Analyzes location information, generates data for augmented reality display, and sends it to the device

[0259] Output: Augmented reality display data

[0260] Step 9:

[0261] Based on the data received from the server, the device displays a three-dimensional model as augmented reality at a specified location.

[0262] Input: Augmented reality display data

[0263] Operation: A 3D model is superimposed on the camera image based on the received data.

[0264] Output: 3D model displayed as augmented reality in the real world

[0265] The above are the processing steps for carrying out the invention.

[0266] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0267] The present invention relates to a system that allows anyone to easily convert two-dimensional images into three-dimensional models, add movement, and display them in augmented reality for sharing. The present invention enables the generation of more personalized content by combining an emotion engine that recognizes user emotions. This system has a function that allows users to upload image data from their devices, and a server analyzes the data to generate a three-dimensional model and add animation. The system also has a function that allows users to use their devices to set the augmented reality display position of the three-dimensional model, and the server acquires that position information and displays it at the specified location. Additionally, the emotion engine can recognize user emotions and change behavior or adjust animations based on those emotions.

[0268] A natural language description of the program's operation

[0269] 1. Ability for users to upload illustrations

[0270] The user opens the app on their device, selects the illustration they want to upload from the gallery, and presses the "Upload" button to send the selected illustration to the server.

[0271] The server receives the illustration data and stores it in a temporary storage area. It checks the integrity of the file and, if there are no problems, proceeds to the next step.

[0272] 2. 3D conversion function for illustrations

[0273] The server analyzes the received illustration data, extracts key features and contours, and creates initial data for generating a 3D model from the 2D image based on the analysis results.

[0274] The server calls the generative AI model and converts the 2D illustration into a 3D model, which is then stored in a database.

[0275] The server notifies the user when the 3D rendering is complete and provides a link or preview for confirmation.

[0276] 3. 3D model animation function

[0277] The user opens a chat box through the app on their device, inputs actions for the three-dimensional model (e.g., "run," "jump," etc.), and sends them to the server.

[0278] The server receives instructions from the user and analyzes the content of the instructions. For example, if the instruction to "run" is sent, the server analyzes the action.

[0279] The server uses Text2Motion technology to generate animation data for adding movement to the three-dimensional model based on the analyzed instructions.

[0280] The server saves the generated animation data in the 3D model database and notifies the user when it is complete.

[0281] 4. AR display position setting function

[0282] The user uses the device's camera to determine where in the real world the 3D model should be displayed, and then drags and positions the 3D model on the device screen.

[0283] The terminal transmits the set position information of the augmented reality display to the server.

[0284] The server analyzes the received location information and generates data for displaying the three-dimensional model at the specified location.

[0285] The server transmits this data to the terminal, and the terminal displays the three-dimensional model as augmented reality at the specified position based on the received data.

[0286] 5. Implementing the Emotion Engine

[0287] The device is equipped with sensors (e.g., facial recognition using a camera, voice analysis using a microphone) to acquire the user's emotions, and uses these to collect the user's emotional data.

[0288] The device transmits emotion data to the server, which then analyzes the data using an emotion engine.

[0289] The server adjusts the behavior and animation of the 3D model based on the user's emotions, for example, making the model more active if the user is happy, or slowing down if the user is sad.

[0290] The server transmits the adjustment results to the terminal, and the terminal updates the display of the three-dimensional model based on the results.

[0291] Specific examples

[0292] A child uploads an illustration of a character they have drawn and commands "run" via chat. As a result, the character is generated as a 3D model and a running animation is added. The user then uses the camera function of their device to place the character on the floor of their living room. This placement information is sent to the server, and the character is displayed in augmented reality as if running at the specified location in the living room.

[0293] Furthermore, the user's device uses a camera to analyze their facial expressions, and if the user is having fun, the character will adjust to run more energetically. When the device detects that the user is happy, the server's emotion engine activates the character's movements, realizing behavior that matches the user's emotions.

[0294] As described above, the present invention relates to a system that allows users to easily convert 2D illustrations into 3D models, animate them, and display them as augmented reality. Furthermore, by combining this with an emotion engine, it is possible to provide personalized content that responds to the user's emotions.

[0295] The processing flow will be explained below.

[0296] Specific processing flow of the program (when combined with an emotion engine)

[0297] 1. Ability for users to upload illustrations

[0298] Step 1:

[0299] The user opens the app on their device and selects the illustration they want to upload from the gallery.

[0300] When the user presses the "Upload" button, the terminal sends the selected illustration to the server.

[0301] Step 2:

[0302] The server receives the transmitted illustration data and stores it in a temporary storage area.

[0303] The server performs a file integrity check to ensure the file was uploaded successfully.

[0304] Step 3:

[0305] The server analyzes the quality of the uploaded illustration and, if any parts are distorted or of low resolution, notifies the user with suggestions for correcting them.

[0306] 2. 3D conversion function for illustrations

[0307] Step 1:

[0308] The server analyzes the received illustration data and extracts key features and contours, which correspond to each part of the 2D illustration.

[0309] Step 2:

[0310] Based on the analysis results, the server extracts the initial data necessary to generate a basic three-dimensional model.

[0311] Step 3:

[0312] The server invokes a generative AI model to convert the 2D illustration into a 3D model, which is made up of data including shape and texture.

[0313] The server stores the generated 3D model in a database and notifies the user that the 3D model generation is complete.

[0314] 3. 3D model animation function

[0315] Step 1:

[0316] The user opens a chat box through the device app and inputs actions (e.g., "run," "jump," etc.) for the three-dimensional model.

[0317] When the user sends an instruction, the terminal sends the instruction to the server.

[0318] Step 2:

[0319] The server receives instructions from the user and analyzes the content of the instructions. For example, if the instruction to "run" is sent, the server analyzes the action in detail.

[0320] Step 3:

[0321] The server uses Text2Motion technology to generate animation data for adding movement to the three-dimensional model based on the analyzed instructions.

[0322] Step 4:

[0323] The server saves the generated animation data in the 3D model database and notifies the user when it is complete.

[0324] 4. AR display position setting function

[0325] Step 1:

[0326] The user uses the device's camera to determine where in the real world the 3D model should be displayed, and then drags and positions the 3D model on the device screen.

[0327] Step 2:

[0328] The terminal transmits the set position information of the augmented reality display to the server.

[0329] Step 3:

[0330] The server analyzes the received location information and generates data for displaying the three-dimensional model at the specified location.

[0331] Step 4:

[0332] The server transmits this data to the terminal, and the terminal displays the three-dimensional model as augmented reality at the specified position based on the received data.

[0333] 5. Implementing the Emotion Engine

[0334] Step 1:

[0335] The device is equipped with sensors (e.g., facial recognition using a camera, voice analysis using a microphone) to acquire the user's emotions, and uses these to collect the user's emotional data.

[0336] Step 2:

[0337] The device sends emotion data to the server, which then uses an emotion engine to analyze the data, for example, recognizing whether the user is surprised or happy through facial expression analysis.

[0338] Step 3:

[0339] The server adjusts the behavior and animation of the 3D model based on the user's emotions: for example, if the user is having fun, the model's movements will be more active, and conversely, if the user is sad, the model's movements will be more gentle.

[0340] Step 4:

[0341] The server transmits the adjustment results to the terminal, and the terminal updates the display of the three-dimensional model based on the results.

[0342] Specific examples

[0343] As a concrete example, a child uploads an illustration of a character they have drawn and commands "run" via chat. As a result, the character is generated as a 3D model and a running animation is added. The user then uses the camera function of their device to place the character on the floor of their living room. This placement information is sent to the server, and the character is displayed in augmented reality as if running at the specified location in the living room.

[0344] Furthermore, the user's device uses a camera to analyze their facial expressions, and if the user is having fun, the character will adjust to run more energetically. When the device detects that the user is happy, the server's emotion engine activates the character's movements, realizing behavior that matches the user's emotions.

[0345] Example 2

[0346] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0347] Currently, converting two-dimensional illustrations into three-dimensional models, adding movement to those models, and displaying them as augmented reality requires extensive specialized knowledge and complex software. These operations are difficult for ordinary users to perform due to the high technical barriers. Furthermore, the content created is rarely properly personalized based on the user's emotions. This limits the user experience and makes it difficult to create more engaging content.

[0348] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for a user to upload image data from a terminal, a means for the server to analyze the image data received and generate a three-dimensional model from a two-dimensional image, and a means for animating the three-dimensional model generated by the server. This allows a user, without specialized knowledge, to intuitively convert a two-dimensional illustration into a three-dimensional model, add animation, and display it as augmented reality. Furthermore, the terminal includes a means for acquiring the user's emotions and adjusting the behavior of the three-dimensional model based on the emotions, thereby enabling the generation of more personalized and attractive content.

[0349] A "user" is an entity that uses the system to upload image data and generate three-dimensional models and display augmented reality.

[0350] A "terminal" is an electronic device that a user uses to upload image data, such as a smartphone or tablet.

[0351] "Image data" refers to two-dimensional illustrations and image files uploaded by users.

[0352] A "server" is a computer system that analyzes received image data, generates three-dimensional models, animates them, and manages the augmented reality display.

[0353] "Analysis" is the process of extracting features and contours from received image data, which serve as the basis for generating a three-dimensional model from the two-dimensional image.

[0354] A "two-dimensional image" refers to a flat image that has only width and height information.

[0355] A "3D model" is a three-dimensional digital object that has width, height, and depth information.

[0356] "Animation" refers to the process of adding movement to a three-dimensional model.

[0357] "Augmented reality" refers to a technology that overlays digital information onto images of the real world.

[0358] "Location information" is data that indicates where in the real world a three-dimensional model is to be displayed.

[0359] "Emotion" refers to the psychological and sensory state that can be read from the user's facial expressions, voice, etc.

[0360] "Adjusting the behavior" refers to changing the animation of the three-dimensional model according to the user's emotions.

[0361] The present invention relates to a system that allows a user to easily convert a two-dimensional illustration into a three-dimensional model, add movement to the model, and display it as an augmented reality. An embodiment of the system will be described in detail below.

[0362] 1. Ability for users to upload illustrations

[0363] Users launch the dedicated application using a device such as a smartphone or tablet. They tap the "Select from Gallery" button within the application to open the device's image gallery and select the illustration they wish to upload. The user then presses the "Upload" button, which sends the illustration data to the server. The server receives the illustration data and stores it in a temporary storage area. At this time, the integrity of the file is also checked.

[0364] 2. 3D conversion function for illustrations

[0365] The server analyzes the received illustration data and extracts key features and contours. This process uses image analysis algorithms such as OpenCV. Based on the analysis results, the server invokes a generative AI model (e.g., DALL-E or Stable Diffusion) to convert the 2D illustration into a 3D model. The generated 3D model is stored in a database, and the user is notified when generation is complete. The notification includes a confirmation link and a preview image.

[0366] 3. 3D model animation function

[0367] The user inputs action instructions (e.g., "run," "jump," etc.) for the 3D model in the app's chat box and sends them to the server. The server analyzes the received instructions and generates animation data using Text2Motion technology. This animation data is also stored in the database, and the user is notified when generation is complete.

[0368] 4. AR display position setting function

[0369] The user activates the device's camera function and drags the 3D model on the screen to determine where in the real world the model should be displayed. The location information is sent to the server, which analyzes the received location information and generates augmented reality data for displaying the 3D model at the specified location. The generated data is sent to the device, which then displays the 3D model in the real world based on the received data.

[0370] 5. Implementing the Emotion Engine

[0371] The device uses sensors such as a camera and microphone to collect the user's emotional data in real time. For example, the camera recognizes the user's face and uses software (e.g., Emotion API) to analyze their facial expressions. The collected emotional data is sent to a server, which then analyzes it using an emotion engine. Based on the analysis results, the server adjusts the movement and animation of the 3D model and sends the results to the device. This causes the model's movement to change according to the user's emotions.

[0372] Specific examples

[0373] For example, a child uploads an illustration of a character they drew from their device and commands "run" in the chat box. In this case, the server generates a 3D model and adds a running animation. The user then uses the device's camera to place the character on the living room floor. Once the placement information is sent to the server, the character is displayed in augmented reality as if running in the specified position. Furthermore, if the device detects the user's facial expression of enjoyment, the server's emotion engine actively adjusts the character's movements.

[0374] Prompt Sentence Examples

[0375] "Turn your drawn character into a 3D model and run it around your living room."

[0376] By implementing these functions, the system of the present invention enables users to intuitively convert 2D illustrations into 3D models, animate them, and display them as augmented reality without requiring specialized knowledge. It also provides personalized content based on the user's emotions using an emotion engine.

[0377] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0378] Step 1:

[0379] Input: The user launches the app on their device and selects the 2D illustration to upload.

[0380] How it works: A user taps the "Select from Gallery" button in the app, opens the image gallery, selects the illustration they want to upload, and then presses the "Upload" button.

[0381] Data processing: The terminal prepares the selected illustration data and encodes the image data for transmission to the server.

[0382] Output: The encoded image data is sent to the server.

[0383] Step 2:

[0384] Input: The server receives the illustration data sent by the user.

[0385] Operation: The server receives the illustration data, stores it in a temporary storage area, and verifies the integrity of the file.

[0386] Data processing: Analyze the received data and apply integrity check algorithms.

[0387] Output: The illustration data whose consistency has been confirmed proceeds to the next processing step.

[0388] Step 3:

[0389] Input: Illustration data whose integrity has been confirmed.

[0390] How it works: The server analyzes the illustration data and extracts key features and contours using image analysis algorithms such as OpenCV.

[0391] Data processing: A feature extraction algorithm is run to obtain the features of the 2D illustration as data.

[0392] Output: The extracted feature data is used as input to a generative AI model.

[0393] Step 4:

[0394] Input: Extracted feature data.

[0395] How it works: The server invokes a generative AI model (e.g., DALL-E or Stable Diffusion) to generate a 3D model from a 2D illustration.

[0396] Data processing: Feature data is input into the generative AI model, and data calculations are performed to convert the two-dimensional illustration into a three-dimensional model.

[0397] Output: The generated 3D model data is saved in a database.

[0398] Step 5:

[0399] Input: Generated 3D model data.

[0400] What it does: The server notifies the user that generation is complete, including a confirmation link and a preview image.

[0401] Data processing: Generate notification data and send it to the user terminal.

[0402] Output: The user receives a link and a preview image to view the 3D model.

[0403] Step 6:

[0404] Input: The user inputs an action instruction (e.g., "run") in text format.

[0405] How it works: The user types instructions into the app's chat box and sends them to the server.

[0406] Data processing: The terminal collects text instructions from the user and converts them into a data format to be sent to the server.

[0407] Output: A text instruction is sent to the server.

[0408] Step 7:

[0409] Input: Text instruction data.

[0410] Operation: The server parses the received instruction data and uses Text2Motion technology to parse it.

[0411] Data processing: Use a natural language processing engine (e.g., GPT-3 (registered trademark)) to convert text instructions into concrete animation data.

[0412] Output: The generated animation data is saved in a database.

[0413] Step 8:

[0414] Input: Animation data.

[0415] Action: The server notifies the user that animation generation is complete.

[0416] Data processing: Generate notification data and send it to the user terminal.

[0417] Output: The user receives a notification that the animation data has been generated.

[0418] Step 9:

[0419] Input: Camera launch for user to set viewing position.

[0420] Action: The user activates the device's camera and drags the 3D model to place it in the real world.

[0421] Data processing: The device collects location data based on camera footage and converts it into a data format to be sent to the server.

[0422] Output: The location data is sent to the server.

[0423] Step 10:

[0424] Input: Location data.

[0425] Operation: The server analyzes the received location data and generates augmented reality data for displaying a three-dimensional model at the specified location.

[0426] Data processing: Generate data for augmented reality display based on location data.

[0427] Output: The generated augmented reality data is sent to the device.

[0428] Step 11:

[0429] Input: Augmented reality data.

[0430] How it works: The device uses the data it receives to display a 3D model in the real world.

[0431] Data processing: Using the device's AR engine, model data is overlaid onto the camera image.

[0432] Output: The user sees the 3D model displayed in the real world.

[0433] Step 12:

[0434] Input: User emotion data.

[0435] How it works: The device uses sensors (camera and microphone) to collect user emotional data in real time.

[0436] Data processing: The collected emotion data is initially analyzed on the device and converted into a data format that is sent to the server.

[0437] Output: Emotion data is sent to the server.

[0438] Step 13:

[0439] Input: Emotion data.

[0440] Operation: The server analyzes the emotion data to determine the user's emotional state.

[0441] Data Processing: An emotion engine is used to analyze the emotion data and identify the user's emotional state.

[0442] Output: The emotion analysis results are used as motion adjustment data for the 3D model.

[0443] Step 14:

[0444] Input: Sentiment analysis results.

[0445] Movement: The server adjusts the movement and animation of the 3D model based on the analysis results.

[0446] Data processing: Generate adjustment parameters and update the animation data of the 3D model.

[0447] Output: The adjusted animation data is sent to the device.

[0448] Step 15:

[0449] Input: The adjusted animation data.

[0450] Operation: The device updates the display of the 3D model based on the received data.

[0451] Data processing: Update the animation data in the device's AR engine to create new model behavior.

[0452] Output: The user can see a 3D model with movements adjusted according to the emotion.

[0453] (Application example 2)

[0454] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0455] Currently, operating and managing robots in factories requires complex procedures and specialized knowledge, making it difficult to provide effective operating instructions and maintenance information. Furthermore, there is a lack of support systems that take into account the emotions of workers, which can lead to a decline in operational efficiency and safety. In response to this situation, a more intuitive and effective robot management system is needed.

[0456] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for a user to upload image data from a terminal, means for the server to analyze the received image data and generate a 3D model from a 2D image, means for the server to animate the generated 3D model, means for the user to use the terminal to set a position at which the 3D model will be displayed in the real world as augmented reality, means for the server to acquire the set position information and display the 3D model at the specified position, means for acquiring and analyzing user emotion data using an emotion recognition engine, and means for adjusting the movement and animation of the 3D model based on the user emotion data. This not only enables intuitive and efficient management of robots in a factory but also enables appropriate support according to the emotions of workers.

[0457] Definition of terms

[0458] "User" refers to a person who uses the system to upload image data and generate three-dimensional models and set animations.

[0459] A "terminal" is any device operated by a user, including smartphones, tablets, and personal computers.

[0460] "Image data" refers to digital images such as two-dimensional illustrations and paintings uploaded by users.

[0461] "Server" refers to a central processing unit that analyzes image data, generates three-dimensional models, adds animation, and sets the position of the augmented reality display.

[0462] A "two-dimensional image" refers to a digital image that has a flat composition, usually having width and height.

[0463] A "3D model" refers to a three-dimensional digital object that has width, height, and depth in three-dimensional space.

[0464] "Animation" refers to the process of adding movement to a three-dimensional model to create motion.

[0465] "Augmented reality" refers to the technology of overlaying digital content onto the real world.

[0466] "Location information" refers to data that indicates where in the real world a three-dimensional model is to be displayed.

[0467] An "emotion recognition engine" refers to software or hardware that acquires and analyzes a user's emotional data and adjusts the system's behavior based on that information.

[0468] An "action" refers to a movement or motion instruction that a user specifies for a three-dimensional model.

[0469] MODE FOR CARRYING OUT THE INVENTION

[0470] This invention relates to a system that allows users to convert two-dimensional images into three-dimensional models, add movement, and display them in augmented reality. It also recognizes the user's emotions and enables the generation of personalized content.

[0471] Program Generation

[0472] 1. Ability for users to upload illustrations

[0473] A user uses a device to upload image data (e.g., a two-dimensional illustration). The device sends the image data to the server, which then stores the received image data in a temporary storage area. This allows the server to check the integrity of the illustration before proceeding to the next step.

[0474] 2. 3D conversion function for illustrations

[0475] The server analyzes the received image data and extracts key features and contours. Based on the analysis results, it creates initial data for generating a 3D model from the 2D image. It then invokes OpenAI's (registered trademark) generative AI model to convert the 2D illustration into a 3D model. The generated 3D model is saved in a database, and the user is notified when the conversion is complete.

[0476] 3. 3D model animation function

[0477] The user uses a terminal to input specific actions (e.g., "run" or "jump") for the 3D model. The terminal sends this to the server, which analyzes the instructions. Based on the analysis results, Text2Motion technology is applied to generate animation data, adding movement to the 3D model. The generated animation data is stored in a database, and the user is notified when it is complete.

[0478] 4. AR display position setting function

[0479] The user uses the device's camera function to determine the position where the 3D model will be displayed in the real world. The device then sends the configured augmented reality display position information to the server, which then analyzes the received position information and generates data to display the 3D model at the specified position. The device then displays the model as augmented reality based on the received data.

[0480] 5. Implementing the Emotion Engine

[0481] The device is equipped with sensors (e.g., facial recognition using a camera, voice analysis using a microphone) to acquire the user's emotions and uses these to collect emotional data. The collected emotional data is sent to a server, which analyzes the data using an emotion engine. The server adjusts the movement and animation of the 3D model based on the user's emotions and sends the adjustment results to the device. The device then updates the display of the 3D model based on this.

[0482] Hardware and software used

[0483] Hardware: smartphone, tablet, computer, camera, microphone

[0484] Software: OpenAI generated AI models, Text2Motion technology, emotion recognition software

[0485] Specific examples

[0486] A factory manager uploads a diagram of the robot's layout within the factory and issues the command "move to maintenance position." As a result, a 3D model of the robot is generated and an animation of it moving to the designated maintenance position is added. The user's device analyzes the emotional data, and if, for example, fatigue is detected, the operation interface is simplified. This improves work efficiency and safety.

[0487] Example prompts for generative AI models

[0488] "Convert the provided 2D layout image of the factory robots into a 3D model. Add the animation for the command "move to maintenance position"."

[0489] In this way, the present invention is a system that allows users to easily convert 2D illustrations into 3D models, animate them, and display them as augmented reality. Furthermore, by combining it with an emotion engine, it is possible to provide personalized content that responds to the user's emotions.

[0490] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0491] Program processing steps

[0492] Step 1:

[0493] Users upload illustrations

[0494] operation

[0495] The user opens the app on their device, selects the image data (e.g., a two-dimensional illustration) they want to upload from the gallery, and presses the "Upload" button to send the image data to the server.

[0496] input

[0497] Image data selected on the device

[0498] output

[0499] Notification of completion of uploading image data to the server

[0500] Data processing and calculation

[0501] Image data is sent from the terminal to the server, and the server stores the data in a temporary storage area. It checks the consistency and, if there are no problems, proceeds to the next processing step.

[0502] Step 2:

[0503] The server converts the illustration into 3D.

[0504] operation

[0505] The server analyzes the uploaded image data, extracts key features and contours, and generates a 3D model from the 2D image based on the analysis results.

[0506] input

[0507] Uploaded image data

[0508] output

[0509] Generated 3D model data

[0510] Data processing and calculation

[0511] The server applies its own image analysis algorithms to detect features in the 2D illustration, then invokes OpenAI's generative AI model to generate a 3D model using a prompt, which is then stored in a database.

[0512] Step 3:

[0513] Add animation to a 3D model

[0514] operation

[0515] The user inputs actions (e.g., "run" or "jump") for the three-dimensional model through the app on the device and sends them to the server.

[0516] input

[0517] User-specified actions (e.g., "run," "jump")

[0518] output

[0519] Animated 3D models

[0520] Data processing and calculation

[0521] The server analyzes the user's instructions and generates animation data for the specified actions using Text2Motion technology. The generated animation data is added to the 3D model and stored in a database.

[0522] Step 4:

[0523] Setting the AR display position

[0524] operation

[0525] The user uses the device's camera to determine where in the real world the 3D model will be displayed, and then drags and positions the model on the device screen.

[0526] input

[0527] Real-world images captured through a camera and a display position specified by the user

[0528] output

[0529] 3D model displayed at the specified position

[0530] Data processing and calculation

[0531] The device sends camera images to the server in real time, acquires the set location information, and sends it back to the server. The server analyzes the location information and generates data to display a 3D model at that location. The device displays the model at the specified location based on the received data.

[0532] Step 5:

[0533] Utilizing the Emotion Engine

[0534] operation

[0535] The device collects emotion data using sensors (e.g., facial recognition by a camera, voice analysis by a microphone) to obtain the user's emotion. The collected data is sent to a server.

[0536] input

[0537] Emotion data from cameras and microphones

[0538] output

[0539] 3D model movements and animations adjusted based on emotion data

[0540] Data processing and calculation

[0541] The server analyzes the emotion data using an emotion engine and adjusts the movements and animations of the 3D model based on the user's emotions. The adjustment results are sent to the terminal, and the display is updated.

[0542] The above is a series of processing steps from uploading image data to displaying a 3D model using emotion recognition.

[0543] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0544] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0545] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0546] [Second embodiment]

[0547] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0548] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0549] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0550] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0551] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0552] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0553] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0554] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0555] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0556] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0557] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0558] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0559] This invention relates to a system that allows anyone to easily convert two-dimensional images into three-dimensional models, add animation, and display them in augmented reality for sharing. This system has the function of allowing users to upload image data from their devices, and a server analyzes the data to generate a three-dimensional model and add animation. It also has the function of allowing users to set the augmented reality display position of the three-dimensional model using their devices, and the server acquires the location information and displays it at the specified location.

[0560] A natural language description of the program's operation

[0561] 1. Ability for users to upload illustrations

[0562] The user opens the app on their device, selects the illustration they want to upload from the gallery, and presses the "Upload" button to send the selected illustration to the server.

[0563] The server receives the uploaded illustration data and stores it in a temporary storage area. It checks the integrity of the file and, if there are no problems, proceeds to the next step.

[0564] 2. 3D conversion function for illustrations

[0565] The server analyzes the received illustration data, extracts key features and contours, and creates initial data for generating a 3D model from the 2D image based on the analysis results.

[0566] The server calls the generative AI model to convert the 2D illustration into a 3D model, which is then saved in a database and a notification is sent to the user when the conversion is complete.

[0567] 3. 3D model animation function

[0568] The user uses the chat box on the terminal to input actions (e.g., "walk," "jump," etc.) for the three-dimensional model and sends them to the server.

[0569] The server analyzes the user's instructions, generates the corresponding animation using Text2Motion technology, and adds it to the 3D model. The generated animation data is also saved in the database. Once completed, the server notifies the user of the results.

[0570] 4. AR display position setting function

[0571] The user uses the device's camera function to determine where in the real world the 3D model should be displayed, specifically by dragging and positioning the 3D model on the device screen.

[0572] The terminal transmits the set position information to the server, and the server analyzes the information to determine the display position of the three-dimensional model in the real world.

[0573] The server sends data for displaying the 3D model at the specified location to the terminal, and the terminal displays the 3D model at the specified location as augmented reality based on the received data.

[0574] Specific examples

[0575] When a child uploads an illustration of a character they've drawn and commands "run" via chat, the character is converted into a three-dimensional model and a running animation is added. The user can then use their device's camera to position the character in their living room. This positioning information is sent to the server, and the character is displayed in augmented reality, running at the specified location in the living room. The user can view this display or share it with others.

[0576] As described above, the present invention relates to a system that allows users to easily convert two-dimensional illustrations into three-dimensional models, animate them, and display them as augmented reality. This system can provide creative activities and new communication methods.

[0577] The processing flow will be explained below.

[0578] Specific processing flow of the program

[0579] 1. Ability for users to upload illustrations

[0580] Step 1:

[0581] The user opens the app on their device and selects the illustration they want to upload from the gallery.

[0582] When the user presses the "Upload" button, the terminal sends the selected illustration to the server.

[0583] Step 2:

[0584] The server receives the transmitted illustration data and stores it in a temporary storage area.

[0585] The server performs a file integrity check to ensure the file was uploaded successfully.

[0586] Step 3:

[0587] The server analyzes the quality of the uploaded illustrations and, if necessary, notifies the user of suggestions for improving the quality.

[0588] 2. 3D conversion function for illustrations

[0589] Step 1:

[0590] The server analyzes the received illustration data and extracts key features and contours, allowing each part of the illustration to be recognized.

[0591] Step 2:

[0592] Based on the analysis results, the server creates initial data for generating a three-dimensional model from the two-dimensional image.

[0593] Step 3:

[0594] The server invokes a generative AI model (e.g., generative AI model technology) to convert the 2D illustration into a 3D model.

[0595] The server stores the generated 3D model in a database.

[0596] Step 4:

[0597] The server notifies the user when the 3D rendering is complete and provides a link or preview for confirmation.

[0598] 3. 3D model animation function

[0599] Step 1:

[0600] The user opens a chat box through the device app and inputs actions (e.g., "run," "jump," etc.) for the three-dimensional model.

[0601] When the user sends an instruction, the terminal sends the instruction to the server.

[0602] Step 2:

[0603] The server receives instructions from the user and analyzes the content of the instructions. For example, if the instruction to "run" is sent, the server analyzes the action.

[0604] Step 3:

[0605] The server uses Text2Motion technology to generate animation data for adding movement to the three-dimensional model based on the analyzed instructions.

[0606] Step 4:

[0607] The server saves the generated animation data in the 3D model database and notifies the user when it is complete.

[0608] 4. AR display position setting function

[0609] Step 1:

[0610] The user uses the device's camera to determine where in the real world the 3D model should be displayed, and then drags and positions the 3D model on the device screen.

[0611] Step 2:

[0612] The terminal transmits the set position information of the augmented reality display to the server.

[0613] Step 3:

[0614] The server analyzes the received location information and generates data for displaying the three-dimensional model at the specified location.

[0615] Step 4:

[0616] The server transmits this data to the terminal, and the terminal displays the three-dimensional model as augmented reality at the specified position based on the received data.

[0617] For example, a child can upload an illustration of a character they have drawn and use chat to say "run." As a result, the character is generated as a 3D model, a running animation is added, and it is displayed as augmented reality on the living room floor of the user's home. The user can then use their device's camera to watch the character running around the living room.

[0618] Example 1

[0619] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0620] With conventional technologies, converting 2D images into 3D models, adding movement, and displaying them in augmented reality is extremely difficult and requires specialized knowledge and time. Furthermore, there is a lack of easy ways for users to set their desired actions and location information, creating a demand for intuitive and easy operation. This creates a need for an efficient system for creating and sharing creative content.

[0621] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0622] In this invention, the server includes means for a user to upload image data from a terminal, means for analyzing the image data received by the server and generating a 3D model from a 2D image, means for converting the 2D image into a 3D model using a generative AI model, means for adding animation to the 3D model based on an action specified by the user, means for the user to use the terminal to set a position where the 3D model is to be displayed as augmented reality in the real world, and means for the server to acquire the set position information and display the 3D model at the specified position. This allows users to easily convert 2D illustrations into 3D models, add animation, and display and share them as augmented reality.

[0623] A "user" is any individual or organization who wishes to use the system to convert two-dimensional images into three-dimensional models and display them as augmented reality.

[0624] "Terminal" refers to a device operated by a user to upload image data, place 3D models, and specify actions. Specifically, this includes smartphones, tablets, and PCs.

[0625] A "server" is a computer or network system that receives and analyzes data sent from a terminal, generates a three-dimensional model from a two-dimensional image, adds actions, and performs calculations to display it as augmented reality.

[0626] "Image data" is digital data containing visual information such as two-dimensional illustrations and photographs that are uploaded by users from their terminals.

[0627] "Analysis" is the process by which the server understands the structure and characteristics of the image data it receives and extracts the necessary information.

[0628] A "two-dimensional image" is flat image data that has two dimensions, vertical and horizontal.

[0629] A "three-dimensional model" is digital data that has three dimensions - length, width, and depth - and can be displayed three-dimensionally in a virtual space.

[0630] "Generative AI model" refers to an algorithm or technique that uses artificial intelligence to convert two-dimensional images into three-dimensional models.

[0631] "Animation" is digital data that adds movement to a three-dimensional model and makes it operate continuously.

[0632] "Text2Motion technology" is a technology that analyzes text instructions entered by the user and adds corresponding movements to a three-dimensional model.

[0633] "Augmented reality (AR)" is a technology that displays computer-generated visual information overlaid on a real-world environment.

[0634] "Location information" is digital data relating to a location designated by a user to display a three-dimensional model in the real world.

[0635] A "database" is a storage device or system for storing and managing three-dimensional models and animation data.

[0636] The present invention relates to a system that allows users to easily convert two-dimensional images into three-dimensional models, add motion, and display and share them as augmented reality. The system has several main functions, each of which is realized by specific hardware and software.

[0637] The system's hardware includes terminal devices such as smartphones, tablets, and PCs, as well as a server, while the software uses the generative AI model "DeepArt3D" and Text2Motion technology.

[0638] Ability for users to upload illustrations

[0639] Users launch the application using a terminal device such as a smartphone or PC and select the 2D illustration they want to upload from local storage or a gallery. The selected illustration is sent from the device to the server. The server receives the uploaded illustration data and stores it in a temporary storage area. Once the integrity of the file is confirmed, the process proceeds to the next step.

[0640] 3D conversion function for illustrations

[0641] The server analyzes the received illustration data and uses the DeepArt3D model to generate a 3D model from the 2D image. First, it extracts the main features and contours of the illustration and creates initial data for the 3D model based on them. Next, the DeepArt3D model uses this initial data to generate the 3D model. The generated 3D model is saved in a database, and the user is notified when the conversion is complete.

[0642] 3D model animation function

[0643] The user opens the chat box on their device and inputs actions for the 3D model (such as "walk" or "jump") in text format. The input instructions are sent to the server, which uses Text2Motion technology to generate the corresponding animation and add it to the 3D model. The generated animation data is saved in a database, and the user is notified when the animation has been added.

[0644] For example, if a user inputs "Make this character run," the server will analyze this prompt and generate and add a "running" animation using Text2Motion technology. This method allows users to easily implement their desired movements in a 3D model.

[0645] AR display position setting function

[0646] The user activates the device's camera and decides where in the real world to display the 3D model. Specifically, they drag and drop the 3D model onto the device screen to place it. The device then sends this location information to the server, which analyzes it and determines where to display the 3D model. The server then sends the necessary data to the device, and the device displays the 3D model at the specified location as augmented reality.

[0647] For example, if a child uploads an illustration of a character and commands "run" via chat, the character is converted into a three-dimensional model and a running animation is added. Furthermore, the user can use the device's camera function to position the character in their living room. This positioning information is sent to the server, and the character is displayed in augmented reality as if running at the specified location in the living room.

[0648] As described above, the present invention relates to a system that allows users to intuitively operate it, quickly and easily convert two-dimensional illustrations into three-dimensional models, and then use them as augmented reality. This system provides new ways of creative activities and communication, and greatly enhances user convenience.

[0649] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0650] Step 1:

[0651] The user launches the application using their device. The user selects the 2D illustration they want to upload from local storage or the gallery. The selected illustration data is sent from the device to the server by pressing the "Upload" button.

[0652] Input: 2D illustration data selected from local storage or gallery

[0653] Output: Illustration data sent to the server

[0654] Step 2:

[0655] The server receives the uploaded illustration data and stores it in a temporary storage area. The server then checks the integrity of the file and, if there are no problems, begins analysis.

[0656] Input: Received illustration data

[0657] Output: Illustration data with consistency confirmed

[0658] Step 3:

[0659] The server analyzes the received illustration data, extracting the main features and contours of the image and creating initial data for generating a 3D model from the 2D image.

[0660] Input: Illustration data that has passed the consistency check

[0661] Output: Initial data including key feature and contour extraction results

[0662] Step 4:

[0663] The server calls the generative AI model "DeepArt3D" and converts the 2D illustration into a 3D model using the initial data. At this time, the server constructs the model and adds texture.

[0664] Input: Initial data

[0665] Output: Generated 3D model

[0666] Step 5:

[0667] The server stores the generated 3D model in a database and also notifies the user that the conversion of the 3D model is complete.

[0668] Input: Generated 3D model

[0669] Output: 3D model stored in database, notification to user

[0670] Step 6:

[0671] The user opens a chat box on the terminal and inputs text for actions to be performed on the 3D model (for example, "walk," "jump," etc.). The input instructions are sent from the terminal to the server.

[0672] Input: User instruction text

[0673] Output: Instruction text sent to the server

[0674] Step 7:

[0675] The server analyzes the user's text instructions and generates the corresponding animation using Text2Motion technology, a process that adds movement to the 3D model based on the user's instructions.

[0676] Input: Instruction text

[0677] Output: An animated 3D model

[0678] Step 8:

[0679] The server stores the animated 3D model in a database and also notifies the user that the animation has been completed.

[0680] Input: An animated 3D model

[0681] Output: Animated 3D model stored in database, notification to user

[0682] Step 9:

[0683] The user activates the device's camera function and sets the location in the real world where they want to display the 3D model on the device screen. Specifically, the user drags and drops the 3D model onto the device screen to place it. The set location information is sent to the server.

[0684] Input: Set location information

[0685] Output: Location information sent to the server

[0686] Step 10:

[0687] The server analyzes the received location information and determines the display position of the 3D model in the real world. It then transmits the necessary data to the device and displays the 3D model in the specified position as augmented reality.

[0688] Input: Location

[0689] Output: Display data of a 3D model of the real world generated from engine data

[0690] Step 11:

[0691] The device uses the received display data to display the 3D model as augmented reality at a specified location in the real world, allowing the user to view and share the display with others.

[0692] Input: Display data

[0693] Output: Augmented reality display of 3D model in the real world

[0694] (Application example 1)

[0695] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0696] Conventional systems that convert 2D images into 3D models and add animation have had the problem of being complex to operate, making them difficult for users to operate intuitively. Furthermore, when displaying the generated 3D model in the real world, it is difficult for users to set the exact display position, which is time-consuming. Furthermore, in advertising and promotions, there is a lack of user-participation interactive elements, and there is a lack of a way to easily create visually appealing content.

[0697] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0698] In this invention, the server includes: means for a user to upload image data from a terminal; means for the server to analyze the received image data and generate a three-dimensional model from a two-dimensional image; means for the server to animate the generated three-dimensional model; means for the user to set a position in the real world where the three-dimensional model will be displayed as augmented reality using the terminal; means for the server to acquire the set position information and display the three-dimensional model at the specified position; and means for analyzing the user's action instructions regarding the three-dimensional model and generating corresponding animations. This allows a user to generate a three-dimensional model from a two-dimensional image with simple operations, add animation to the model, and display it in augmented reality. Furthermore, incorporating interactive elements makes it easy to create visually appealing advertisements and promotional content.

[0699] "User" refers to the entity that uses the system to upload image data, generate three-dimensional models, add animations, and set display positions as augmented reality.

[0700] A "terminal" is a device used by a user, and refers to an electronic device such as a smartphone, smart glasses, or a head-mounted display.

[0701] "Image data" refers to two-dimensional visual information uploaded by a user through a terminal.

[0702] A "server" refers to a computing device that receives image data from a terminal, analyzes it, generates a three-dimensional model from the two-dimensional image, adds animation, and notifies the user of the results.

[0703] A "two-dimensional image" refers to image information that is composed of two dimensions, vertical and horizontal.

[0704] A "three-dimensional model" refers to a three-dimensional model expressed in three dimensions: length, width, and height.

[0705] "Animation" refers to a technique of expression that gives the visual impression of movement by adding movement to a three-dimensional model.

[0706] "Augmented reality" refers to the technology of overlaying digital information (in this case, three-dimensional models) onto a real-world environment.

[0707] "Location information" refers to data that allows a user to use a device to set where in the real world a three-dimensional model is to be displayed.

[0708] An "action instruction" refers to input information instructing a three-dimensional model to perform a specific action or movement.

[0709] The present invention provides a system that allows a user to convert a two-dimensional image into a three-dimensional model, add movement to the model, and display it in augmented reality. The following description will explain in detail how to specifically implement the present invention.

[0710] 1. System Configuration

[0711] The system for implementing the present invention mainly uses the following hardware and software.

[0712] Hardware:

[0713] User devices: smartphones (Android or iOS), smart glasses, head-mounted displays

[0714] Server: A cloud-based computing device (such as AWS or Google Cloud)

[0715] software:

[0716] Smartphone applications: Kotlin for Android and Swift for iOS

[0717] Server software: Python, TensorFlow for generative AI models, OpenCV for image analysis

[0718] Database: PostgreSQL

[0719] 2. Program Processing Overview

[0720] The program on the server performs the following process.

[0721] 1. Ability for users to upload illustrations:

[0722] The user launches the app on their smartphone, selects the 2D image they want to upload from their gallery, and presses the "Upload" button.

[0723] The server receives the uploaded image data and stores it in a temporary storage area. It checks the integrity of the data and, if there are no problems, proceeds to the next step.

[0724] 2. 3D conversion function for illustrations:

[0725] The server analyzes the received image data, extracts key features and contours, and creates initial data for generating a 3D model from the 2D image based on the analysis results.

[0726] The server invokes the generative AI model to convert the 2D image into a 3D model, which is then stored in a database and a notification is sent to the user when the conversion is complete.

[0727] 3. 3D model animation features:

[0728] The user uses the chat box on their smartphone to instruct the 3D model to take actions (e.g., "spin," "jump," "glow," etc.) and sends the instructions to the server.

[0729] The server analyzes the user's instructions, generates the corresponding animation using Text2Motion technology, and adds it to the 3D model. The generated animation data is also saved in a database, and the user is notified of the results once the process is complete.

[0730] 4. AR display position setting function:

[0731] The user uses the smartphone's camera to decide where in the real world the 3D model should be displayed, by dragging and positioning the 3D model on the device screen.

[0732] The terminal transmits the set position information to the server, and the server analyzes the information to determine the display position of the three-dimensional model in the real world.

[0733] The server transmits data for displaying a three-dimensional model at a specified position to the terminal, and the terminal displays the three-dimensional model at the specified position as augmented reality based on the received data.

[0734] Specific examples

[0735] Suppose a company wants to run an advertising campaign. A company representative opens a smartphone app and uploads a promotional company logo. Then, they type "jump" into the chat box. The server analyzes the logo, converts it into a 3D model, and adds a jumping animation to the model. The representative then places a promotional character on a cafe table, which is displayed in the real world as augmented reality. Examples of prompts used in this process include "upload an image file selected by the user to the server," "extract key features from the 2D logo and convert it into a 3D model," and "add a rotation animation to the 3D model."

[0736] Through the above process, users can intuitively and simply convert two-dimensional images into three-dimensional models, add movement to the models, and display them in the real world as augmented reality, making it possible to generate visually appealing advertisements and promotional content.

[0737] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0738] Program processing steps

[0739] Step 1:

[0740] The user launches the smartphone app, selects a two-dimensional image (e.g., a company logo) from the gallery, and presses the "Upload" button.

[0741] Input: 2D image data from the user's terminal

[0742] Operation: The terminal sends the selected image data to the server.

[0743] Output: 2D image data uploaded to the server

[0744] Step 2:

[0745] The server analyzes the received image data and stores it in a temporary storage area. It checks the data for consistency and, if there are no problems, it proceeds to the next step.

[0746] Input: Received image data

[0747] Action: Check the data integrity and save it to a temporary storage area.

[0748] Output: Image data with integrity confirmed

[0749] Step 3:

[0750] The server extracts the main features and contours of the received image data, and based on the analysis results, creates initial data for generating a 3D model from the 2D image.

[0751] Input: Image data whose integrity has been verified

[0752] How it works: Feature extraction is performed using OpenCV.

[0753] Output: Key features and contours

[0754] Step 4:

[0755] The server invokes a generative AI model (TensorFlow) to convert the 2D image into a 3D model, which is then saved in a database and a notification is sent to the user when the conversion is complete.

[0756] Input: feature and contour data

[0757] How it works: Generates a 3D model using a TensorFlow model and stores it in a database

[0758] Output: 3D model data, conversion completion notification

[0759] Step 5:

[0760] The user uses the chat box on their smartphone to instruct the 3D model to take an action (e.g., "spin," "jump," "glow") and send it to the server.

[0761] Input: User instruction for action

[0762] Action: Enter instructions in the chat box and send them to the server

[0763] Output: Action instructions sent to the server

[0764] Step 6:

[0765] The server analyzes the user's instructions, generates the corresponding animation using Text2Motion technology, and adds it to the 3D model. This animation data is also stored in the database, and the result is notified to the user after completion.

[0766] Input: User instruction for action

[0767] Action: Animation is generated using Text2Motion technology and saved in the database

[0768] Output: 3D model data with animation, notification of completion of generation

[0769] Step 7:

[0770] Users use the smartphone's camera to decide where in the real world they want to display the 3D model, then drag and place the model on the device screen.

[0771] Input: User's display position setting information

[0772] Operation: Set the display position using the camera function and send the position information to the server

[0773] Output: Display position information sent to the server

[0774] Step 8:

[0775] The server analyzes the location information set by the user and determines the display position of the 3D model in the real world. The server then sends data to the device to display the 3D model at the specified position.

[0776] Input: Display location information

[0777] Operation: Analyzes location information, generates data for augmented reality display, and sends it to the device

[0778] Output: Augmented reality display data

[0779] Step 9:

[0780] Based on the data received from the server, the device displays a three-dimensional model as augmented reality at a specified location.

[0781] Input: Augmented reality display data

[0782] Operation: A 3D model is superimposed on the camera image based on the received data.

[0783] Output: 3D model displayed as augmented reality in the real world

[0784] The above are the processing steps for carrying out the invention.

[0785] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0786] The present invention relates to a system that allows anyone to easily convert two-dimensional images into three-dimensional models, add movement, and display them in augmented reality for sharing. The present invention enables the generation of more personalized content by combining an emotion engine that recognizes user emotions. This system has a function that allows users to upload image data from their devices, and a server analyzes the data to generate a three-dimensional model and add animation. The system also has a function that allows users to use their devices to set the augmented reality display position of the three-dimensional model, and the server acquires that position information and displays it at the specified location. Additionally, the emotion engine can recognize user emotions and change behavior or adjust animations based on those emotions.

[0787] A natural language description of the program's operation

[0788] 1. Ability for users to upload illustrations

[0789] The user opens the app on their device, selects the illustration they want to upload from the gallery, and presses the "Upload" button to send the selected illustration to the server.

[0790] The server receives the illustration data and stores it in a temporary storage area. It checks the integrity of the file and, if there are no problems, proceeds to the next step.

[0791] 2. 3D conversion function for illustrations

[0792] The server analyzes the received illustration data, extracts key features and contours, and creates initial data for generating a 3D model from the 2D image based on the analysis results.

[0793] The server calls the generative AI model and converts the 2D illustration into a 3D model, which is then stored in a database.

[0794] The server notifies the user when the 3D rendering is complete and provides a link or preview for confirmation.

[0795] 3. 3D model animation function

[0796] The user opens a chat box through the app on their device, inputs actions for the three-dimensional model (e.g., "run," "jump," etc.), and sends them to the server.

[0797] The server receives instructions from the user and analyzes the content of the instructions. For example, if the instruction to "run" is sent, the server analyzes the action.

[0798] The server uses Text2Motion technology to generate animation data for adding movement to the three-dimensional model based on the analyzed instructions.

[0799] The server saves the generated animation data in the 3D model database and notifies the user when it is complete.

[0800] 4. AR display position setting function

[0801] The user uses the device's camera to determine where in the real world the 3D model should be displayed, and then drags and positions the 3D model on the device screen.

[0802] The terminal transmits the set position information of the augmented reality display to the server.

[0803] The server analyzes the received location information and generates data for displaying the three-dimensional model at the specified location.

[0804] The server transmits this data to the terminal, and the terminal displays the three-dimensional model as augmented reality at the specified position based on the received data.

[0805] 5. Implementing the Emotion Engine

[0806] The device is equipped with sensors (e.g., facial recognition using a camera, voice analysis using a microphone) to acquire the user's emotions, and uses these to collect the user's emotional data.

[0807] The device transmits emotion data to the server, which then analyzes the data using an emotion engine.

[0808] The server adjusts the behavior and animation of the 3D model based on the user's emotions, for example, making the model more active if the user is happy, or slowing down if the user is sad.

[0809] The server transmits the adjustment results to the terminal, and the terminal updates the display of the three-dimensional model based on the results.

[0810] Specific examples

[0811] A child uploads an illustration of a character they have drawn and commands "run" via chat. As a result, the character is generated as a 3D model and a running animation is added. The user then uses the camera function of their device to place the character on the floor of their living room. This placement information is sent to the server, and the character is displayed in augmented reality as if running at the specified location in the living room.

[0812] Furthermore, the user's device uses a camera to analyze their facial expressions, and if the user is having fun, the character will adjust to run more energetically. When the device detects that the user is happy, the server's emotion engine activates the character's movements, realizing behavior that matches the user's emotions.

[0813] As described above, the present invention relates to a system that allows users to easily convert 2D illustrations into 3D models, animate them, and display them as augmented reality. Furthermore, by combining this with an emotion engine, it is possible to provide personalized content that responds to the user's emotions.

[0814] The processing flow will be explained below.

[0815] Specific processing flow of the program (when combined with an emotion engine)

[0816] 1. Ability for users to upload illustrations

[0817] Step 1:

[0818] The user opens the app on their device and selects the illustration they want to upload from the gallery.

[0819] When the user presses the "Upload" button, the terminal sends the selected illustration to the server.

[0820] Step 2:

[0821] The server receives the transmitted illustration data and stores it in a temporary storage area.

[0822] The server performs a file integrity check to ensure the file was uploaded successfully.

[0823] Step 3:

[0824] The server analyzes the quality of the uploaded illustration and, if any parts are distorted or of low resolution, notifies the user with suggestions for correcting them.

[0825] 2. 3D conversion function for illustrations

[0826] Step 1:

[0827] The server analyzes the received illustration data and extracts key features and contours, which correspond to each part of the 2D illustration.

[0828] Step 2:

[0829] Based on the analysis results, the server extracts the initial data necessary to generate a basic three-dimensional model.

[0830] Step 3:

[0831] The server invokes a generative AI model to convert the 2D illustration into a 3D model, which is made up of data including shape and texture.

[0832] The server stores the generated 3D model in a database and notifies the user that the 3D model generation is complete.

[0833] 3. 3D model animation function

[0834] Step 1:

[0835] The user opens a chat box through the device app and inputs actions (e.g., "run," "jump," etc.) for the three-dimensional model.

[0836] When the user sends an instruction, the terminal sends the instruction to the server.

[0837] Step 2:

[0838] The server receives instructions from the user and analyzes the content of the instructions. For example, if the instruction to "run" is sent, the server analyzes the action in detail.

[0839] Step 3:

[0840] The server uses Text2Motion technology to generate animation data for adding movement to the three-dimensional model based on the analyzed instructions.

[0841] Step 4:

[0842] The server saves the generated animation data in the 3D model database and notifies the user when it is complete.

[0843] 4. AR display position setting function

[0844] Step 1:

[0845] The user uses the device's camera to determine where in the real world the 3D model should be displayed, and then drags and positions the 3D model on the device screen.

[0846] Step 2:

[0847] The terminal transmits the set position information of the augmented reality display to the server.

[0848] Step 3:

[0849] The server analyzes the received location information and generates data for displaying the three-dimensional model at the specified location.

[0850] Step 4:

[0851] The server transmits this data to the terminal, and the terminal displays the three-dimensional model as augmented reality at the specified position based on the received data.

[0852] 5. Implementing the Emotion Engine

[0853] Step 1:

[0854] The device is equipped with sensors (e.g., facial recognition using a camera, voice analysis using a microphone) to acquire the user's emotions, and uses these to collect the user's emotional data.

[0855] Step 2:

[0856] The device sends emotion data to the server, which then uses an emotion engine to analyze the data, for example, recognizing whether the user is surprised or happy through facial expression analysis.

[0857] Step 3:

[0858] The server adjusts the behavior and animation of the 3D model based on the user's emotions: for example, if the user is having fun, the model's movements will be more active, and conversely, if the user is sad, the model's movements will be more gentle.

[0859] Step 4:

[0860] The server transmits the adjustment results to the terminal, and the terminal updates the display of the three-dimensional model based on the results.

[0861] Specific examples

[0862] As a concrete example, a child uploads an illustration of a character they have drawn and commands "run" via chat. As a result, the character is generated as a 3D model and a running animation is added. The user then uses the camera function of their device to place the character on the floor of their living room. This placement information is sent to the server, and the character is displayed in augmented reality as if running at the specified location in the living room.

[0863] Furthermore, the user's device uses a camera to analyze their facial expressions, and if the user is having fun, the character will adjust to run more energetically. When the device detects that the user is happy, the server's emotion engine activates the character's movements, realizing behavior that matches the user's emotions.

[0864] Example 2

[0865] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0866] Currently, converting two-dimensional illustrations into three-dimensional models, adding movement to those models, and displaying them as augmented reality requires extensive specialized knowledge and complex software. These operations are difficult for ordinary users to perform due to the high technical barriers. Furthermore, the content created is rarely properly personalized based on the user's emotions. This limits the user experience and makes it difficult to create more engaging content.

[0867] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for a user to upload image data from a terminal, a means for the server to analyze the image data received and generate a three-dimensional model from a two-dimensional image, and a means for animating the three-dimensional model generated by the server. This allows a user, without specialized knowledge, to intuitively convert a two-dimensional illustration into a three-dimensional model, add animation, and display it as augmented reality. Furthermore, the terminal includes a means for acquiring the user's emotions and adjusting the behavior of the three-dimensional model based on the emotions, thereby enabling the generation of more personalized and attractive content.

[0868] A "user" is an entity that uses the system to upload image data and generate three-dimensional models and display augmented reality.

[0869] A "terminal" is an electronic device that a user uses to upload image data, such as a smartphone or tablet.

[0870] "Image data" refers to two-dimensional illustrations and image files uploaded by users.

[0871] A "server" is a computer system that analyzes received image data, generates three-dimensional models, animates them, and manages the augmented reality display.

[0872] "Analysis" is the process of extracting features and contours from received image data, which serve as the basis for generating a three-dimensional model from the two-dimensional image.

[0873] A "two-dimensional image" refers to a flat image that has only width and height information.

[0874] A "3D model" is a three-dimensional digital object that has width, height, and depth information.

[0875] "Animation" refers to the process of adding movement to a three-dimensional model.

[0876] "Augmented reality" refers to a technology that overlays digital information onto images of the real world.

[0877] "Location information" is data that indicates where in the real world a three-dimensional model is to be displayed.

[0878] "Emotion" refers to the psychological and sensory state that can be read from the user's facial expressions, voice, etc.

[0879] "Adjusting the behavior" refers to changing the animation of the three-dimensional model according to the user's emotions.

[0880] The present invention relates to a system that allows a user to easily convert a two-dimensional illustration into a three-dimensional model, add movement to the model, and display it as an augmented reality. An embodiment of the system will be described in detail below.

[0881] 1. Ability for users to upload illustrations

[0882] Users launch the dedicated application using a device such as a smartphone or tablet. They tap the "Select from Gallery" button within the application to open the device's image gallery and select the illustration they wish to upload. The user then presses the "Upload" button, which sends the illustration data to the server. The server receives the illustration data and stores it in a temporary storage area. At this time, the integrity of the file is also checked.

[0883] 2. 3D conversion function for illustrations

[0884] The server analyzes the received illustration data and extracts key features and contours. This process uses image analysis algorithms such as OpenCV. Based on the analysis results, the server invokes a generative AI model (e.g., DALL-E or Stable Diffusion) to convert the 2D illustration into a 3D model. The generated 3D model is stored in a database, and the user is notified when generation is complete. The notification includes a confirmation link and a preview image.

[0885] 3. 3D model animation function

[0886] The user inputs action instructions (e.g., "run," "jump," etc.) for the 3D model in the app's chat box and sends them to the server. The server analyzes the received instructions and generates animation data using Text2Motion technology. This animation data is also stored in the database, and the user is notified when generation is complete.

[0887] 4. AR display position setting function

[0888] The user activates the device's camera function and drags the 3D model on the screen to determine where in the real world the model should be displayed. The location information is sent to the server, which analyzes the received location information and generates augmented reality data for displaying the 3D model at the specified location. The generated data is sent to the device, which then displays the 3D model in the real world based on the received data.

[0889] 5. Implementing the Emotion Engine

[0890] The device uses sensors such as a camera and microphone to collect the user's emotional data in real time. For example, the camera recognizes the user's face and uses software (e.g., Emotion API) to analyze their facial expressions. The collected emotional data is sent to a server, which then analyzes it using an emotion engine. Based on the analysis results, the server adjusts the movement and animation of the 3D model and sends the results to the device. This causes the model's movement to change according to the user's emotions.

[0891] Specific examples

[0892] For example, a child uploads an illustration of a character they drew from their device and commands "run" in the chat box. In this case, the server generates a 3D model and adds a running animation. The user then uses the device's camera to place the character on the living room floor. Once the placement information is sent to the server, the character is displayed in augmented reality as if running in the specified position. Furthermore, if the device detects the user's facial expression of enjoyment, the server's emotion engine actively adjusts the character's movements.

[0893] Prompt Sentence Examples

[0894] "Turn your drawn character into a 3D model and run it around your living room."

[0895] By implementing these functions, the system of the present invention enables users to intuitively convert 2D illustrations into 3D models, animate them, and display them as augmented reality without requiring specialized knowledge. It also provides personalized content based on the user's emotions using an emotion engine.

[0896] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0897] Step 1:

[0898] Input: The user launches the app on their device and selects the 2D illustration to upload.

[0899] How it works: A user taps the "Select from Gallery" button in the app, opens the image gallery, selects the illustration they want to upload, and then presses the "Upload" button.

[0900] Data processing: The terminal prepares the selected illustration data and encodes the image data for transmission to the server.

[0901] Output: The encoded image data is sent to the server.

[0902] Step 2:

[0903] Input: The server receives the illustration data sent by the user.

[0904] Operation: The server receives the illustration data, stores it in a temporary storage area, and verifies the integrity of the file.

[0905] Data processing: Analyze the received data and apply integrity check algorithms.

[0906] Output: The illustration data whose consistency has been confirmed proceeds to the next processing step.

[0907] Step 3:

[0908] Input: Illustration data whose integrity has been confirmed.

[0909] How it works: The server analyzes the illustration data and extracts key features and contours using image analysis algorithms such as OpenCV.

[0910] Data processing: A feature extraction algorithm is run to obtain the features of the 2D illustration as data.

[0911] Output: The extracted feature data is used as input to a generative AI model.

[0912] Step 4:

[0913] Input: Extracted feature data.

[0914] How it works: The server invokes a generative AI model (e.g., DALL-E or Stable Diffusion) to generate a 3D model from a 2D illustration.

[0915] Data processing: Feature data is input into the generative AI model, and data calculations are performed to convert the two-dimensional illustration into a three-dimensional model.

[0916] Output: The generated 3D model data is saved in a database.

[0917] Step 5:

[0918] Input: Generated 3D model data.

[0919] What it does: The server notifies the user that generation is complete, including a confirmation link and a preview image.

[0920] Data processing: Generate notification data and send it to the user terminal.

[0921] Output: The user receives a link and a preview image to view the 3D model.

[0922] Step 6:

[0923] Input: The user inputs an action instruction (e.g., "run") in text format.

[0924] How it works: The user types instructions into the app's chat box and sends them to the server.

[0925] Data processing: The terminal collects text instructions from the user and converts them into a data format to be sent to the server.

[0926] Output: A text instruction is sent to the server.

[0927] Step 7:

[0928] Input: Text instruction data.

[0929] Operation: The server parses the received instruction data and uses Text2Motion technology to parse it.

[0930] Data processing: Use a natural language processing engine (e.g., GPT-3) to convert text instructions into concrete animation data.

[0931] Output: The generated animation data is saved in a database.

[0932] Step 8:

[0933] Input: Animation data.

[0934] Action: The server notifies the user that animation generation is complete.

[0935] Data processing: Generate notification data and send it to the user terminal.

[0936] Output: The user receives a notification that the animation data has been generated.

[0937] Step 9:

[0938] Input: Camera launch for user to set viewing position.

[0939] Action: The user activates the device's camera and drags the 3D model to place it in the real world.

[0940] Data processing: The device collects location data based on camera footage and converts it into a data format to be sent to the server.

[0941] Output: The location data is sent to the server.

[0942] Step 10:

[0943] Input: Location data.

[0944] Operation: The server analyzes the received location data and generates augmented reality data for displaying a three-dimensional model at the specified location.

[0945] Data processing: Generate data for augmented reality display based on location data.

[0946] Output: The generated augmented reality data is sent to the device.

[0947] Step 11:

[0948] Input: Augmented reality data.

[0949] How it works: The device uses the data it receives to display a 3D model in the real world.

[0950] Data processing: Using the device's AR engine, model data is overlaid onto the camera image.

[0951] Output: The user sees the 3D model displayed in the real world.

[0952] Step 12:

[0953] Input: User emotion data.

[0954] How it works: The device uses sensors (camera and microphone) to collect user emotional data in real time.

[0955] Data processing: The collected emotion data is initially analyzed on the device and converted into a data format that is sent to the server.

[0956] Output: Emotion data is sent to the server.

[0957] Step 13:

[0958] Input: Emotion data.

[0959] Operation: The server analyzes the emotion data to determine the user's emotional state.

[0960] Data Processing: An emotion engine is used to analyze the emotion data and identify the user's emotional state.

[0961] Output: The emotion analysis results are used as motion adjustment data for the 3D model.

[0962] Step 14:

[0963] Input: Sentiment analysis results.

[0964] Movement: The server adjusts the movement and animation of the 3D model based on the analysis results.

[0965] Data processing: Generate adjustment parameters and update the animation data of the 3D model.

[0966] Output: The adjusted animation data is sent to the device.

[0967] Step 15:

[0968] Input: The adjusted animation data.

[0969] Operation: The device updates the display of the 3D model based on the received data.

[0970] Data processing: Update the animation data in the device's AR engine to create new model behavior.

[0971] Output: The user can see a 3D model with movements adjusted according to the emotion.

[0972] (Application example 2)

[0973] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0974] Currently, operating and managing robots in factories requires complex procedures and specialized knowledge, making it difficult to provide effective operating instructions and maintenance information. Furthermore, there is a lack of support systems that take into account the emotions of workers, which can lead to a decline in operational efficiency and safety. In response to this situation, a more intuitive and effective robot management system is needed.

[0975] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for a user to upload image data from a terminal, means for the server to analyze the received image data and generate a 3D model from a 2D image, means for the server to animate the generated 3D model, means for the user to use the terminal to set a position at which the 3D model will be displayed in the real world as augmented reality, means for the server to acquire the set position information and display the 3D model at the specified position, means for acquiring and analyzing user emotion data using an emotion recognition engine, and means for adjusting the movement and animation of the 3D model based on the user emotion data. This not only enables intuitive and efficient management of robots in a factory but also enables appropriate support according to the emotions of workers.

[0976] Definition of terms

[0977] "User" refers to a person who uses the system to upload image data and generate three-dimensional models and set animations.

[0978] A "terminal" is any device operated by a user, including smartphones, tablets, and personal computers.

[0979] "Image data" refers to digital images such as two-dimensional illustrations and paintings uploaded by users.

[0980] "Server" refers to a central processing unit that analyzes image data, generates three-dimensional models, adds animation, and sets the position of the augmented reality display.

[0981] A "two-dimensional image" refers to a digital image that has a flat composition, usually having width and height.

[0982] A "3D model" refers to a three-dimensional digital object that has width, height, and depth in three-dimensional space.

[0983] "Animation" refers to the process of adding movement to a three-dimensional model to create motion.

[0984] "Augmented reality" refers to the technology of overlaying digital content onto the real world.

[0985] "Location information" refers to data that indicates where in the real world a three-dimensional model is to be displayed.

[0986] An "emotion recognition engine" refers to software or hardware that acquires and analyzes a user's emotional data and adjusts the system's behavior based on that information.

[0987] An "action" refers to a movement or motion instruction that a user specifies for a three-dimensional model.

[0988] MODE FOR CARRYING OUT THE INVENTION

[0989] This invention relates to a system that allows users to convert two-dimensional images into three-dimensional models, add movement, and display them in augmented reality. It also recognizes the user's emotions and enables the generation of personalized content.

[0990] Program Generation

[0991] 1. Ability for users to upload illustrations

[0992] A user uses a device to upload image data (e.g., a two-dimensional illustration). The device sends the image data to the server, which then stores the received image data in a temporary storage area. This allows the server to check the integrity of the illustration before proceeding to the next step.

[0993] 2. 3D conversion function for illustrations

[0994] The server analyzes the received image data and extracts key features and contours. Based on the analysis results, it creates initial data for generating a 3D model from the 2D image. It then invokes OpenAI's generative AI model to convert the 2D illustration into a 3D model. The generated 3D model is saved in a database and the user is notified when the conversion is complete.

[0995] 3. 3D model animation function

[0996] The user uses a terminal to input specific actions (e.g., "run" or "jump") for the 3D model. The terminal sends this to the server, which analyzes the instructions. Based on the analysis results, Text2Motion technology is applied to generate animation data, adding movement to the 3D model. The generated animation data is stored in a database, and the user is notified when it is complete.

[0997] 4. AR display position setting function

[0998] The user uses the device's camera function to determine the position where the 3D model will be displayed in the real world. The device then sends the configured augmented reality display position information to the server, which then analyzes the received position information and generates data to display the 3D model at the specified position. The device then displays the model as augmented reality based on the received data.

[0999] 5. Implementing the Emotion Engine

[1000] The device is equipped with sensors (e.g., facial recognition using a camera, voice analysis using a microphone) to acquire the user's emotions and uses these to collect emotional data. The collected emotional data is sent to a server, which analyzes the data using an emotion engine. The server adjusts the movement and animation of the 3D model based on the user's emotions and sends the adjustment results to the device. The device then updates the display of the 3D model based on this.

[1001] Hardware and software used

[1002] Hardware: smartphone, tablet, computer, camera, microphone

[1003] Software: OpenAI generated AI models, Text2Motion technology, emotion recognition software

[1004] Specific examples

[1005] A factory manager uploads a diagram of the robot's layout within the factory and issues the command "move to maintenance position." As a result, a 3D model of the robot is generated and an animation of it moving to the designated maintenance position is added. The user's device analyzes the emotional data, and if, for example, fatigue is detected, the operation interface is simplified. This improves work efficiency and safety.

[1006] Example prompts for generative AI models

[1007] "Convert the provided 2D layout image of the factory robots into a 3D model. Add the animation for the command "move to maintenance position"."

[1008] In this way, the present invention is a system that allows users to easily convert 2D illustrations into 3D models, animate them, and display them as augmented reality. Furthermore, by combining it with an emotion engine, it is possible to provide personalized content that responds to the user's emotions.

[1009] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1010] Program processing steps

[1011] Step 1:

[1012] Users upload illustrations

[1013] operation

[1014] The user opens the app on their device, selects the image data (e.g., a two-dimensional illustration) they want to upload from the gallery, and presses the "Upload" button to send the image data to the server.

[1015] input

[1016] Image data selected on the device

[1017] output

[1018] Notification of completion of uploading image data to the server

[1019] Data processing and calculation

[1020] Image data is sent from the terminal to the server, and the server stores the data in a temporary storage area. It checks the consistency and, if there are no problems, proceeds to the next processing step.

[1021] Step 2:

[1022] The server converts the illustration into 3D.

[1023] operation

[1024] The server analyzes the uploaded image data, extracts key features and contours, and generates a 3D model from the 2D image based on the analysis results.

[1025] input

[1026] Uploaded image data

[1027] output

[1028] Generated 3D model data

[1029] Data processing and calculation

[1030] The server applies its own image analysis algorithms to detect features in the 2D illustration, then invokes OpenAI's generative AI model to generate a 3D model using a prompt, which is then stored in a database.

[1031] Step 3:

[1032] Add animation to a 3D model

[1033] operation

[1034] The user inputs actions (e.g., "run" or "jump") for the three-dimensional model through the app on the device and sends them to the server.

[1035] input

[1036] User-specified actions (e.g., "run," "jump")

[1037] output

[1038] Animated 3D models

[1039] Data processing and calculation

[1040] The server analyzes the user's instructions and generates animation data for the specified actions using Text2Motion technology. The generated animation data is added to the 3D model and stored in a database.

[1041] Step 4:

[1042] Setting the AR display position

[1043] operation

[1044] The user uses the device's camera to determine where in the real world the 3D model will be displayed, and then drags and positions the model on the device screen.

[1045] input

[1046] Real-world images captured through a camera and a display position specified by the user

[1047] output

[1048] 3D model displayed at the specified position

[1049] Data processing and calculation

[1050] The device sends camera images to the server in real time, acquires the set location information, and sends it back to the server. The server analyzes the location information and generates data to display a 3D model at that location. The device displays the model at the specified location based on the received data.

[1051] Step 5:

[1052] Utilizing the Emotion Engine

[1053] operation

[1054] The device collects emotion data using sensors (e.g., facial recognition by a camera, voice analysis by a microphone) to obtain the user's emotion. The collected data is sent to a server.

[1055] input

[1056] Emotion data from cameras and microphones

[1057] output

[1058] 3D model movements and animations adjusted based on emotion data

[1059] Data processing and calculation

[1060] The server analyzes the emotion data using an emotion engine and adjusts the movements and animations of the 3D model based on the user's emotions. The adjustment results are sent to the terminal, and the display is updated.

[1061] The above is a series of processing steps from uploading image data to displaying a 3D model using emotion recognition.

[1062] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1063] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1064] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1065] [Third embodiment]

[1066] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1067] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[1068] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1069] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1070] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1071] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1072] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1073] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1074] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1075] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1076] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1077] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1078] This invention relates to a system that allows anyone to easily convert two-dimensional images into three-dimensional models, add animation, and display them in augmented reality for sharing. This system has the function of allowing users to upload image data from their devices, and a server analyzes the data to generate a three-dimensional model and add animation. It also has the function of allowing users to set the augmented reality display position of the three-dimensional model using their devices, and the server acquires the location information and displays it at the specified location.

[1079] A natural language description of the program's operation

[1080] 1. Ability for users to upload illustrations

[1081] The user opens the app on their device, selects the illustration they want to upload from the gallery, and presses the "Upload" button to send the selected illustration to the server.

[1082] The server receives the uploaded illustration data and stores it in a temporary storage area. It checks the integrity of the file and, if there are no problems, proceeds to the next step.

[1083] 2. 3D conversion function for illustrations

[1084] The server analyzes the received illustration data, extracts key features and contours, and creates initial data for generating a 3D model from the 2D image based on the analysis results.

[1085] The server calls the generative AI model to convert the 2D illustration into a 3D model, which is then saved in a database and a notification is sent to the user when the conversion is complete.

[1086] 3. 3D model animation function

[1087] The user uses the chat box on the terminal to input actions (e.g., "walk," "jump," etc.) for the three-dimensional model and sends them to the server.

[1088] The server analyzes the user's instructions, generates the corresponding animation using Text2Motion technology, and adds it to the 3D model. The generated animation data is also saved in the database. Once completed, the server notifies the user of the results.

[1089] 4. AR display position setting function

[1090] The user uses the device's camera function to determine where in the real world the 3D model should be displayed, specifically by dragging and positioning the 3D model on the device screen.

[1091] The terminal transmits the set position information to the server, and the server analyzes the information to determine the display position of the three-dimensional model in the real world.

[1092] The server sends data for displaying the 3D model at the specified location to the terminal, and the terminal displays the 3D model at the specified location as augmented reality based on the received data.

[1093] Specific examples

[1094] When a child uploads an illustration of a character they've drawn and commands "run" via chat, the character is converted into a three-dimensional model and a running animation is added. The user can then use their device's camera to position the character in their living room. This positioning information is sent to the server, and the character is displayed in augmented reality, running at the specified location in the living room. The user can view this display or share it with others.

[1095] As described above, the present invention relates to a system that allows users to easily convert two-dimensional illustrations into three-dimensional models, animate them, and display them as augmented reality. This system can provide creative activities and new communication methods.

[1096] The processing flow will be explained below.

[1097] Specific processing flow of the program

[1098] 1. Ability for users to upload illustrations

[1099] Step 1:

[1100] The user opens the app on their device and selects the illustration they want to upload from the gallery.

[1101] When the user presses the "Upload" button, the terminal sends the selected illustration to the server.

[1102] Step 2:

[1103] The server receives the transmitted illustration data and stores it in a temporary storage area.

[1104] The server performs a file integrity check to ensure the file was uploaded successfully.

[1105] Step 3:

[1106] The server analyzes the quality of the uploaded illustrations and, if necessary, notifies the user of suggestions for improving the quality.

[1107] 2. 3D conversion function for illustrations

[1108] Step 1:

[1109] The server analyzes the received illustration data and extracts key features and contours, allowing each part of the illustration to be recognized.

[1110] Step 2:

[1111] Based on the analysis results, the server creates initial data for generating a three-dimensional model from the two-dimensional image.

[1112] Step 3:

[1113] The server invokes a generative AI model (e.g., generative AI model technology) to convert the 2D illustration into a 3D model.

[1114] The server stores the generated 3D model in a database.

[1115] Step 4:

[1116] The server notifies the user when the 3D rendering is complete and provides a link or preview for confirmation.

[1117] 3. 3D model animation function

[1118] Step 1:

[1119] The user opens a chat box through the device app and inputs actions (e.g., "run," "jump," etc.) for the three-dimensional model.

[1120] When the user sends an instruction, the terminal sends the instruction to the server.

[1121] Step 2:

[1122] The server receives instructions from the user and analyzes the content of the instructions. For example, if the instruction to "run" is sent, the server analyzes the action.

[1123] Step 3:

[1124] The server uses Text2Motion technology to generate animation data for adding movement to the three-dimensional model based on the analyzed instructions.

[1125] Step 4:

[1126] The server saves the generated animation data in the 3D model database and notifies the user when it is complete.

[1127] 4. AR display position setting function

[1128] Step 1:

[1129] The user uses the device's camera to determine where in the real world the 3D model should be displayed, and then drags and positions the 3D model on the device screen.

[1130] Step 2:

[1131] The terminal transmits the set position information of the augmented reality display to the server.

[1132] Step 3:

[1133] The server analyzes the received location information and generates data for displaying the three-dimensional model at the specified location.

[1134] Step 4:

[1135] The server transmits this data to the terminal, and the terminal displays the three-dimensional model as augmented reality at the specified position based on the received data.

[1136] For example, a child can upload an illustration of a character they have drawn and use chat to say "run." As a result, the character is generated as a 3D model, a running animation is added, and it is displayed as augmented reality on the living room floor of the user's home. The user can then use their device's camera to watch the character running around the living room.

[1137] Example 1

[1138] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1139] With conventional technologies, converting 2D images into 3D models, adding movement, and displaying them in augmented reality is extremely difficult and requires specialized knowledge and time. Furthermore, there is a lack of easy ways for users to set their desired actions and location information, creating a demand for intuitive and easy operation. This creates a need for an efficient system for creating and sharing creative content.

[1140] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1141] In this invention, the server includes means for a user to upload image data from a terminal, means for analyzing the image data received by the server and generating a 3D model from a 2D image, means for converting the 2D image into a 3D model using a generative AI model, means for adding animation to the 3D model based on an action specified by the user, means for the user to use the terminal to set a position where the 3D model is to be displayed as augmented reality in the real world, and means for the server to acquire the set position information and display the 3D model at the specified position. This allows users to easily convert 2D illustrations into 3D models, add animation, and display and share them as augmented reality.

[1142] A "user" is any individual or organization who wishes to use the system to convert two-dimensional images into three-dimensional models and display them as augmented reality.

[1143] "Terminal" refers to a device operated by a user to upload image data, place 3D models, and specify actions. Specifically, this includes smartphones, tablets, and PCs.

[1144] A "server" is a computer or network system that receives and analyzes data sent from a terminal, generates a three-dimensional model from a two-dimensional image, adds actions, and performs calculations to display it as augmented reality.

[1145] "Image data" is digital data containing visual information such as two-dimensional illustrations and photographs that are uploaded by users from their terminals.

[1146] "Analysis" is the process by which the server understands the structure and characteristics of the image data it receives and extracts the necessary information.

[1147] A "two-dimensional image" is flat image data that has two dimensions, vertical and horizontal.

[1148] A "three-dimensional model" is digital data that has three dimensions - length, width, and depth - and can be displayed three-dimensionally in a virtual space.

[1149] "Generative AI model" refers to an algorithm or technique that uses artificial intelligence to convert two-dimensional images into three-dimensional models.

[1150] "Animation" is digital data that adds movement to a three-dimensional model and makes it operate continuously.

[1151] "Text2Motion technology" is a technology that analyzes text instructions entered by the user and adds corresponding movements to a three-dimensional model.

[1152] "Augmented reality (AR)" is a technology that displays computer-generated visual information overlaid on a real-world environment.

[1153] "Location information" is digital data relating to a location designated by a user to display a three-dimensional model in the real world.

[1154] A "database" is a storage device or system for storing and managing three-dimensional models and animation data.

[1155] The present invention relates to a system that allows users to easily convert two-dimensional images into three-dimensional models, add motion, and display and share them as augmented reality. The system has several main functions, each of which is realized by specific hardware and software.

[1156] The system's hardware includes terminal devices such as smartphones, tablets, and PCs, as well as a server, while the software uses the generative AI model "DeepArt3D" and Text2Motion technology.

[1157] Ability for users to upload illustrations

[1158] Users launch the application using a terminal device such as a smartphone or PC and select the 2D illustration they want to upload from local storage or a gallery. The selected illustration is sent from the device to the server. The server receives the uploaded illustration data and stores it in a temporary storage area. Once the integrity of the file is confirmed, the process proceeds to the next step.

[1159] 3D conversion function for illustrations

[1160] The server analyzes the received illustration data and uses the DeepArt3D model to generate a 3D model from the 2D image. First, it extracts the main features and contours of the illustration and creates initial data for the 3D model based on them. Next, the DeepArt3D model uses this initial data to generate the 3D model. The generated 3D model is saved in a database, and the user is notified when the conversion is complete.

[1161] 3D model animation function

[1162] The user opens the chat box on their device and inputs actions for the 3D model (such as "walk" or "jump") in text format. The input instructions are sent to the server, which uses Text2Motion technology to generate the corresponding animation and add it to the 3D model. The generated animation data is saved in a database, and the user is notified when the animation has been added.

[1163] For example, if a user inputs "Make this character run," the server will analyze this prompt and generate and add a "running" animation using Text2Motion technology. This method allows users to easily implement their desired movements in a 3D model.

[1164] AR display position setting function

[1165] The user activates the device's camera and decides where in the real world to display the 3D model. Specifically, they drag and drop the 3D model onto the device screen to place it. The device then sends this location information to the server, which analyzes it and determines where to display the 3D model. The server then sends the necessary data to the device, and the device displays the 3D model at the specified location as augmented reality.

[1166] For example, if a child uploads an illustration of a character and commands "run" via chat, the character is converted into a three-dimensional model and a running animation is added. Furthermore, the user can use the device's camera function to position the character in their living room. This positioning information is sent to the server, and the character is displayed in augmented reality as if running at the specified location in the living room.

[1167] As described above, the present invention relates to a system that allows users to intuitively operate it, quickly and easily convert two-dimensional illustrations into three-dimensional models, and then use them as augmented reality. This system provides new ways of creative activities and communication, and greatly enhances user convenience.

[1168] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1169] Step 1:

[1170] The user launches the application using their device. The user selects the 2D illustration they want to upload from local storage or the gallery. The selected illustration data is sent from the device to the server by pressing the "Upload" button.

[1171] Input: 2D illustration data selected from local storage or gallery

[1172] Output: Illustration data sent to the server

[1173] Step 2:

[1174] The server receives the uploaded illustration data and stores it in a temporary storage area. The server then checks the integrity of the file and, if there are no problems, begins analysis.

[1175] Input: Received illustration data

[1176] Output: Illustration data with consistency confirmed

[1177] Step 3:

[1178] The server analyzes the received illustration data, extracting the main features and contours of the image and creating initial data for generating a 3D model from the 2D image.

[1179] Input: Illustration data that has passed the consistency check

[1180] Output: Initial data including key feature and contour extraction results

[1181] Step 4:

[1182] The server calls the generative AI model "DeepArt3D" and converts the 2D illustration into a 3D model using the initial data. At this time, the server constructs the model and adds texture.

[1183] Input: Initial data

[1184] Output: Generated 3D model

[1185] Step 5:

[1186] The server stores the generated 3D model in a database and also notifies the user that the conversion of the 3D model is complete.

[1187] Input: Generated 3D model

[1188] Output: 3D model stored in database, notification to user

[1189] Step 6:

[1190] The user opens a chat box on the terminal and inputs text for actions to be performed on the 3D model (for example, "walk," "jump," etc.). The input instructions are sent from the terminal to the server.

[1191] Input: User instruction text

[1192] Output: Instruction text sent to the server

[1193] Step 7:

[1194] The server analyzes the user's text instructions and generates the corresponding animation using Text2Motion technology, a process that adds movement to the 3D model based on the user's instructions.

[1195] Input: Instruction text

[1196] Output: An animated 3D model

[1197] Step 8:

[1198] The server stores the animated 3D model in a database and also notifies the user that the animation has been completed.

[1199] Input: An animated 3D model

[1200] Output: Animated 3D model stored in database, notification to user

[1201] Step 9:

[1202] The user activates the device's camera function and sets the location in the real world where they want to display the 3D model on the device screen. Specifically, the user drags and drops the 3D model onto the device screen to place it. The set location information is sent to the server.

[1203] Input: Set location information

[1204] Output: Location information sent to the server

[1205] Step 10:

[1206] The server analyzes the received location information and determines the display position of the 3D model in the real world. It then transmits the necessary data to the device and displays the 3D model in the specified position as augmented reality.

[1207] Input: Location

[1208] Output: Display data of a 3D model of the real world generated from engine data

[1209] Step 11:

[1210] The device uses the received display data to display the 3D model as augmented reality at a specified location in the real world, allowing the user to view and share the display with others.

[1211] Input: Display data

[1212] Output: Augmented reality display of 3D model in the real world

[1213] (Application example 1)

[1214] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1215] Conventional systems that convert 2D images into 3D models and add animation have had the problem of being complex to operate, making them difficult for users to operate intuitively. Furthermore, when displaying the generated 3D model in the real world, it is difficult for users to set the exact display position, which is time-consuming. Furthermore, in advertising and promotions, there is a lack of user-participation interactive elements, and there is a lack of a way to easily create visually appealing content.

[1216] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1217] In this invention, the server includes: means for a user to upload image data from a terminal; means for the server to analyze the received image data and generate a three-dimensional model from a two-dimensional image; means for the server to animate the generated three-dimensional model; means for the user to set a position in the real world where the three-dimensional model will be displayed as augmented reality using the terminal; means for the server to acquire the set position information and display the three-dimensional model at the specified position; and means for analyzing the user's action instructions regarding the three-dimensional model and generating corresponding animations. This allows a user to generate a three-dimensional model from a two-dimensional image with simple operations, add animation to the model, and display it in augmented reality. Furthermore, incorporating interactive elements makes it easy to create visually appealing advertisements and promotional content.

[1218] "User" refers to the entity that uses the system to upload image data, generate three-dimensional models, add animations, and set display positions as augmented reality.

[1219] A "terminal" is a device used by a user, and refers to an electronic device such as a smartphone, smart glasses, or a head-mounted display.

[1220] "Image data" refers to two-dimensional visual information uploaded by a user through a terminal.

[1221] A "server" refers to a computing device that receives image data from a terminal, analyzes it, generates a three-dimensional model from the two-dimensional image, adds animation, and notifies the user of the results.

[1222] A "two-dimensional image" refers to image information that is composed of two dimensions, vertical and horizontal.

[1223] A "three-dimensional model" refers to a three-dimensional model expressed in three dimensions: length, width, and height.

[1224] "Animation" refers to a technique of expression that gives the visual impression of movement by adding movement to a three-dimensional model.

[1225] "Augmented reality" refers to the technology of overlaying digital information (in this case, three-dimensional models) onto a real-world environment.

[1226] "Location information" refers to data that allows a user to use a device to set where in the real world a three-dimensional model is to be displayed.

[1227] An "action instruction" refers to input information instructing a three-dimensional model to perform a specific action or movement.

[1228] The present invention provides a system that allows a user to convert a two-dimensional image into a three-dimensional model, add movement to the model, and display it in augmented reality. The following description will explain in detail how to specifically implement the present invention.

[1229] 1. System Configuration

[1230] The system for implementing the present invention mainly uses the following hardware and software.

[1231] Hardware:

[1232] User devices: smartphones (Android or iOS), smart glasses, head-mounted displays

[1233] Server: A cloud-based computing device (such as AWS or Google Cloud)

[1234] software:

[1235] Smartphone applications: Kotlin for Android and Swift for iOS

[1236] Server software: Python, TensorFlow for generative AI models, OpenCV for image analysis

[1237] Database: PostgreSQL

[1238] 2. Program Processing Overview

[1239] The program on the server performs the following process.

[1240] 1. Ability for users to upload illustrations:

[1241] The user launches the app on their smartphone, selects the 2D image they want to upload from their gallery, and presses the "Upload" button.

[1242] The server receives the uploaded image data and stores it in a temporary storage area. It checks the integrity of the data and, if there are no problems, proceeds to the next step.

[1243] 2. 3D conversion function for illustrations:

[1244] The server analyzes the received image data, extracts key features and contours, and creates initial data for generating a 3D model from the 2D image based on the analysis results.

[1245] The server invokes the generative AI model to convert the 2D image into a 3D model, which is then stored in a database and a notification is sent to the user when the conversion is complete.

[1246] 3. 3D model animation features:

[1247] The user uses the chat box on their smartphone to instruct the 3D model to take actions (e.g., "spin," "jump," "glow," etc.) and sends the instructions to the server.

[1248] The server analyzes the user's instructions, generates the corresponding animation using Text2Motion technology, and adds it to the 3D model. The generated animation data is also saved in a database, and the user is notified of the results once the process is complete.

[1249] 4. AR display position setting function:

[1250] The user uses the smartphone's camera to decide where in the real world the 3D model should be displayed, by dragging and positioning the 3D model on the device screen.

[1251] The terminal transmits the set position information to the server, and the server analyzes the information to determine the display position of the three-dimensional model in the real world.

[1252] The server transmits data for displaying a three-dimensional model at a specified position to the terminal, and the terminal displays the three-dimensional model at the specified position as augmented reality based on the received data.

[1253] Specific examples

[1254] Suppose a company wants to run an advertising campaign. A company representative opens a smartphone app and uploads a promotional company logo. Then, they type "jump" into the chat box. The server analyzes the logo, converts it into a 3D model, and adds a jumping animation to the model. The representative then places a promotional character on a cafe table, which is displayed in the real world as augmented reality. Examples of prompts used in this process include "upload an image file selected by the user to the server," "extract key features from the 2D logo and convert it into a 3D model," and "add a rotation animation to the 3D model."

[1255] Through the above process, users can intuitively and simply convert two-dimensional images into three-dimensional models, add movement to the models, and display them in the real world as augmented reality, making it possible to generate visually appealing advertisements and promotional content.

[1256] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1257] Program processing steps

[1258] Step 1:

[1259] The user launches the smartphone app, selects a two-dimensional image (e.g., a company logo) from the gallery, and presses the "Upload" button.

[1260] Input: 2D image data from the user's terminal

[1261] Operation: The terminal sends the selected image data to the server.

[1262] Output: 2D image data uploaded to the server

[1263] Step 2:

[1264] The server analyzes the received image data and stores it in a temporary storage area. It checks the data for consistency and, if there are no problems, it proceeds to the next step.

[1265] Input: Received image data

[1266] Action: Check the data integrity and save it to a temporary storage area.

[1267] Output: Image data with integrity confirmed

[1268] Step 3:

[1269] The server extracts the main features and contours of the received image data, and based on the analysis results, creates initial data for generating a 3D model from the 2D image.

[1270] Input: Image data whose integrity has been verified

[1271] How it works: Feature extraction is performed using OpenCV.

[1272] Output: Key features and contours

[1273] Step 4:

[1274] The server invokes a generative AI model (TensorFlow) to convert the 2D image into a 3D model, which is then saved in a database and a notification is sent to the user when the conversion is complete.

[1275] Input: feature and contour data

[1276] How it works: Generates a 3D model using a TensorFlow model and stores it in a database

[1277] Output: 3D model data, conversion completion notification

[1278] Step 5:

[1279] The user uses the chat box on their smartphone to instruct the 3D model to take an action (e.g., "spin," "jump," "glow") and send it to the server.

[1280] Input: User instruction for action

[1281] Action: Enter instructions in the chat box and send them to the server

[1282] Output: Action instructions sent to the server

[1283] Step 6:

[1284] The server analyzes the user's instructions, generates the corresponding animation using Text2Motion technology, and adds it to the 3D model. This animation data is also stored in the database, and the result is notified to the user after completion.

[1285] Input: User instruction for action

[1286] Action: Animation is generated using Text2Motion technology and saved in the database

[1287] Output: 3D model data with animation, notification of completion of generation

[1288] Step 7:

[1289] Users use the smartphone's camera to decide where in the real world they want to display the 3D model, then drag and place the model on the device screen.

[1290] Input: User's display position setting information

[1291] Operation: Set the display position using the camera function and send the position information to the server

[1292] Output: Display position information sent to the server

[1293] Step 8:

[1294] The server analyzes the location information set by the user and determines the display position of the 3D model in the real world. The server then sends data to the device to display the 3D model at the specified position.

[1295] Input: Display location information

[1296] Operation: Analyzes location information, generates data for augmented reality display, and sends it to the device

[1297] Output: Augmented reality display data

[1298] Step 9:

[1299] Based on the data received from the server, the device displays a three-dimensional model as augmented reality at a specified location.

[1300] Input: Augmented reality display data

[1301] Operation: A 3D model is superimposed on the camera image based on the received data.

[1302] Output: 3D model displayed as augmented reality in the real world

[1303] The above are the processing steps for carrying out the invention.

[1304] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1305] The present invention relates to a system that allows anyone to easily convert two-dimensional images into three-dimensional models, add movement, and display them in augmented reality for sharing. The present invention enables the generation of more personalized content by combining an emotion engine that recognizes user emotions. This system has a function that allows users to upload image data from their devices, and a server analyzes the data to generate a three-dimensional model and add animation. The system also has a function that allows users to use their devices to set the augmented reality display position of the three-dimensional model, and the server acquires that position information and displays it at the specified location. Additionally, the emotion engine can recognize user emotions and change behavior or adjust animations based on those emotions.

[1306] A natural language description of the program's operation

[1307] 1. Ability for users to upload illustrations

[1308] The user opens the app on their device, selects the illustration they want to upload from the gallery, and presses the "Upload" button to send the selected illustration to the server.

[1309] The server receives the illustration data and stores it in a temporary storage area. It checks the integrity of the file and, if there are no problems, proceeds to the next step.

[1310] 2. 3D conversion function for illustrations

[1311] The server analyzes the received illustration data, extracts key features and contours, and creates initial data for generating a 3D model from the 2D image based on the analysis results.

[1312] The server calls the generative AI model and converts the 2D illustration into a 3D model, which is then stored in a database.

[1313] The server notifies the user when the 3D rendering is complete and provides a link or preview for confirmation.

[1314] 3. 3D model animation function

[1315] The user opens a chat box through the app on their device, inputs actions for the three-dimensional model (e.g., "run," "jump," etc.), and sends them to the server.

[1316] The server receives instructions from the user and analyzes the content of the instructions. For example, if the instruction to "run" is sent, the server analyzes the action.

[1317] The server uses Text2Motion technology to generate animation data for adding movement to the three-dimensional model based on the analyzed instructions.

[1318] The server saves the generated animation data in the 3D model database and notifies the user when it is complete.

[1319] 4. AR display position setting function

[1320] The user uses the device's camera to determine where in the real world the 3D model should be displayed, and then drags and positions the 3D model on the device screen.

[1321] The terminal transmits the set position information of the augmented reality display to the server.

[1322] The server analyzes the received location information and generates data for displaying the three-dimensional model at the specified location.

[1323] The server transmits this data to the terminal, and the terminal displays the three-dimensional model as augmented reality at the specified position based on the received data.

[1324] 5. Implementing the Emotion Engine

[1325] The device is equipped with sensors (e.g., facial recognition using a camera, voice analysis using a microphone) to acquire the user's emotions, and uses these to collect the user's emotional data.

[1326] The device transmits emotion data to the server, which then analyzes the data using an emotion engine.

[1327] The server adjusts the behavior and animation of the 3D model based on the user's emotions, for example, making the model more active if the user is happy, or slowing down if the user is sad.

[1328] The server transmits the adjustment results to the terminal, and the terminal updates the display of the three-dimensional model based on the results.

[1329] Specific examples

[1330] A child uploads an illustration of a character they have drawn and commands "run" via chat. As a result, the character is generated as a 3D model and a running animation is added. The user then uses the camera function of their device to place the character on the floor of their living room. This placement information is sent to the server, and the character is displayed in augmented reality as if running at the specified location in the living room.

[1331] Furthermore, the user's device uses a camera to analyze their facial expressions, and if the user is having fun, the character will adjust to run more energetically. When the device detects that the user is happy, the server's emotion engine activates the character's movements, realizing behavior that matches the user's emotions.

[1332] As described above, the present invention relates to a system that allows users to easily convert 2D illustrations into 3D models, animate them, and display them as augmented reality. Furthermore, by combining this with an emotion engine, it is possible to provide personalized content that responds to the user's emotions.

[1333] The processing flow will be explained below.

[1334] Specific processing flow of the program (when combined with an emotion engine)

[1335] 1. Ability for users to upload illustrations

[1336] Step 1:

[1337] The user opens the app on their device and selects the illustration they want to upload from the gallery.

[1338] When the user presses the "Upload" button, the terminal sends the selected illustration to the server.

[1339] Step 2:

[1340] The server receives the transmitted illustration data and stores it in a temporary storage area.

[1341] The server performs a file integrity check to ensure the file was uploaded successfully.

[1342] Step 3:

[1343] The server analyzes the quality of the uploaded illustration and, if any parts are distorted or of low resolution, notifies the user with suggestions for correcting them.

[1344] 2. 3D conversion function for illustrations

[1345] Step 1:

[1346] The server analyzes the received illustration data and extracts key features and contours, which correspond to each part of the 2D illustration.

[1347] Step 2:

[1348] Based on the analysis results, the server extracts the initial data necessary to generate a basic three-dimensional model.

[1349] Step 3:

[1350] The server invokes a generative AI model to convert the 2D illustration into a 3D model, which is made up of data including shape and texture.

[1351] The server stores the generated 3D model in a database and notifies the user that the 3D model generation is complete.

[1352] 3. 3D model animation function

[1353] Step 1:

[1354] The user opens a chat box through the device app and inputs actions (e.g., "run," "jump," etc.) for the three-dimensional model.

[1355] When the user sends an instruction, the terminal sends the instruction to the server.

[1356] Step 2:

[1357] The server receives instructions from the user and analyzes the content of the instructions. For example, if the instruction to "run" is sent, the server analyzes the action in detail.

[1358] Step 3:

[1359] The server uses Text2Motion technology to generate animation data for adding movement to the three-dimensional model based on the analyzed instructions.

[1360] Step 4:

[1361] The server saves the generated animation data in the 3D model database and notifies the user when it is complete.

[1362] 4. AR display position setting function

[1363] Step 1:

[1364] The user uses the device's camera to determine where in the real world the 3D model should be displayed, and then drags and positions the 3D model on the device screen.

[1365] Step 2:

[1366] The terminal transmits the set position information of the augmented reality display to the server.

[1367] Step 3:

[1368] The server analyzes the received location information and generates data for displaying the three-dimensional model at the specified location.

[1369] Step 4:

[1370] The server transmits this data to the terminal, and the terminal displays the three-dimensional model as augmented reality at the specified position based on the received data.

[1371] 5. Implementing the Emotion Engine

[1372] Step 1:

[1373] The device is equipped with sensors (e.g., facial recognition using a camera, voice analysis using a microphone) to acquire the user's emotions, and uses these to collect the user's emotional data.

[1374] Step 2:

[1375] The device sends emotion data to the server, which then uses an emotion engine to analyze the data, for example, recognizing whether the user is surprised or happy through facial expression analysis.

[1376] Step 3:

[1377] The server adjusts the behavior and animation of the 3D model based on the user's emotions: for example, if the user is having fun, the model's movements will be more active, and conversely, if the user is sad, the model's movements will be more gentle.

[1378] Step 4:

[1379] The server transmits the adjustment results to the terminal, and the terminal updates the display of the three-dimensional model based on the results.

[1380] Specific examples

[1381] As a concrete example, a child uploads an illustration of a character they have drawn and commands "run" via chat. As a result, the character is generated as a 3D model and a running animation is added. The user then uses the camera function of their device to place the character on the floor of their living room. This placement information is sent to the server, and the character is displayed in augmented reality as if running at the specified location in the living room.

[1382] Furthermore, the user's device uses a camera to analyze their facial expressions, and if the user is having fun, the character will adjust to run more energetically. When the device detects that the user is happy, the server's emotion engine activates the character's movements, realizing behavior that matches the user's emotions.

[1383] Example 2

[1384] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1385] Currently, converting two-dimensional illustrations into three-dimensional models, adding movement to those models, and displaying them as augmented reality requires extensive specialized knowledge and complex software. These operations are difficult for ordinary users to perform due to the high technical barriers. Furthermore, the content created is rarely properly personalized based on the user's emotions. This limits the user experience and makes it difficult to create more engaging content.

[1386] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for a user to upload image data from a terminal, a means for the server to analyze the image data received and generate a three-dimensional model from a two-dimensional image, and a means for animating the three-dimensional model generated by the server. This allows a user, without specialized knowledge, to intuitively convert a two-dimensional illustration into a three-dimensional model, add animation, and display it as augmented reality. Furthermore, the terminal includes a means for acquiring the user's emotions and adjusting the behavior of the three-dimensional model based on the emotions, thereby enabling the generation of more personalized and attractive content.

[1387] A "user" is an entity that uses the system to upload image data and generate three-dimensional models and display augmented reality.

[1388] A "terminal" is an electronic device that a user uses to upload image data, such as a smartphone or tablet.

[1389] "Image data" refers to two-dimensional illustrations and image files uploaded by users.

[1390] A "server" is a computer system that analyzes received image data, generates three-dimensional models, animates them, and manages the augmented reality display.

[1391] "Analysis" is the process of extracting features and contours from received image data, which serve as the basis for generating a three-dimensional model from the two-dimensional image.

[1392] A "two-dimensional image" refers to a flat image that has only width and height information.

[1393] A "3D model" is a three-dimensional digital object that has width, height, and depth information.

[1394] "Animation" refers to the process of adding movement to a three-dimensional model.

[1395] "Augmented reality" refers to a technology that overlays digital information onto images of the real world.

[1396] "Location information" is data that indicates where in the real world a three-dimensional model is to be displayed.

[1397] "Emotion" refers to the psychological and sensory state that can be read from the user's facial expressions, voice, etc.

[1398] "Adjusting the behavior" refers to changing the animation of the three-dimensional model according to the user's emotions.

[1399] The present invention relates to a system that allows a user to easily convert a two-dimensional illustration into a three-dimensional model, add movement to the model, and display it as an augmented reality. An embodiment of the system will be described in detail below.

[1400] 1. Ability for users to upload illustrations

[1401] Users launch the dedicated application using a device such as a smartphone or tablet. They tap the "Select from Gallery" button within the application to open the device's image gallery and select the illustration they wish to upload. The user then presses the "Upload" button, which sends the illustration data to the server. The server receives the illustration data and stores it in a temporary storage area. At this time, the integrity of the file is also checked.

[1402] 2. 3D conversion function for illustrations

[1403] The server analyzes the received illustration data and extracts key features and contours. This process uses image analysis algorithms such as OpenCV. Based on the analysis results, the server invokes a generative AI model (e.g., DALL-E or Stable Diffusion) to convert the 2D illustration into a 3D model. The generated 3D model is stored in a database, and the user is notified when generation is complete. The notification includes a confirmation link and a preview image.

[1404] 3. 3D model animation function

[1405] The user inputs action instructions (e.g., "run," "jump," etc.) for the 3D model in the app's chat box and sends them to the server. The server analyzes the received instructions and generates animation data using Text2Motion technology. This animation data is also stored in the database, and the user is notified when generation is complete.

[1406] 4. AR display position setting function

[1407] The user activates the device's camera function and drags the 3D model on the screen to determine where in the real world the model should be displayed. The location information is sent to the server, which analyzes the received location information and generates augmented reality data for displaying the 3D model at the specified location. The generated data is sent to the device, which then displays the 3D model in the real world based on the received data.

[1408] 5. Implementing the Emotion Engine

[1409] The device uses sensors such as a camera and microphone to collect the user's emotional data in real time. For example, the camera recognizes the user's face and uses software (e.g., Emotion API) to analyze their facial expressions. The collected emotional data is sent to a server, which then analyzes it using an emotion engine. Based on the analysis results, the server adjusts the movement and animation of the 3D model and sends the results to the device. This causes the model's movement to change according to the user's emotions.

[1410] Specific examples

[1411] For example, a child uploads an illustration of a character they drew from their device and commands "run" in the chat box. In this case, the server generates a 3D model and adds a running animation. The user then uses the device's camera to place the character on the living room floor. Once the placement information is sent to the server, the character is displayed in augmented reality as if running in the specified position. Furthermore, if the device detects the user's facial expression of enjoyment, the server's emotion engine actively adjusts the character's movements.

[1412] Prompt Sentence Examples

[1413] "Turn your drawn character into a 3D model and run it around your living room."

[1414] By implementing these functions, the system of the present invention enables users to intuitively convert 2D illustrations into 3D models, animate them, and display them as augmented reality without requiring specialized knowledge. It also provides personalized content based on the user's emotions using an emotion engine.

[1415] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1416] Step 1:

[1417] Input: The user launches the app on their device and selects the 2D illustration to upload.

[1418] How it works: A user taps the "Select from Gallery" button in the app, opens the image gallery, selects the illustration they want to upload, and then presses the "Upload" button.

[1419] Data processing: The terminal prepares the selected illustration data and encodes the image data for transmission to the server.

[1420] Output: The encoded image data is sent to the server.

[1421] Step 2:

[1422] Input: The server receives the illustration data sent by the user.

[1423] Operation: The server receives the illustration data, stores it in a temporary storage area, and verifies the integrity of the file.

[1424] Data processing: Analyze the received data and apply integrity check algorithms.

[1425] Output: The illustration data whose consistency has been confirmed proceeds to the next processing step.

[1426] Step 3:

[1427] Input: Illustration data whose integrity has been confirmed.

[1428] How it works: The server analyzes the illustration data and extracts key features and contours using image analysis algorithms such as OpenCV.

[1429] Data processing: A feature extraction algorithm is run to obtain the features of the 2D illustration as data.

[1430] Output: The extracted feature data is used as input to a generative AI model.

[1431] Step 4:

[1432] Input: Extracted feature data.

[1433] How it works: The server invokes a generative AI model (e.g., DALL-E or Stable Diffusion) to generate a 3D model from a 2D illustration.

[1434] Data processing: Feature data is input into the generative AI model, and data calculations are performed to convert the two-dimensional illustration into a three-dimensional model.

[1435] Output: The generated 3D model data is saved in a database.

[1436] Step 5:

[1437] Input: Generated 3D model data.

[1438] What it does: The server notifies the user that generation is complete, including a confirmation link and a preview image.

[1439] Data processing: Generate notification data and send it to the user terminal.

[1440] Output: The user receives a link and a preview image to view the 3D model.

[1441] Step 6:

[1442] Input: The user inputs an action instruction (e.g., "run") in text format.

[1443] How it works: The user types instructions into the app's chat box and sends them to the server.

[1444] Data processing: The terminal collects text instructions from the user and converts them into a data format to be sent to the server.

[1445] Output: A text instruction is sent to the server.

[1446] Step 7:

[1447] Input: Text instruction data.

[1448] Operation: The server parses the received instruction data and uses Text2Motion technology to parse it.

[1449] Data processing: Use a natural language processing engine (e.g., GPT-3) to convert text instructions into concrete animation data.

[1450] Output: The generated animation data is saved in a database.

[1451] Step 8:

[1452] Input: Animation data.

[1453] Action: The server notifies the user that animation generation is complete.

[1454] Data processing: Generate notification data and send it to the user terminal.

[1455] Output: The user receives a notification that the animation data has been generated.

[1456] Step 9:

[1457] Input: Camera launch for user to set viewing position.

[1458] Action: The user activates the device's camera and drags the 3D model to place it in the real world.

[1459] Data processing: The device collects location data based on camera footage and converts it into a data format to be sent to the server.

[1460] Output: The location data is sent to the server.

[1461] Step 10:

[1462] Input: Location data.

[1463] Operation: The server analyzes the received location data and generates augmented reality data for displaying a three-dimensional model at the specified location.

[1464] Data processing: Generate data for augmented reality display based on location data.

[1465] Output: The generated augmented reality data is sent to the device.

[1466] Step 11:

[1467] Input: Augmented reality data.

[1468] How it works: The device uses the data it receives to display a 3D model in the real world.

[1469] Data processing: Using the device's AR engine, model data is overlaid onto the camera image.

[1470] Output: The user sees the 3D model displayed in the real world.

[1471] Step 12:

[1472] Input: User emotion data.

[1473] How it works: The device uses sensors (camera and microphone) to collect user emotional data in real time.

[1474] Data processing: The collected emotion data is initially analyzed on the device and converted into a data format that is sent to the server.

[1475] Output: Emotion data is sent to the server.

[1476] Step 13:

[1477] Input: Emotion data.

[1478] Operation: The server analyzes the emotion data to determine the user's emotional state.

[1479] Data Processing: An emotion engine is used to analyze the emotion data and identify the user's emotional state.

[1480] Output: The emotion analysis results are used as motion adjustment data for the 3D model.

[1481] Step 14:

[1482] Input: Sentiment analysis results.

[1483] Movement: The server adjusts the movement and animation of the 3D model based on the analysis results.

[1484] Data processing: Generate adjustment parameters and update the animation data of the 3D model.

[1485] Output: The adjusted animation data is sent to the device.

[1486] Step 15:

[1487] Input: The adjusted animation data.

[1488] Operation: The device updates the display of the 3D model based on the received data.

[1489] Data processing: Update the animation data in the device's AR engine to create new model behavior.

[1490] Output: The user can see a 3D model with movements adjusted according to the emotion.

[1491] (Application example 2)

[1492] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1493] Currently, operating and managing robots in factories requires complex procedures and specialized knowledge, making it difficult to provide effective operating instructions and maintenance information. Furthermore, there is a lack of support systems that take into account the emotions of workers, which can lead to a decline in operational efficiency and safety. In response to this situation, a more intuitive and effective robot management system is needed.

[1494] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for a user to upload image data from a terminal, means for the server to analyze the received image data and generate a 3D model from a 2D image, means for the server to animate the generated 3D model, means for the user to use the terminal to set a position at which the 3D model will be displayed in the real world as augmented reality, means for the server to acquire the set position information and display the 3D model at the specified position, means for acquiring and analyzing user emotion data using an emotion recognition engine, and means for adjusting the movement and animation of the 3D model based on the user emotion data. This not only enables intuitive and efficient management of robots in a factory but also enables appropriate support according to the emotions of workers.

[1495] Definition of terms

[1496] "User" refers to a person who uses the system to upload image data and generate three-dimensional models and set animations.

[1497] A "terminal" is any device operated by a user, including smartphones, tablets, and personal computers.

[1498] "Image data" refers to digital images such as two-dimensional illustrations and paintings uploaded by users.

[1499] "Server" refers to a central processing unit that analyzes image data, generates three-dimensional models, adds animation, and sets the position of the augmented reality display.

[1500] A "two-dimensional image" refers to a digital image that has a flat composition, usually having width and height.

[1501] A "3D model" refers to a three-dimensional digital object that has width, height, and depth in three-dimensional space.

[1502] "Animation" refers to the process of adding movement to a three-dimensional model to create motion.

[1503] "Augmented reality" refers to the technology of overlaying digital content onto the real world.

[1504] "Location information" refers to data that indicates where in the real world a three-dimensional model is to be displayed.

[1505] An "emotion recognition engine" refers to software or hardware that acquires and analyzes a user's emotional data and adjusts the system's behavior based on that information.

[1506] An "action" refers to a movement or motion instruction that a user specifies for a three-dimensional model.

[1507] MODE FOR CARRYING OUT THE INVENTION

[1508] This invention relates to a system that allows users to convert two-dimensional images into three-dimensional models, add movement, and display them in augmented reality. It also recognizes the user's emotions and enables the generation of personalized content.

[1509] Program Generation

[1510] 1. Ability for users to upload illustrations

[1511] A user uses a device to upload image data (e.g., a two-dimensional illustration). The device sends the image data to the server, which then stores the received image data in a temporary storage area. This allows the server to check the integrity of the illustration before proceeding to the next step.

[1512] 2. 3D conversion function for illustrations

[1513] The server analyzes the received image data and extracts key features and contours. Based on the analysis results, it creates initial data for generating a 3D model from the 2D image. It then invokes OpenAI's generative AI model to convert the 2D illustration into a 3D model. The generated 3D model is saved in a database and the user is notified when the conversion is complete.

[1514] 3. 3D model animation function

[1515] The user uses a terminal to input specific actions (e.g., "run" or "jump") for the 3D model. The terminal sends this to the server, which analyzes the instructions. Based on the analysis results, Text2Motion technology is applied to generate animation data, adding movement to the 3D model. The generated animation data is stored in a database, and the user is notified when it is complete.

[1516] 4. AR display position setting function

[1517] The user uses the device's camera function to determine the position where the 3D model will be displayed in the real world. The device then sends the configured augmented reality display position information to the server, which then analyzes the received position information and generates data to display the 3D model at the specified position. The device then displays the model as augmented reality based on the received data.

[1518] 5. Implementing the Emotion Engine

[1519] The device is equipped with sensors (e.g., facial recognition using a camera, voice analysis using a microphone) to acquire the user's emotions and uses these to collect emotional data. The collected emotional data is sent to a server, which analyzes the data using an emotion engine. The server adjusts the movement and animation of the 3D model based on the user's emotions and sends the adjustment results to the device. The device then updates the display of the 3D model based on this.

[1520] Hardware and software used

[1521] Hardware: smartphone, tablet, computer, camera, microphone

[1522] Software: OpenAI generated AI models, Text2Motion technology, emotion recognition software

[1523] Specific examples

[1524] A factory manager uploads a diagram of the robot's layout within the factory and issues the command "move to maintenance position." As a result, a 3D model of the robot is generated and an animation of it moving to the designated maintenance position is added. The user's device analyzes the emotional data, and if, for example, fatigue is detected, the operation interface is simplified. This improves work efficiency and safety.

[1525] Example prompts for generative AI models

[1526] "Convert the provided 2D layout image of the factory robots into a 3D model. Add the animation for the command "move to maintenance position"."

[1527] In this way, the present invention is a system that allows users to easily convert 2D illustrations into 3D models, animate them, and display them as augmented reality. Furthermore, by combining it with an emotion engine, it is possible to provide personalized content that responds to the user's emotions.

[1528] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1529] Program processing steps

[1530] Step 1:

[1531] Users upload illustrations

[1532] operation

[1533] The user opens the app on their device, selects the image data (e.g., a two-dimensional illustration) they want to upload from the gallery, and presses the "Upload" button to send the image data to the server.

[1534] input

[1535] Image data selected on the device

[1536] output

[1537] Notification of completion of uploading image data to the server

[1538] Data processing and calculation

[1539] Image data is sent from the terminal to the server, and the server stores the data in a temporary storage area. It checks the consistency and, if there are no problems, proceeds to the next processing step.

[1540] Step 2:

[1541] The server converts the illustration into 3D.

[1542] operation

[1543] The server analyzes the uploaded image data, extracts key features and contours, and generates a 3D model from the 2D image based on the analysis results.

[1544] input

[1545] Uploaded image data

[1546] output

[1547] Generated 3D model data

[1548] Data processing and calculation

[1549] The server applies its own image analysis algorithms to detect features in the 2D illustration, then invokes OpenAI's generative AI model to generate a 3D model using a prompt, which is then stored in a database.

[1550] Step 3:

[1551] Add animation to a 3D model

[1552] operation

[1553] The user inputs actions (e.g., "run" or "jump") for the three-dimensional model through the app on the device and sends them to the server.

[1554] input

[1555] User-specified actions (e.g., "run," "jump")

[1556] output

[1557] Animated 3D models

[1558] Data processing and calculation

[1559] The server analyzes the user's instructions and generates animation data for the specified actions using Text2Motion technology. The generated animation data is added to the 3D model and stored in a database.

[1560] Step 4:

[1561] Setting the AR display position

[1562] operation

[1563] The user uses the device's camera to determine where in the real world the 3D model will be displayed, and then drags and positions the model on the device screen.

[1564] input

[1565] Real-world images captured through a camera and a display position specified by the user

[1566] output

[1567] 3D model displayed at the specified position

[1568] Data processing and calculation

[1569] The device sends camera images to the server in real time, acquires the set location information, and sends it back to the server. The server analyzes the location information and generates data to display a 3D model at that location. The device displays the model at the specified location based on the received data.

[1570] Step 5:

[1571] Utilizing the Emotion Engine

[1572] operation

[1573] The device collects emotion data using sensors (e.g., facial recognition by a camera, voice analysis by a microphone) to obtain the user's emotion. The collected data is sent to a server.

[1574] input

[1575] Emotion data from cameras and microphones

[1576] output

[1577] 3D model movements and animations adjusted based on emotion data

[1578] Data processing and calculation

[1579] The server analyzes the emotion data using an emotion engine and adjusts the movements and animations of the 3D model based on the user's emotions. The adjustment results are sent to the terminal, and the display is updated.

[1580] The above is a series of processing steps from uploading image data to displaying a 3D model using emotion recognition.

[1581] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1582] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1583] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1584] [Fourth embodiment]

[1585] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1586] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1587] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1588] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1589] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1590] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1591] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1592] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1593] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1594] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1595] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1596] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1597] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1598] This invention relates to a system that allows anyone to easily convert two-dimensional images into three-dimensional models, add animation, and display them in augmented reality for sharing. This system has the function of allowing users to upload image data from their devices, and a server analyzes the data to generate a three-dimensional model and add animation. It also has the function of allowing users to set the augmented reality display position of the three-dimensional model using their devices, and the server acquires the location information and displays it at the specified location.

[1599] A natural language description of the program's operation

[1600] 1. Ability for users to upload illustrations

[1601] The user opens the app on their device, selects the illustration they want to upload from the gallery, and presses the "Upload" button to send the selected illustration to the server.

[1602] The server receives the uploaded illustration data and stores it in a temporary storage area. It checks the integrity of the file and, if there are no problems, proceeds to the next step.

[1603] 2. 3D conversion function for illustrations

[1604] The server analyzes the received illustration data, extracts key features and contours, and creates initial data for generating a 3D model from the 2D image based on the analysis results.

[1605] The server calls the generative AI model to convert the 2D illustration into a 3D model, which is then saved in a database and a notification is sent to the user when the conversion is complete.

[1606] 3. 3D model animation function

[1607] The user uses the chat box on the terminal to input actions (e.g., "walk," "jump," etc.) for the three-dimensional model and sends them to the server.

[1608] The server analyzes the user's instructions, generates the corresponding animation using Text2Motion technology, and adds it to the 3D model. The generated animation data is also saved in the database. Once completed, the server notifies the user of the results.

[1609] 4. AR display position setting function

[1610] The user uses the device's camera function to determine where in the real world the 3D model should be displayed, specifically by dragging and positioning the 3D model on the device screen.

[1611] The terminal transmits the set position information to the server, and the server analyzes the information to determine the display position of the three-dimensional model in the real world.

[1612] The server sends data for displaying the 3D model at the specified location to the terminal, and the terminal displays the 3D model at the specified location as augmented reality based on the received data.

[1613] Specific examples

[1614] When a child uploads an illustration of a character they've drawn and commands "run" via chat, the character is converted into a three-dimensional model and a running animation is added. The user can then use their device's camera to position the character in their living room. This positioning information is sent to the server, and the character is displayed in augmented reality, running at the specified location in the living room. The user can view this display or share it with others.

[1615] As described above, the present invention relates to a system that allows users to easily convert two-dimensional illustrations into three-dimensional models, animate them, and display them as augmented reality. This system can provide creative activities and new communication methods.

[1616] The processing flow will be explained below.

[1617] Specific processing flow of the program

[1618] 1. Ability for users to upload illustrations

[1619] Step 1:

[1620] The user opens the app on their device and selects the illustration they want to upload from the gallery.

[1621] When the user presses the "Upload" button, the terminal sends the selected illustration to the server.

[1622] Step 2:

[1623] The server receives the transmitted illustration data and stores it in a temporary storage area.

[1624] The server performs a file integrity check to ensure the file was uploaded successfully.

[1625] Step 3:

[1626] The server analyzes the quality of the uploaded illustrations and, if necessary, notifies the user of suggestions for improving the quality.

[1627] 2. 3D conversion function for illustrations

[1628] Step 1:

[1629] The server analyzes the received illustration data and extracts key features and contours, allowing each part of the illustration to be recognized.

[1630] Step 2:

[1631] Based on the analysis results, the server creates initial data for generating a three-dimensional model from the two-dimensional image.

[1632] Step 3:

[1633] The server invokes a generative AI model (e.g., generative AI model technology) to convert the 2D illustration into a 3D model.

[1634] The server stores the generated 3D model in a database.

[1635] Step 4:

[1636] The server notifies the user when the 3D rendering is complete and provides a link or preview for confirmation.

[1637] 3. 3D model animation function

[1638] Step 1:

[1639] The user opens a chat box through the device app and inputs actions (e.g., "run," "jump," etc.) for the three-dimensional model.

[1640] When the user sends an instruction, the terminal sends the instruction to the server.

[1641] Step 2:

[1642] The server receives instructions from the user and analyzes the content of the instructions. For example, if the instruction to "run" is sent, the server analyzes the action.

[1643] Step 3:

[1644] The server uses Text2Motion technology to generate animation data for adding movement to the three-dimensional model based on the analyzed instructions.

[1645] Step 4:

[1646] The server saves the generated animation data in the 3D model database and notifies the user when it is complete.

[1647] 4. AR display position setting function

[1648] Step 1:

[1649] The user uses the device's camera to determine where in the real world the 3D model should be displayed, and then drags and positions the 3D model on the device screen.

[1650] Step 2:

[1651] The terminal transmits the set position information of the augmented reality display to the server.

[1652] Step 3:

[1653] The server analyzes the received location information and generates data for displaying the three-dimensional model at the specified location.

[1654] Step 4:

[1655] The server transmits this data to the terminal, and the terminal displays the three-dimensional model as augmented reality at the specified position based on the received data.

[1656] For example, a child can upload an illustration of a character they have drawn and use chat to say "run." As a result, the character is generated as a 3D model, a running animation is added, and it is displayed as augmented reality on the living room floor of the user's home. The user can then use their device's camera to watch the character running around the living room.

[1657] Example 1

[1658] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1659] With conventional technologies, converting 2D images into 3D models, adding movement, and displaying them in augmented reality is extremely difficult and requires specialized knowledge and time. Furthermore, there is a lack of easy ways for users to set their desired actions and location information, creating a demand for intuitive and easy operation. This creates a need for an efficient system for creating and sharing creative content.

[1660] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1661] In this invention, the server includes means for a user to upload image data from a terminal, means for analyzing the image data received by the server and generating a 3D model from a 2D image, means for converting the 2D image into a 3D model using a generative AI model, means for adding animation to the 3D model based on an action specified by the user, means for the user to use the terminal to set a position where the 3D model is to be displayed as augmented reality in the real world, and means for the server to acquire the set position information and display the 3D model at the specified position. This allows users to easily convert 2D illustrations into 3D models, add animation, and display and share them as augmented reality.

[1662] A "user" is any individual or organization who wishes to use the system to convert two-dimensional images into three-dimensional models and display them as augmented reality.

[1663] "Terminal" refers to a device operated by a user to upload image data, place 3D models, and specify actions. Specifically, this includes smartphones, tablets, and PCs.

[1664] A "server" is a computer or network system that receives and analyzes data sent from a terminal, generates a three-dimensional model from a two-dimensional image, adds actions, and performs calculations to display it as augmented reality.

[1665] "Image data" is digital data containing visual information such as two-dimensional illustrations and photographs that are uploaded by users from their terminals.

[1666] "Analysis" is the process by which the server understands the structure and characteristics of the image data it receives and extracts the necessary information.

[1667] A "two-dimensional image" is flat image data that has two dimensions, vertical and horizontal.

[1668] A "three-dimensional model" is digital data that has three dimensions - length, width, and depth - and can be displayed three-dimensionally in a virtual space.

[1669] "Generative AI model" refers to an algorithm or technique that uses artificial intelligence to convert two-dimensional images into three-dimensional models.

[1670] "Animation" is digital data that adds movement to a three-dimensional model and makes it operate continuously.

[1671] "Text2Motion technology" is a technology that analyzes text instructions entered by the user and adds corresponding movements to a three-dimensional model.

[1672] "Augmented reality (AR)" is a technology that displays computer-generated visual information overlaid on a real-world environment.

[1673] "Location information" is digital data relating to a location designated by a user to display a three-dimensional model in the real world.

[1674] A "database" is a storage device or system for storing and managing three-dimensional models and animation data.

[1675] The present invention relates to a system that allows users to easily convert two-dimensional images into three-dimensional models, add motion, and display and share them as augmented reality. The system has several main functions, each of which is realized by specific hardware and software.

[1676] The system's hardware includes terminal devices such as smartphones, tablets, and PCs, as well as a server, while the software uses the generative AI model "DeepArt3D" and Text2Motion technology.

[1677] Ability for users to upload illustrations

[1678] Users launch the application using a terminal device such as a smartphone or PC and select the 2D illustration they want to upload from local storage or a gallery. The selected illustration is sent from the device to the server. The server receives the uploaded illustration data and stores it in a temporary storage area. Once the integrity of the file is confirmed, the process proceeds to the next step.

[1679] 3D conversion function for illustrations

[1680] The server analyzes the received illustration data and uses the DeepArt3D model to generate a 3D model from the 2D image. First, it extracts the main features and contours of the illustration and creates initial data for the 3D model based on them. Next, the DeepArt3D model uses this initial data to generate the 3D model. The generated 3D model is saved in a database, and the user is notified when the conversion is complete.

[1681] 3D model animation function

[1682] The user opens the chat box on their device and inputs actions for the 3D model (such as "walk" or "jump") in text format. The input instructions are sent to the server, which uses Text2Motion technology to generate the corresponding animation and add it to the 3D model. The generated animation data is saved in a database, and the user is notified when the animation has been added.

[1683] For example, if a user inputs "Make this character run," the server will analyze this prompt and generate and add a "running" animation using Text2Motion technology. This method allows users to easily implement their desired movements in a 3D model.

[1684] AR display position setting function

[1685] The user activates the device's camera and decides where in the real world to display the 3D model. Specifically, they drag and drop the 3D model onto the device screen to place it. The device then sends this location information to the server, which analyzes it and determines where to display the 3D model. The server then sends the necessary data to the device, and the device displays the 3D model at the specified location as augmented reality.

[1686] For example, if a child uploads an illustration of a character and commands "run" via chat, the character is converted into a three-dimensional model and a running animation is added. Furthermore, the user can use the device's camera function to position the character in their living room. This positioning information is sent to the server, and the character is displayed in augmented reality as if running at the specified location in the living room.

[1687] As described above, the present invention relates to a system that allows users to intuitively operate it, quickly and easily convert two-dimensional illustrations into three-dimensional models, and then use them as augmented reality. This system provides new ways of creative activities and communication, and greatly enhances user convenience.

[1688] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1689] Step 1:

[1690] The user launches the application using their device. The user selects the 2D illustration they want to upload from local storage or the gallery. The selected illustration data is sent from the device to the server by pressing the "Upload" button.

[1691] Input: 2D illustration data selected from local storage or gallery

[1692] Output: Illustration data sent to the server

[1693] Step 2:

[1694] The server receives the uploaded illustration data and stores it in a temporary storage area. The server then checks the integrity of the file and, if there are no problems, begins analysis.

[1695] Input: Received illustration data

[1696] Output: Illustration data with consistency confirmed

[1697] Step 3:

[1698] The server analyzes the received illustration data, extracting the main features and contours of the image and creating initial data for generating a 3D model from the 2D image.

[1699] Input: Illustration data that has passed the consistency check

[1700] Output: Initial data including key feature and contour extraction results

[1701] Step 4:

[1702] The server calls the generative AI model "DeepArt3D" and converts the 2D illustration into a 3D model using the initial data. At this time, the server constructs the model and adds texture.

[1703] Input: Initial data

[1704] Output: Generated 3D model

[1705] Step 5:

[1706] The server stores the generated 3D model in a database and also notifies the user that the conversion of the 3D model is complete.

[1707] Input: Generated 3D model

[1708] Output: 3D model stored in database, notification to user

[1709] Step 6:

[1710] The user opens a chat box on the terminal and inputs text for actions to be performed on the 3D model (for example, "walk," "jump," etc.). The input instructions are sent from the terminal to the server.

[1711] Input: User instruction text

[1712] Output: Instruction text sent to the server

[1713] Step 7:

[1714] The server analyzes the user's text instructions and generates the corresponding animation using Text2Motion technology, a process that adds movement to the 3D model based on the user's instructions.

[1715] Input: Instruction text

[1716] Output: An animated 3D model

[1717] Step 8:

[1718] The server stores the animated 3D model in a database and also notifies the user that the animation has been completed.

[1719] Input: An animated 3D model

[1720] Output: Animated 3D model stored in database, notification to user

[1721] Step 9:

[1722] The user activates the device's camera function and sets the location in the real world where they want to display the 3D model on the device screen. Specifically, the user drags and drops the 3D model onto the device screen to place it. The set location information is sent to the server.

[1723] Input: Set location information

[1724] Output: Location information sent to the server

[1725] Step 10:

[1726] The server analyzes the received location information and determines the display position of the 3D model in the real world. It then transmits the necessary data to the device and displays the 3D model in the specified position as augmented reality.

[1727] Input: Location

[1728] Output: Display data of a 3D model of the real world generated from engine data

[1729] Step 11:

[1730] The device uses the received display data to display the 3D model as augmented reality at a specified location in the real world, allowing the user to view and share the display with others.

[1731] Input: Display data

[1732] Output: Augmented reality display of 3D model in the real world

[1733] (Application example 1)

[1734] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1735] Conventional systems that convert 2D images into 3D models and add animation have had the problem of being complex to operate, making them difficult for users to operate intuitively. Furthermore, when displaying the generated 3D model in the real world, it is difficult for users to set the exact display position, which is time-consuming. Furthermore, in advertising and promotions, there is a lack of user-participation interactive elements, and there is a lack of a way to easily create visually appealing content.

[1736] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1737] In this invention, the server includes: means for a user to upload image data from a terminal; means for the server to analyze the received image data and generate a three-dimensional model from a two-dimensional image; means for the server to animate the generated three-dimensional model; means for the user to set a position in the real world where the three-dimensional model will be displayed as augmented reality using the terminal; means for the server to acquire the set position information and display the three-dimensional model at the specified position; and means for analyzing the user's action instructions regarding the three-dimensional model and generating corresponding animations. This allows a user to generate a three-dimensional model from a two-dimensional image with simple operations, add animation to the model, and display it in augmented reality. Furthermore, incorporating interactive elements makes it easy to create visually appealing advertisements and promotional content.

[1738] "User" refers to the entity that uses the system to upload image data, generate three-dimensional models, add animations, and set display positions as augmented reality.

[1739] A "terminal" is a device used by a user, and refers to an electronic device such as a smartphone, smart glasses, or a head-mounted display.

[1740] "Image data" refers to two-dimensional visual information uploaded by a user through a terminal.

[1741] A "server" refers to a computing device that receives image data from a terminal, analyzes it, generates a three-dimensional model from the two-dimensional image, adds animation, and notifies the user of the results.

[1742] A "two-dimensional image" refers to image information that is composed of two dimensions, vertical and horizontal.

[1743] A "three-dimensional model" refers to a three-dimensional model expressed in three dimensions: length, width, and height.

[1744] "Animation" refers to a technique of expression that gives the visual impression of movement by adding movement to a three-dimensional model.

[1745] "Augmented reality" refers to the technology of overlaying digital information (in this case, three-dimensional models) onto a real-world environment.

[1746] "Location information" refers to data that allows a user to use a device to set where in the real world a three-dimensional model is to be displayed.

[1747] An "action instruction" refers to input information instructing a three-dimensional model to perform a specific action or movement.

[1748] The present invention provides a system that allows a user to convert a two-dimensional image into a three-dimensional model, add movement to the model, and display it in augmented reality. The following description will explain in detail how to specifically implement the present invention.

[1749] 1. System Configuration

[1750] The system for implementing the present invention mainly uses the following hardware and software.

[1751] Hardware:

[1752] User devices: smartphones (Android or iOS), smart glasses, head-mounted displays

[1753] Server: A cloud-based computing device (such as AWS or Google Cloud)

[1754] software:

[1755] Smartphone applications: Kotlin for Android and Swift for iOS

[1756] Server software: Python, TensorFlow for generative AI models, OpenCV for image analysis

[1757] Database: PostgreSQL

[1758] 2. Program Processing Overview

[1759] The program on the server performs the following process.

[1760] 1. Ability for users to upload illustrations:

[1761] The user launches the app on their smartphone, selects the 2D image they want to upload from their gallery, and presses the "Upload" button.

[1762] The server receives the uploaded image data and stores it in a temporary storage area. It checks the integrity of the data and, if there are no problems, proceeds to the next step.

[1763] 2. 3D conversion function for illustrations:

[1764] The server analyzes the received image data, extracts key features and contours, and creates initial data for generating a 3D model from the 2D image based on the analysis results.

[1765] The server invokes the generative AI model to convert the 2D image into a 3D model, which is then stored in a database and a notification is sent to the user when the conversion is complete.

[1766] 3. 3D model animation features:

[1767] The user uses the chat box on their smartphone to instruct the 3D model to take actions (e.g., "spin," "jump," "glow," etc.) and sends the instructions to the server.

[1768] The server analyzes the user's instructions, generates the corresponding animation using Text2Motion technology, and adds it to the 3D model. The generated animation data is also saved in a database, and the user is notified of the results once the process is complete.

[1769] 4. AR display position setting function:

[1770] The user uses the smartphone's camera to decide where in the real world the 3D model should be displayed, by dragging and positioning the 3D model on the device screen.

[1771] The terminal transmits the set position information to the server, and the server analyzes the information to determine the display position of the three-dimensional model in the real world.

[1772] The server transmits data for displaying a three-dimensional model at a specified position to the terminal, and the terminal displays the three-dimensional model at the specified position as augmented reality based on the received data.

[1773] Specific examples

[1774] Suppose a company wants to run an advertising campaign. A company representative opens a smartphone app and uploads a promotional company logo. Then, they type "jump" into the chat box. The server analyzes the logo, converts it into a 3D model, and adds a jumping animation to the model. The representative then places a promotional character on a cafe table, which is displayed in the real world as augmented reality. Examples of prompts used in this process include "upload an image file selected by the user to the server," "extract key features from the 2D logo and convert it into a 3D model," and "add a rotation animation to the 3D model."

[1775] Through the above process, users can intuitively and simply convert two-dimensional images into three-dimensional models, add movement to the models, and display them in the real world as augmented reality, making it possible to generate visually appealing advertisements and promotional content.

[1776] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1777] Program processing steps

[1778] Step 1:

[1779] The user launches the smartphone app, selects a two-dimensional image (e.g., a company logo) from the gallery, and presses the "Upload" button.

[1780] Input: 2D image data from the user's terminal

[1781] Operation: The terminal sends the selected image data to the server.

[1782] Output: 2D image data uploaded to the server

[1783] Step 2:

[1784] The server analyzes the received image data and stores it in a temporary storage area. It checks the data for consistency and, if there are no problems, it proceeds to the next step.

[1785] Input: Received image data

[1786] Action: Check the data integrity and save it to a temporary storage area.

[1787] Output: Image data with integrity confirmed

[1788] Step 3:

[1789] The server extracts the main features and contours of the received image data, and based on the analysis results, creates initial data for generating a 3D model from the 2D image.

[1790] Input: Image data whose integrity has been verified

[1791] How it works: Feature extraction is performed using OpenCV.

[1792] Output: Key features and contours

[1793] Step 4:

[1794] The server invokes a generative AI model (TensorFlow) to convert the 2D image into a 3D model, which is then saved in a database and a notification is sent to the user when the conversion is complete.

[1795] Input: feature and contour data

[1796] How it works: Generates a 3D model using a TensorFlow model and stores it in a database

[1797] Output: 3D model data, conversion completion notification

[1798] Step 5:

[1799] The user uses the chat box on their smartphone to instruct the 3D model to take an action (e.g., "spin," "jump," "glow") and send it to the server.

[1800] Input: User instruction for action

[1801] Action: Enter instructions in the chat box and send them to the server

[1802] Output: Action instructions sent to the server

[1803] Step 6:

[1804] The server analyzes the user's instructions, generates the corresponding animation using Text2Motion technology, and adds it to the 3D model. This animation data is also stored in the database, and the result is notified to the user after completion.

[1805] Input: User instruction for action

[1806] Action: Animation is generated using Text2Motion technology and saved in the database

[1807] Output: 3D model data with animation, notification of completion of generation

[1808] Step 7:

[1809] Users use the smartphone's camera to decide where in the real world they want to display the 3D model, then drag and place the model on the device screen.

[1810] Input: User's display position setting information

[1811] Operation: Set the display position using the camera function and send the position information to the server

[1812] Output: Display position information sent to the server

[1813] Step 8:

[1814] The server analyzes the location information set by the user and determines the display position of the 3D model in the real world. The server then sends data to the device to display the 3D model at the specified position.

[1815] Input: Display location information

[1816] Operation: Analyzes location information, generates data for augmented reality display, and sends it to the device

[1817] Output: Augmented reality display data

[1818] Step 9:

[1819] Based on the data received from the server, the device displays a three-dimensional model as augmented reality at a specified location.

[1820] Input: Augmented reality display data

[1821] Operation: A 3D model is superimposed on the camera image based on the received data.

[1822] Output: 3D model displayed as augmented reality in the real world

[1823] The above are the processing steps for carrying out the invention.

[1824] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1825] The present invention relates to a system that allows anyone to easily convert two-dimensional images into three-dimensional models, add movement, and display them in augmented reality for sharing. The present invention enables the generation of more personalized content by combining an emotion engine that recognizes user emotions. This system has a function that allows users to upload image data from their devices, and a server analyzes the data to generate a three-dimensional model and add animation. The system also has a function that allows users to use their devices to set the augmented reality display position of the three-dimensional model, and the server acquires that position information and displays it at the specified location. Additionally, the emotion engine can recognize user emotions and change behavior or adjust animations based on those emotions.

[1826] A natural language description of the program's operation

[1827] 1. Ability for users to upload illustrations

[1828] The user opens the app on their device, selects the illustration they want to upload from the gallery, and presses the "Upload" button to send the selected illustration to the server.

[1829] The server receives the illustration data and stores it in a temporary storage area. It checks the integrity of the file and, if there are no problems, proceeds to the next step.

[1830] 2. 3D conversion function for illustrations

[1831] The server analyzes the received illustration data, extracts key features and contours, and creates initial data for generating a 3D model from the 2D image based on the analysis results.

[1832] The server calls the generative AI model and converts the 2D illustration into a 3D model, which is then stored in a database.

[1833] The server notifies the user when the 3D rendering is complete and provides a link or preview for confirmation.

[1834] 3. 3D model animation function

[1835] The user opens a chat box through the app on their device, inputs actions for the three-dimensional model (e.g., "run," "jump," etc.), and sends them to the server.

[1836] The server receives instructions from the user and analyzes the content of the instructions. For example, if the instruction to "run" is sent, the server analyzes the action.

[1837] The server uses Text2Motion technology to generate animation data for adding movement to the three-dimensional model based on the analyzed instructions.

[1838] The server saves the generated animation data in the 3D model database and notifies the user when it is complete.

[1839] 4. AR display position setting function

[1840] The user uses the device's camera to determine where in the real world the 3D model should be displayed, and then drags and positions the 3D model on the device screen.

[1841] The terminal transmits the set position information of the augmented reality display to the server.

[1842] The server analyzes the received location information and generates data for displaying the three-dimensional model at the specified location.

[1843] The server transmits this data to the terminal, and the terminal displays the three-dimensional model as augmented reality at the specified position based on the received data.

[1844] 5. Implementing the Emotion Engine

[1845] The device is equipped with sensors (e.g., facial recognition using a camera, voice analysis using a microphone) to acquire the user's emotions, and uses these to collect the user's emotional data.

[1846] The device transmits emotion data to the server, which then analyzes the data using an emotion engine.

[1847] The server adjusts the behavior and animation of the 3D model based on the user's emotions, for example, making the model more active if the user is happy, or slowing down if the user is sad.

[1848] The server transmits the adjustment results to the terminal, and the terminal updates the display of the three-dimensional model based on the results.

[1849] Specific examples

[1850] A child uploads an illustration of a character they have drawn and commands "run" via chat. As a result, the character is generated as a 3D model and a running animation is added. The user then uses the camera function of their device to place the character on the floor of their living room. This placement information is sent to the server, and the character is displayed in augmented reality as if running at the specified location in the living room.

[1851] Furthermore, the user's device uses a camera to analyze their facial expressions, and if the user is having fun, the character will adjust to run more energetically. When the device detects that the user is happy, the server's emotion engine activates the character's movements, realizing behavior that matches the user's emotions.

[1852] As described above, the present invention relates to a system that allows users to easily convert 2D illustrations into 3D models, animate them, and display them as augmented reality. Furthermore, by combining this with an emotion engine, it is possible to provide personalized content that responds to the user's emotions.

[1853] The processing flow will be explained below.

[1854] Specific processing flow of the program (when combined with an emotion engine)

[1855] 1. Ability for users to upload illustrations

[1856] Step 1:

[1857] The user opens the app on their device and selects the illustration they want to upload from the gallery.

[1858] When the user presses the "Upload" button, the terminal sends the selected illustration to the server.

[1859] Step 2:

[1860] The server receives the transmitted illustration data and stores it in a temporary storage area.

[1861] The server performs a file integrity check to ensure the file was uploaded successfully.

[1862] Step 3:

[1863] The server analyzes the quality of the uploaded illustration and, if any parts are distorted or of low resolution, notifies the user with suggestions for correcting them.

[1864] 2. 3D conversion function for illustrations

[1865] Step 1:

[1866] The server analyzes the received illustration data and extracts key features and contours, which correspond to each part of the 2D illustration.

[1867] Step 2:

[1868] Based on the analysis results, the server extracts the initial data necessary to generate a basic three-dimensional model.

[1869] Step 3:

[1870] The server invokes a generative AI model to convert the 2D illustration into a 3D model, which is made up of data including shape and texture.

[1871] The server stores the generated 3D model in a database and notifies the user that the 3D model generation is complete.

[1872] 3. 3D model animation function

[1873] Step 1:

[1874] The user opens a chat box through the device app and inputs actions (e.g., "run," "jump," etc.) for the three-dimensional model.

[1875] When the user sends an instruction, the terminal sends the instruction to the server.

[1876] Step 2:

[1877] The server receives instructions from the user and analyzes the content of the instructions. For example, if the instruction to "run" is sent, the server analyzes the action in detail.

[1878] Step 3:

[1879] The server uses Text2Motion technology to generate animation data for adding movement to the three-dimensional model based on the analyzed instructions.

[1880] Step 4:

[1881] The server saves the generated animation data in the 3D model database and notifies the user when it is complete.

[1882] 4. AR display position setting function

[1883] Step 1:

[1884] The user uses the device's camera to determine where in the real world the 3D model should be displayed, and then drags and positions the 3D model on the device screen.

[1885] Step 2:

[1886] The terminal transmits the set position information of the augmented reality display to the server.

[1887] Step 3:

[1888] The server analyzes the received location information and generates data for displaying the three-dimensional model at the specified location.

[1889] Step 4:

[1890] The server transmits this data to the terminal, and the terminal displays the three-dimensional model as augmented reality at the specified position based on the received data.

[1891] 5. Implementing the Emotion Engine

[1892] Step 1:

[1893] The device is equipped with sensors (e.g., facial recognition using a camera, voice analysis using a microphone) to acquire the user's emotions, and uses these to collect the user's emotional data.

[1894] Step 2:

[1895] The device sends emotion data to the server, which then uses an emotion engine to analyze the data, for example, recognizing whether the user is surprised or happy through facial expression analysis.

[1896] Step 3:

[1897] The server adjusts the behavior and animation of the 3D model based on the user's emotions: for example, if the user is having fun, the model's movements will be more active, and conversely, if the user is sad, the model's movements will be more gentle.

[1898] Step 4:

[1899] The server transmits the adjustment results to the terminal, and the terminal updates the display of the three-dimensional model based on the results.

[1900] Specific examples

[1901] As a concrete example, a child uploads an illustration of a character they have drawn and commands "run" via chat. As a result, the character is generated as a 3D model and a running animation is added. The user then uses the camera function of their device to place the character on the floor of their living room. This placement information is sent to the server, and the character is displayed in augmented reality as if running at the specified location in the living room.

[1902] Furthermore, the user's device uses a camera to analyze their facial expressions, and if the user is having fun, the character will adjust to run more energetically. When the device detects that the user is happy, the server's emotion engine activates the character's movements, realizing behavior that matches the user's emotions.

[1903] Example 2

[1904] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1905] Currently, converting two-dimensional illustrations into three-dimensional models, adding movement to those models, and displaying them as augmented reality requires extensive specialized knowledge and complex software. These operations are difficult for ordinary users to perform due to the high technical barriers. Furthermore, the content created is rarely properly personalized based on the user's emotions. This limits the user experience and makes it difficult to create more engaging content.

[1906] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for a user to upload image data from a terminal, a means for the server to analyze the image data received and generate a three-dimensional model from a two-dimensional image, and a means for animating the three-dimensional model generated by the server. This allows a user, without specialized knowledge, to intuitively convert a two-dimensional illustration into a three-dimensional model, add animation, and display it as augmented reality. Furthermore, the terminal includes a means for acquiring the user's emotions and adjusting the behavior of the three-dimensional model based on the emotions, thereby enabling the generation of more personalized and attractive content.

[1907] A "user" is an entity that uses the system to upload image data and generate three-dimensional models and display augmented reality.

[1908] A "terminal" is an electronic device that a user uses to upload image data, such as a smartphone or tablet.

[1909] "Image data" refers to two-dimensional illustrations and image files uploaded by users.

[1910] A "server" is a computer system that analyzes received image data, generates three-dimensional models, animates them, and manages the augmented reality display.

[1911] "Analysis" is the process of extracting features and contours from received image data, which serve as the basis for generating a three-dimensional model from the two-dimensional image.

[1912] A "two-dimensional image" refers to a flat image that has only width and height information.

[1913] A "3D model" is a three-dimensional digital object that has width, height, and depth information.

[1914] "Animation" refers to the process of adding movement to a three-dimensional model.

[1915] "Augmented reality" refers to a technology that overlays digital information onto images of the real world.

[1916] "Location information" is data that indicates where in the real world a three-dimensional model is to be displayed.

[1917] "Emotion" refers to the psychological and sensory state that can be read from the user's facial expressions, voice, etc.

[1918] "Adjusting the behavior" refers to changing the animation of the three-dimensional model according to the user's emotions.

[1919] The present invention relates to a system that allows a user to easily convert a two-dimensional illustration into a three-dimensional model, add movement to the model, and display it as an augmented reality. An embodiment of the system will be described in detail below.

[1920] 1. Ability for users to upload illustrations

[1921] Users launch the dedicated application using a device such as a smartphone or tablet. They tap the "Select from Gallery" button within the application to open the device's image gallery and select the illustration they wish to upload. The user then presses the "Upload" button, which sends the illustration data to the server. The server receives the illustration data and stores it in a temporary storage area. At this time, the integrity of the file is also checked.

[1922] 2. 3D conversion function for illustrations

[1923] The server analyzes the received illustration data and extracts key features and contours. This process uses image analysis algorithms such as OpenCV. Based on the analysis results, the server invokes a generative AI model (e.g., DALL-E or Stable Diffusion) to convert the 2D illustration into a 3D model. The generated 3D model is stored in a database, and the user is notified when generation is complete. The notification includes a confirmation link and a preview image.

[1924] 3. 3D model animation function

[1925] The user inputs action instructions (e.g., "run," "jump," etc.) for the 3D model in the app's chat box and sends them to the server. The server analyzes the received instructions and generates animation data using Text2Motion technology. This animation data is also stored in the database, and the user is notified when generation is complete.

[1926] 4. AR display position setting function

[1927] The user activates the device's camera function and drags the 3D model on the screen to determine where in the real world the model should be displayed. The location information is sent to the server, which analyzes the received location information and generates augmented reality data for displaying the 3D model at the specified location. The generated data is sent to the device, which then displays the 3D model in the real world based on the received data.

[1928] 5. Implementing the Emotion Engine

[1929] The device uses sensors such as a camera and microphone to collect the user's emotional data in real time. For example, the camera recognizes the user's face and uses software (e.g., Emotion API) to analyze their facial expressions. The collected emotional data is sent to a server, which then analyzes it using an emotion engine. Based on the analysis results, the server adjusts the movement and animation of the 3D model and sends the results to the device. This causes the model's movement to change according to the user's emotions.

[1930] Specific examples

[1931] For example, a child uploads an illustration of a character they drew from their device and commands "run" in the chat box. In this case, the server generates a 3D model and adds a running animation. The user then uses the device's camera to place the character on the living room floor. Once the placement information is sent to the server, the character is displayed in augmented reality as if running in the specified position. Furthermore, if the device detects the user's facial expression of enjoyment, the server's emotion engine actively adjusts the character's movements.

[1932] Prompt Sentence Examples

[1933] "Turn your drawn character into a 3D model and run it around your living room."

[1934] By implementing these functions, the system of the present invention enables users to intuitively convert 2D illustrations into 3D models, animate them, and display them as augmented reality without requiring specialized knowledge. It also provides personalized content based on the user's emotions using an emotion engine.

[1935] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1936] Step 1:

[1937] Input: The user launches the app on their device and selects the 2D illustration to upload.

[1938] How it works: A user taps the "Select from Gallery" button in the app, opens the image gallery, selects the illustration they want to upload, and then presses the "Upload" button.

[1939] Data processing: The terminal prepares the selected illustration data and encodes the image data for transmission to the server.

[1940] Output: The encoded image data is sent to the server.

[1941] Step 2:

[1942] Input: The server receives the illustration data sent by the user.

[1943] Operation: The server receives the illustration data, stores it in a temporary storage area, and verifies the integrity of the file.

[1944] Data processing: Analyze the received data and apply integrity check algorithms.

[1945] Output: The illustration data whose consistency has been confirmed proceeds to the next processing step.

[1946] Step 3:

[1947] Input: Illustration data whose integrity has been confirmed.

[1948] How it works: The server analyzes the illustration data and extracts key features and contours using image analysis algorithms such as OpenCV.

[1949] Data processing: A feature extraction algorithm is run to obtain the features of the 2D illustration as data.

[1950] Output: The extracted feature data is used as input to a generative AI model.

[1951] Step 4:

[1952] Input: Extracted feature data.

[1953] How it works: The server invokes a generative AI model (e.g., DALL-E or Stable Diffusion) to generate a 3D model from a 2D illustration.

[1954] Data processing: Feature data is input into the generative AI model, and data calculations are performed to convert the two-dimensional illustration into a three-dimensional model.

[1955] Output: The generated 3D model data is saved in a database.

[1956] Step 5:

[1957] Input: Generated 3D model data.

[1958] What it does: The server notifies the user that generation is complete, including a confirmation link and a preview image.

[1959] Data processing: Generate notification data and send it to the user terminal.

[1960] Output: The user receives a link and a preview image to view the 3D model.

[1961] Step 6:

[1962] Input: The user inputs an action instruction (e.g., "run") in text format.

[1963] How it works: The user types instructions into the app's chat box and sends them to the server.

[1964] Data processing: The terminal collects text instructions from the user and converts them into a data format to be sent to the server.

[1965] Output: A text instruction is sent to the server.

[1966] Step 7:

[1967] Input: Text instruction data.

[1968] Operation: The server parses the received instruction data and uses Text2Motion technology to parse it.

[1969] Data processing: Use a natural language processing engine (e.g., GPT-3) to convert text instructions into concrete animation data.

[1970] Output: The generated animation data is saved in a database.

[1971] Step 8:

[1972] Input: Animation data.

[1973] Action: The server notifies the user that animation generation is complete.

[1974] Data processing: Generate notification data and send it to the user terminal.

[1975] Output: The user receives a notification that the animation data has been generated.

[1976] Step 9:

[1977] Input: Camera launch for user to set viewing position.

[1978] Action: The user activates the device's camera and drags the 3D model to place it in the real world.

[1979] Data processing: The device collects location data based on camera footage and converts it into a data format to be sent to the server.

[1980] Output: The location data is sent to the server.

[1981] Step 10:

[1982] Input: Location data.

[1983] Operation: The server analyzes the received location data and generates augmented reality data for displaying a three-dimensional model at the specified location.

[1984] Data processing: Generate data for augmented reality display based on location data.

[1985] Output: The generated augmented reality data is sent to the device.

[1986] Step 11:

[1987] Input: Augmented reality data.

[1988] How it works: The device uses the data it receives to display a 3D model in the real world.

[1989] Data processing: Using the device's AR engine, model data is overlaid onto the camera image.

[1990] Output: The user sees the 3D model displayed in the real world.

[1991] Step 12:

[1992] Input: User emotion data.

[1993] How it works: The device uses sensors (camera and microphone) to collect user emotional data in real time.

[1994] Data processing: The collected emotion data is initially analyzed on the device and converted into a data format that is sent to the server.

[1995] Output: Emotion data is sent to the server.

[1996] Step 13:

[1997] Input: Emotion data.

[1998] Operation: The server analyzes the emotion data to determine the user's emotional state.

[1999] Data Processing: An emotion engine is used to analyze the emotion data and identify the user's emotional state.

[2000] Output: The emotion analysis results are used as motion adjustment data for the 3D model.

[2001] Step 14:

[2002] Input: Sentiment analysis results.

[2003] Movement: The server adjusts the movement and animation of the 3D model based on the analysis results.

[2004] Data processing: Generate adjustment parameters and update the animation data of the 3D model.

[2005] Output: The adjusted animation data is sent to the device.

[2006] Step 15:

[2007] Input: The adjusted animation data.

[2008] Operation: The device updates the display of the 3D model based on the received data.

[2009] Data processing: Update the animation data in the device's AR engine to create new model behavior.

[2010] Output: The user can see a 3D model with movements adjusted according to the emotion.

[2011] (Application example 2)

[2012] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2013] Currently, operating and managing robots in factories requires complex procedures and specialized knowledge, making it difficult to provide effective operating instructions and maintenance information. Furthermore, there is a lack of support systems that take into account the emotions of workers, which can lead to a decline in operational efficiency and safety. In response to this situation, a more intuitive and effective robot management system is needed.

[2014] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for a user to upload image data from a terminal, means for the server to analyze the received image data and generate a 3D model from a 2D image, means for the server to animate the generated 3D model, means for the user to use the terminal to set a position at which the 3D model will be displayed in the real world as augmented reality, means for the server to acquire the set position information and display the 3D model at the specified position, means for acquiring and analyzing user emotion data using an emotion recognition engine, and means for adjusting the movement and animation of the 3D model based on the user emotion data. This not only enables intuitive and efficient management of robots in a factory but also enables appropriate support according to the emotions of workers.

[2015] Definition of terms

[2016] "User" refers to a person who uses the system to upload image data and generate three-dimensional models and set animations.

[2017] A "terminal" is any device operated by a user, including smartphones, tablets, and personal computers.

[2018] "Image data" refers to digital images such as two-dimensional illustrations and paintings uploaded by users.

[2019] "Server" refers to a central processing unit that analyzes image data, generates three-dimensional models, adds animation, and sets the position of the augmented reality display.

[2020] A "two-dimensional image" refers to a digital image that has a flat composition, usually having width and height.

[2021] A "3D model" refers to a three-dimensional digital object that has width, height, and depth in three-dimensional space.

[2022] "Animation" refers to the process of adding movement to a three-dimensional model to create motion.

[2023] "Augmented reality" refers to the technology of overlaying digital content onto the real world.

[2024] "Location information" refers to data that indicates where in the real world a three-dimensional model is to be displayed.

[2025] An "emotion recognition engine" refers to software or hardware that acquires and analyzes a user's emotional data and adjusts the system's behavior based on that information.

[2026] An "action" refers to a movement or motion instruction that a user specifies for a three-dimensional model.

[2027] MODE FOR CARRYING OUT THE INVENTION

[2028] This invention relates to a system that allows users to convert two-dimensional images into three-dimensional models, add movement, and display them in augmented reality. It also recognizes the user's emotions and enables the generation of personalized content.

[2029] Program Generation

[2030] 1. Ability for users to upload illustrations

[2031] A user uses a device to upload image data (e.g., a two-dimensional illustration). The device sends the image data to the server, which then stores the received image data in a temporary storage area. This allows the server to check the integrity of the illustration before proceeding to the next step.

[2032] 2. 3D conversion function for illustrations

[2033] The server analyzes the received image data and extracts key features and contours. Based on the analysis results, it creates initial data for generating a 3D model from the 2D image. It then invokes OpenAI's generative AI model to convert the 2D illustration into a 3D model. The generated 3D model is saved in a database and the user is notified when the conversion is complete.

[2034] 3. 3D model animation function

[2035] The user uses a terminal to input specific actions (e.g., "run" or "jump") for the 3D model. The terminal sends this to the server, which analyzes the instructions. Based on the analysis results, Text2Motion technology is applied to generate animation data, adding movement to the 3D model. The generated animation data is stored in a database, and the user is notified when it is complete.

[2036] 4. AR display position setting function

[2037] The user uses the device's camera function to determine the position where the 3D model will be displayed in the real world. The device then sends the configured augmented reality display position information to the server, which then analyzes the received position information and generates data to display the 3D model at the specified position. The device then displays the model as augmented reality based on the received data.

[2038] 5. Implementing the Emotion Engine

[2039] The device is equipped with sensors (e.g., facial recognition using a camera, voice analysis using a microphone) to acquire the user's emotions and uses these to collect emotional data. The collected emotional data is sent to a server, which analyzes the data using an emotion engine. The server adjusts the movement and animation of the 3D model based on the user's emotions and sends the adjustment results to the device. The device then updates the display of the 3D model based on this.

[2040] Hardware and software used

[2041] Hardware: smartphone, tablet, computer, camera, microphone

[2042] Software: OpenAI generated AI models, Text2Motion technology, emotion recognition software

[2043] Specific examples

[2044] A factory manager uploads a diagram of the robot's layout within the factory and issues the command "move to maintenance position." As a result, a 3D model of the robot is generated and an animation of it moving to the designated maintenance position is added. The user's device analyzes the emotional data, and if, for example, fatigue is detected, the operation interface is simplified. This improves work efficiency and safety.

[2045] Example prompts for generative AI models

[2046] "Convert the provided 2D layout image of the factory robots into a 3D model. Add the animation for the command "move to maintenance position"."

[2047] In this way, the present invention is a system that allows users to easily convert 2D illustrations into 3D models, animate them, and display them as augmented reality. Furthermore, by combining it with an emotion engine, it is possible to provide personalized content that responds to the user's emotions.

[2048] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[2049] Program processing steps

[2050] Step 1:

[2051] Users upload illustrations

[2052] operation

[2053] The user opens the app on their device, selects the image data (e.g., a two-dimensional illustration) they want to upload from the gallery, and presses the "Upload" button to send the image data to the server.

[2054] input

[2055] Image data selected on the device

[2056] output

[2057] Notification of completion of uploading image data to the server

[2058] Data processing and calculation

[2059] Image data is sent from the terminal to the server, and the server stores the data in a temporary storage area. It checks the consistency and, if there are no problems, proceeds to the next processing step.

[2060] Step 2:

[2061] The server converts the illustration into 3D.

[2062] operation

[2063] The server analyzes the uploaded image data, extracts key features and contours, and generates a 3D model from the 2D image based on the analysis results.

[2064] input

[2065] Uploaded image data

[2066] output

[2067] Generated 3D model data

[2068] Data processing and calculation

[2069] The server applies its own image analysis algorithms to detect features in the 2D illustration, then invokes OpenAI's generative AI model to generate a 3D model using a prompt, which is then stored in a database.

[2070] Step 3:

[2071] Add animation to a 3D model

[2072] operation

[2073] The user inputs actions (e.g., "run" or "jump") for the three-dimensional model through the app on the device and sends them to the server.

[2074] input

[2075] User-specified actions (e.g., "run," "jump")

[2076] output

[2077] Animated 3D models

[2078] Data processing and calculation

[2079] The server analyzes the user's instructions and generates animation data for the specified actions using Text2Motion technology. The generated animation data is added to the 3D model and stored in a database.

[2080] Step 4:

[2081] Setting the AR display position

[2082] operation

[2083] The user uses the device's camera to determine where in the real world the 3D model will be displayed, and then drags and positions the model on the device screen.

[2084] input

[2085] Real-world images captured through a camera and a display position specified by the user

[2086] output

[2087] 3D model displayed at the specified position

[2088] Data processing and calculation

[2089] The device sends camera images to the server in real time, acquires the set location information, and sends it back to the server. The server analyzes the location information and generates data to display a 3D model at that location. The device displays the model at the specified location based on the received data.

[2090] Step 5:

[2091] Utilizing the Emotion Engine

[2092] operation

[2093] The device collects emotion data using sensors (e.g., facial recognition by a camera, voice analysis by a microphone) to obtain the user's emotion. The collected data is sent to a server.

[2094] input

[2095] Emotion data from cameras and microphones

[2096] output

[2097] 3D model movements and animations adjusted based on emotion data

[2098] Data processing and calculation

[2099] The server analyzes the emotion data using an emotion engine and adjusts the movements and animations of the 3D model based on the user's emotions. The adjustment results are sent to the terminal, and the display is updated.

[2100] The above is a series of processing steps from uploading image data to displaying a 3D model using emotion recognition.

[2101] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[2102] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[2103] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[2104] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[2105] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[2106] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[2107] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[2108] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as...

Claims

1. A means for a user to upload image data from a terminal; means for analyzing the image data received by the server and generating a three-dimensional model from the two-dimensional image; means for animating the server-generated three-dimensional model; A means for a user to set a position where the three-dimensional model is to be displayed as augmented reality in the real world using a terminal; a means for the server to acquire the set location information and display the three-dimensional model at the specified location; A system including:

2. 2. The system according to claim 1, further comprising means for inspecting the quality of the image data analyzed by the server and notifying the user of suggestions for improving the quality.

3. 2. The system according to claim 1, further comprising means for the server to add animation to the three-dimensional model based on an instruction from the user and notify the user of the result.

Images