system

Abstract

A system is provided.SOLUTION: The system includes a means for acquiring information from an existing API, a means for analyzing the acquired information, a means for generating a 3D model on the basis of the analyzed information, a means for transmitting the generated 3D model to the client terminals, and a means for displaying the 3D model in the actual space by the client terminals.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] With the increasing popularity of augmented reality (AR) technology in recent years, there is a demand for services that combine real-world space with digital data. However, the information that can currently be obtained from existing APIs is limited to text data, and converting this data into 3D models requires specialized knowledge and high costs. For this reason, there is a need for a system that can quickly and efficiently generate 3D models that can be used in AR devices. Furthermore, the process for generating 3D models is complex, and there is a lack of an environment that is easy for general developers to use. Technology to solve this problem is eagerly awaited. [Means for solving the problem]

[0005] To solve this problem, the inventors have devised the following means. The present invention includes a means for acquiring information from an existing API and analyzing it. A means for generating a 3D model using AI based on the analyzed information is provided. Furthermore, a system is provided that includes a means for transmitting the generated 3D model to a client terminal and a means for the client terminal to display the 3D model in real space. This system can efficiently generate a 3D model from existing API information and make it available on an AR device. Specifically, a deep learning algorithm is used to convert the analyzed information into a 3D model, and the generated 3D model is converted into a standard file format (GLTF or OBJ). Furthermore, the system includes a means for the client terminal to overlay the received 3D model in real space using an AR engine. In this way, the system reduces development costs and enables rapid and efficient service provision.

[0006] "API" stands for Application Programming Interface, an interface for exchanging functions and data between programs.

[0007] "Analysis" is the process of breaking down and organizing the acquired data and extracting the necessary information.

[0008] A "3D model" is a digital representation of an object placed in three-dimensional space, and contains information such as shape, position, and orientation.

[0009] "AI" is an abbreviation for artificial intelligence, a technology that allows computers to imitate and carry out intellectual tasks performed by humans.

[0010] A "client terminal" is a user terminal that connects to a server on a network and receives services and data.

[0011] An "AR device" is a device that uses augmented reality (AR) technology to display and manipulate digital information in real space.

[0012] "Overlay display" is a method of displaying additional information or objects over existing real-world footage or images.

[0013] A "deep learning algorithm" is an algorithm that uses a multi-layer neural network to learn features from data and perform analysis and prediction.

[0014] "Standard file format" refers to a commonly used data format, and in this case specifically refers to GLTF and OBJ as formats for expressing 3D models. [Brief explanation of the drawings]

[0015] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11]FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0016] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0017] First, the terms used in the following description will be explained.

[0018] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0019] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0020] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0021] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0022] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0023] [First embodiment]

[0024] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0025] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0026] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0027] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0028] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0029] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0030] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0031] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0032] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0033] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0034] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0035] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0036] This invention relates to a system that analyzes information obtained from an API, generates a 3D model using AI, and displays it in AR on a client device. This system is primarily composed of a server and a client device, where the server obtains and analyzes API data and generates a 3D model, and the client device receives the generated 3D model and displays it in AR.

[0037] System configuration

[0038] 1. Server

[0039] Retrieving API data: The server requests the required data from the specified API. For example, to retrieve building information for a city, the server sends an HTTP request to the API endpoint and retrieves the data in JSON format.

[0040] Data analysis: The server analyzes the acquired JSON data and extracts the information necessary to generate a 3D model, such as the location, size, and shape of the building.

[0041] 3D model generation: The server uses the AI ​​model to generate a 3D model based on the analyzed data. Here, a deep learning algorithm is used.

[0042] Model conversion and provision: The generated 3D model is converted into a standard file format (e.g., GLTF format) and sent to the client terminal.

[0043] 2. Client Terminal

[0044] Receiving and displaying the model: The client device receives the 3D model sent from the server and uses the AR engine to overlay the model in real space.

[0045] Initializing the AR engine: The client device initializes a dedicated AR engine for AR display, for example, using a framework such as ARKit or ARCore.

[0046] Program processing

[0047] Server-side processing

[0048] The server first sends a request to an API endpoint to retrieve the required data.

[0049] The acquired data is analyzed to extract necessary information (e.g., position, size, shape, etc.).

[0050] Based on the extracted data, a 3D model is generated using an AI model, which is based on deep learning algorithms and has advanced shape recognition and generation capabilities.

[0051] The generated 3D model is converted into a standard format such as GLTF and sent to the client terminal as an HTTP response.

[0052] Processing on the client terminal side

[0053] The client terminal receives the 3D model from the server.

[0054] The device initializes the AR engine and loads the received 3D model data.

[0055] The loaded 3D model is overlaid in real space, providing visual feedback to the user.

[0056] Specific examples

[0057] Take the example of AR display of city building data.

[0058] 1. Acquiring API data

[0059] The server retrieves JSON-formatted data from the urban architecture data API, including information on the location, size, and shape of each building.

[0060] 2. Data Analysis

[0061] The server analyzes this data and extracts the location, size, and shape of each building.

[0062] 3. Generating 3D models

[0063] The server uses the extracted information to generate a 3D model of the building using an AI model that uses deep learning technology to reproduce even complex shapes with high accuracy.

[0064] 4. Convert and send the model

[0065] The generated 3D model is converted into GLTF format and sent to the client terminal as an HTTP response.

[0066] 5. Display on the client terminal

[0067] The client device uses an AR engine to overlay the received 3D model onto the real world, allowing users to see the digital building model superimposed on the real cityscape.

[0068] In this way, the present invention effectively utilizes existing API data to efficiently generate advanced 3D models and integrate them into real space, significantly reducing the cost and time required for developing 3D content and enabling rapid service provision.

[0069] The processing flow will be explained below.

[0070] Step 1:

[0071] The server sends an HTTP request to the specified API endpoint to retrieve the required data. This request may include authentication information and filtering conditions as needed. When the API returns a response to the request, the server receives the response data in JSON format.

[0072] Step 2:

[0073] The server analyzes the received JSON data. Specifically, it extracts necessary information from the response data, such as the building's location, size, shape, and color. The extracted data is then organized into an appropriate internal data structure as a preprocessing step for generating a 3D model.

[0074] Step 3:

[0075] The server uses an AI model to generate a 3D model based on the organized data. First, it uses a deep learning algorithm to input the data and run a process to generate a 3D model. At this time, the AI ​​model is pre-trained within the server.

[0076] Step 4:

[0077] The server converts the generated 3D model into a standard file format, such as GLTF or OBJ, which makes it easier for the client device to receive it. After the conversion process is complete, the file is sent to the client device as an HTTP response.

[0078] Step 5:

[0079] The device receives the 3D model data sent from the server. The received data is stored on the device and prepared for AR display. The device then sends this data to the AR engine and loads the 3D model.

[0080] Step 6:

[0081] The device initializes its AR engine, which includes calibrating the camera and scanning the environment, preparing it to accurately display 3D models in real space.

[0082] Step 7:

[0083] The device overlays the loaded 3D model onto the real world. This includes rendering to ensure the 3D model appears in the correct position when the user looks at the real world through the AR device. Once rendering is complete, the user can enjoy a visual experience in which the digital 3D model blends into the real world.

[0084] Through this series of steps, users can easily experience the fusion of real space and digital content.

[0085] Example 1

[0086] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0087] Conventional technology has the problem of being difficult to analyze information obtained from existing APIs, generate 3D models efficiently and with high accuracy based on that information, and display them in real space. It has also been pointed out that the accuracy and realism of the generated 3D models are not sufficient. To solve these problems, an advanced analysis and generation system utilizing AI technology is required.

[0088] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0089] In this invention, the server includes means for acquiring data from existing information sources, means for analyzing the acquired data, means for generating a 3D model based on the analyzed data, means for transmitting the generated 3D model to a customer terminal, and means for displaying the 3D model in real space by the customer terminal, thereby enabling the generation of a highly accurate 3D model and its overlay display in real space.

[0090] "Existing sources" refer to external systems or interfaces that provide data that already exists, such as databases or online services accessible through APIs.

[0091] "Data" refers to information such as specific numbers, characters, images, etc. that are the subject of calculation, analysis, and processing.

[0092] "Analyzing" refers to examining acquired data in detail and processing it to understand its structure and meaning.

[0093] A "solid model" refers to a computer-generated object that has three-dimensional (3D) geometric information, such as 3D data of a building or object.

[0094] "Customer terminal" refers to a computer device used by a user, including, for example, a smartphone, tablet, or personal computer.

[0095] "Displaying in real space" refers to using augmented reality technology to display digital data overlaid on the physical real world.

[0096] "Generative AI" refers to AI techniques that have the ability to generate new data based on specified input data, including deep learning models and neural networks.

[0097] An "artificial neural network" is an algorithm that mimics the activity of nerve cells in the human brain, and refers to a technology that learns from large amounts of data to recognize patterns and generate data.

[0098] This invention relates to a system that acquires data from existing information sources, analyzes the data, generates a highly accurate 3D model using AI technology, transmits the model to a customer's terminal, and displays it in real space. This system is mainly composed of a server and a customer's terminal.

[0099] Server program processing description

[0100] 1. Data Acquisition:

[0101] The server requests the required data from an existing information source (such as an API). It sends an HTTP request to an API endpoint and retrieves the data in JSON format. For example, to retrieve building information for a city, it uses a URL like "https: / / api.example.com / buildings / tokyo".

[0102] 2. Data Analysis:

[0103] The server parses the received JSON data and extracts information such as the building's location, size, shape, etc. The extracted data has the following structure:

[0104] Building 1: Latitude 35.6895, Longitude 139.6917, Height 150, Shape Rectangle

[0105] Building 2: Latitude 35.6581, Longitude 139.7516, Height 200, Shape Circle

[0106] 3. Generate 3D model:

[0107] The server uses the extracted data to provide prompts to a generative artificial intelligence (AI model), which is trained using deep learning frameworks such as TENSORFLOW® and PyTorch.

[0108] For example, enter the following prompt:

[0109] "latitude": 35.6895, "longitude": 139.6917, "height": 150, "shape": "rectangle"

[0110] The server uses an AI model to generate a 3D model from this data.

[0111] 4. Convert and send the model:

[0112] Convert the generated 3D model into the standard GLTF format. For example, save it as "model1.gltf".

[0113] The server sends this model data to the client terminal as an HTTP response.

[0114] Explanation of the client terminal program processing

[0115] 1. Receiving the model:

[0116] The client device receives the GLTF format 3D model sent from the server and saves it as temporary data, for example, "model1.gltf."

[0117] 2. Initialize the AR engine:

[0118] The customer device will initialize a dedicated AR engine (for example, Apple's ARKit or Google's ARCore). At this time, the settings will be configured to allow use of the camera and location information.

[0119] 3. Viewing the 3D model:

[0120] The customer's device loads the received GLTF model into its AR engine and displays it as an overlay in real space. A 3D model of the building is displayed in the direction the user points the camera.

[0121] Examples:

[0122] Taking the example of AR display of city building data, the server retrieves JSON-formatted building information data from "https: / / api.example.com / buildings / tokyo," extracts the location, size, and shape of each building, and supplies this to the AI ​​model. Using TensorFlow, the AI ​​model generates a highly accurate 3D model in response to a prompt, such as "latitude: 35.6895, longitude: 139.6917, height: 150, shape: rectangle," and sends it to the client device as "model1.gltf" in GLTF format. The client device receives this and uses ARKit to overlay the 3D model on the real world. This allows users to enjoy an AR experience where digital building models are overlaid on the Tokyo cityscape.

[0123] This system makes it possible to generate highly accurate 3D models and overlay them in real space.

[0124] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0125] Step 1:

[0126] The server retrieves data from an existing information source (API). Specifically, it sends an HTTP GET request to an API endpoint and receives JSON-formatted data as a response. For example, it retrieves city building information from "https: / / api.example.com / buildings / tokyo."

[0127] Input: API endpoint

[0128] Output: API response data in JSON format

[0129] Specific operation: Send a GET request to "https: / / api.example.com / buildings / tokyo" and receive JSON data containing information such as the building's location, size, and shape.

[0130] Step 2:

[0131] The server analyzes the received JSON data using a JSON parser to extract necessary information such as the location, size, and shape of each building.

[0132] Input: API response data in JSON format

[0133] Output: Information on the location, size, and shape of the extracted buildings

[0134] Specific behavior: Parse the received JSON data and convert it into the following structure:

[0135] Building 1: Latitude 35.6895, Longitude 139.6917, Height 150, Shape Rectangle

[0136] Building 2: Latitude 35.6581, Longitude 139.7516, Height 200, Shape Circle

[0137] Step 3:

[0138] The server inputs prompts into the AI ​​model based on the extracted data to generate a 3D model. Specifically, it inputs prompts into the AI ​​model (a deep learning model trained with TensorFlow or PyTorch) to generate 3D model data.

[0139] Input: Information on the location, size, and shape of the extracted buildings

[0140] Output: Generated 3D model

[0141] Specific behavior:

[0142] Prompt: "latitude": 35.6895, "longitude": 139.6917, "height": 150, "shape": "rectangle"

[0143] is input into the AI ​​model to generate a three-dimensional model.

[0144] Step 4:

[0145] The server converts the generated 3D model into the standard GLTF format and sends it to the client terminal. Specifically, it converts it into the GLTF format using a model conversion tool and sends it to the client terminal as an HTTP response.

[0146] Input: Generated 3D model

[0147] Output: 3D model data in GLTF format

[0148] Specific operation: The generated 3D model is saved as "model1.gltf" and sent to the client terminal.

[0149] Step 5:

[0150] The client device receives the GLTF format 3D model sent from the server and initializes the AR engine, specifically, a dedicated AR engine such as ARKit or ARCore, and makes the camera and location information available.

[0151] Input: 3D model data in GLTF format

[0152] Output: Initialized AR Engine

[0153] Specific operation: The client device receives "model1.gltf", initializes ARKit, and obtains camera access permission.

[0154] Step 6:

[0155] The client device loads the received GLTF model into the AR engine and displays it as an overlay in real space. Specifically, the GLTF model is loaded into the AR engine and displayed in the direction the user points the camera.

[0156] Input: Initialized AR engine, 3D model data in GLTF format

[0157] Output: 3D model displayed in real space

[0158] Specific operation: The client device loads "model1.gltf" into ARKit and displays a 3D model of the building in real space in the direction the user is pointing the camera.

[0159] This allows users to visually experience a digital three-dimensional model superimposed on a real-world scene.

[0160] (Application example 1)

[0161] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0162] In existing brick-and-mortar stores, customers often have to spend a lot of time finding specific products, and detailed product information is difficult to obtain. Stores also face the challenge of having few ways to communicate product layout changes and inventory status to customers in real time. To solve these problems, there is a need for technology that can easily obtain product location information within a store and present it to users in a visually easy-to-understand manner.

[0163] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0164] In this invention, the server includes means for acquiring information from an existing API, means for analyzing the acquired information, means for generating a 3D model based on the analyzed information, means for transmitting the generated 3D model to a client terminal, and means for the client terminal to provide product location information and detailed information to a user using the 3D model displayed in real space. This enables customers to quickly and accurately find specific products in a physical store and obtain detailed product information in real time.

[0165] "Means of obtaining information from an API" refers to a mechanism for requesting and receiving necessary information from external data providers via the Internet.

[0166] "Means for analyzing information" refers to a program or method for interpreting acquired data according to certain rules and extracting or converting necessary items.

[0167] "Means for generating a 3D model" refers to a technology or process for creating a digital model with a three-dimensional shape based on analyzed information.

[0168] "Means for transmitting the generated 3D model to the client terminal" refers to a mechanism for distributing the 3D model created on the server side to the client terminal via the Internet.

[0169] "Means for displaying a 3D model in real space using a client terminal" refers to a technology in which the client terminal uses a camera and display to visually display a 3D model superimposed on the real environment.

[0170] "A means by which a client terminal provides users with product location information and detailed information using a 3D model displayed in real space" is a mechanism that allows users to visually obtain product location and detailed information via a 3D model displayed in AR.

[0171] "Means for generating 3D models using AI" refers to a method that utilizes artificial intelligence technology to automatically create 3D models from analyzed information.

[0172] "Means using deep learning algorithms" refers to the process of using deep learning technology, a type of artificial intelligence, to analyze complex data patterns and generate 3D models.

[0173] This invention is a system that acquires information from an existing API, generates a 3D model based on that information, and displays it in real space on a client terminal, providing users with product location information and detailed information. Here we will explain the specific system configuration and how to implement it.

[0174] System configuration

[0175] 1. Server

[0176] The server sends a request to an external data provision service (API) via the Internet to obtain the required data.

[0177] The acquired data is analyzed to extract necessary information such as the product's position, shape, and size.

[0178] Based on the extracted information, a 3D model is generated using AI technology, using machine learning frameworks such as TensorFlow.

[0179] The generated 3D model is converted into GLTF format and sent to the client device.

[0180] 2. Client Terminal

[0181] The client terminal receives the 3D model sent from the server.

[0182] Using an AR engine (e.g., ARCore or ARKit), the received 3D model is overlaid onto the real world.

[0183] When a user holds the camera over a product, the product's location and detailed information are visually displayed through AR.

[0184] Specific processing details

[0185] 1. Data acquisition by the server

[0186] The server sends an HTTP request to the specified API to obtain JSON formatted data. For example, it uses an API to obtain the location and detailed information of a store's products.

[0187] 2. Data Analysis and 3D Model Generation

[0188] The acquired data is analyzed to extract the position, size, and shape of each product.

[0189] Based on the extracted information, a 3D model is generated using a machine learning model, using a framework such as TensorFlow.

[0190] 3. Convert and send 3D models

[0191] The generated 3D model is converted into GLTF format and sent to the client terminal as an HTTP response.

[0192] 4. AR display on client device

[0193] The client device loads the 3D model received from the server and initializes the AR engine.

[0194] A 3D model is overlaid onto the real space where the camera is pointed, providing product location information and detailed information.

[0195] Specific examples

[0196] As a specific scenario, the following prompt sentence is used:

[0197] "This application displays the location information of products in the store in real time, allowing you to easily search and view the location of specific products. It also uses AR to display detailed information about products (price, reviews, stock, etc.). ARCore is used for AR display, so accurate location display is possible."

[0198] This allows users to efficiently search for products in the store using their smartphones and easily obtain detailed information.

[0199] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0200] Step 1:

[0201] The server sends an HTTP request to an existing API via the Internet to obtain the required data. Specifically, it receives JSON-formatted data that includes the location, size, shape, etc. of in-store products. The input is the API request, and the output is the JSON data returned as the API response.

[0202] Step 2:

[0203] The server analyzes the acquired JSON data and extracts information about the product's position, size, and shape. Specifically, it parses the JSON data and extracts the parameters of the target product. The input is JSON data, and the output is a list of parsed product information.

[0204] Step 3:

[0205] The server uses an AI model to generate a 3D model based on the extracted product information, applying a deep learning algorithm using a machine learning framework such as TensorFlow. The input is a list of product information, and the output is the generated 3D model.

[0206] Step 4:

[0207] The server converts the generated 3D model into GLTF format and sends it to the client device. Specifically, it encodes the 3D model into a standard file format (GLTF) and sends it as an HTTP response. The input is the 3D model, and the output is a GLTF file.

[0208] Step 5:

[0209] The device loads the GLTF format 3D model received from the server. Specifically, it receives the HTTP response, parses the GLTF file, and reads the data. The input is the GLTF file, and the output is the read 3D model data.

[0210] Step 6:

[0211] The device initializes the AR engine (ARCore or ARKit) and overlays the loaded 3D model in real space. When the user points the camera at the product, the product's location and detailed information are displayed. Specifically, the AR engine is started and the 3D model is overlaid on the device's camera image. The input is the loaded 3D model data, and the output is the user's visual interface.

[0212] Step 7:

[0213] Users use their smartphones to explore products in the store and check the product's location and detailed information displayed in AR. Specifically, they move the smartphone's camera to compare the real world with the 3D model to find the product. The input is a video of the real world and a 3D model display, and the output is visual product location information and detailed information.

[0214] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0215] This invention relates to a system that analyzes information obtained from an API, generates 3D models using AI, and combines it with an emotion engine that recognizes user emotions. This system consists of a server and a client terminal. The server obtains and analyzes API data, generates 3D models, recognizes emotions, and transmits the models. The client terminal receives the generated 3D models and displays them in AR according to the user's emotions.

[0216] System configuration

[0217] 1. Server

[0218] Retrieving API data: The server requests the required data from a specified API. For example, to retrieve building information for a city, the server sends an HTTP request to the API endpoint and retrieves the data in JSON format.

[0219] Data analysis: The server analyzes the acquired JSON data and extracts the information necessary for generating a 3D model, such as the location, size, and shape of the building.

[0220] 3D model generation: The server uses the analyzed data to generate a 3D model using an AI model, which uses a deep learning algorithm.

[0221] Model conversion and serving: The generated 3D model is converted into a standard file format (e.g., GLTF format) and sent to the client terminal as an HTTP response.

[0222] Operation of the emotion engine: The server is equipped with an emotion engine that recognizes emotions from the user's image and voice data, and analyzes the acquired emotion data.

[0223] 2. Client Terminal

[0224] Receiving and displaying the model: The client device receives the 3D model sent from the server and uses the AR engine to overlay the model in real space.

[0225] Initializing the AR engine: The client device initializes a dedicated AR engine for AR display, for example, using a framework such as ARKit or ARCore.

[0226] Display change based on emotion recognition: The client device dynamically changes the content of the displayed 3D model based on the emotion data obtained from the emotion engine.

[0227] Program processing

[0228] Server-side processing

[0229] The server first sends a request to an API endpoint to retrieve the required data.

[0230] The acquired data is analyzed to extract necessary information (e.g., position, size, shape, etc.).

[0231] The server generates 3D objects using an AI model that uses deep learning based on the analysis data.

[0232] The generated 3D model is converted into a standard format such as GLTF and sent to the client terminal as an HTTP response.

[0233] The server receives the user's emotional data and analyzes it using the emotion engine. The emotional data is based on the user's image and voice.

[0234] Processing on the client terminal side

[0235] The client terminal acquires the 3D model data received from the server.

[0236] The device initializes the AR engine and loads the received 3D model data.

[0237] The device dynamically changes the content of the 3D model displayed based on the user's emotion recognition results. As emotions change, corresponding objects and effects are displayed.

[0238] The client device adds 3D models to the AR scene and overlays them in real space, allowing users to experience the visual experience of digital objects superimposed on real-world scenery.

[0239] Specific examples

[0240] For example, if a user wants to view city building data in AR:

[0241] 1. Acquiring API data

[0242] The server retrieves JSON-formatted data from the city building data API, including information on the location, size, and shape of each building.

[0243] 2. Data Analysis

[0244] The server analyzes this data and compiles the necessary information for each building, including its location, size, and shape.

[0245] 3. Generating 3D models

[0246] The server uses the organized information to generate a 3D model using an AI model that uses a deep learning algorithm to reproduce even complex shapes with high accuracy.

[0247] 4. Convert and send the model

[0248] The generated 3D model is converted into GLTF format and sent to the client terminal as an HTTP response.

[0249] 5. Display on the client terminal

[0250] The client device uses an AR engine to overlay the received 3D model onto real space.

[0251] 6. Leveraging Emotional Engines

[0252] The client terminal receives the user's emotional data from the emotion engine and dynamically changes the display content, for example, by changing the color or effects of the building if the user is excited.

[0253] As such, this invention is a technology that efficiently generates 3D models from API data and incorporates advanced emotion recognition functionality to enhance the user experience, thereby reducing development costs and enabling services to be provided quickly and efficiently.

[0254] The processing flow will be explained below.

[0255] Step 1:

[0256] The server sends an HTTP request to a specified API endpoint to retrieve the required data, which may include authentication information and filtering criteria as needed. For example, it retrieves data about the location, size, and shape of buildings from a city building data API.

[0257] Step 2:

[0258] The server analyzes the received JSON data. Specifically, it extracts necessary information from the response data, such as the building's location, size, shape, and color. The extracted data is then organized into an appropriate internal data structure as a preprocessing step for generating a 3D model.

[0259] Step 3:

[0260] The server uses an AI model to generate a 3D model based on the organized data. First, it preprocesses the data using a deep learning algorithm and uses it as input for the AI ​​model. The model predicts the 3D shape of the building and generates it in digital form.

[0261] Step 4:

[0262] The server converts the generated 3D model into a standard file format, such as GLTF or OBJ, which makes it easier for the client device to receive it. After the conversion process is complete, the server sends the file to the client device as an HTTP response.

[0263] Step 5:

[0264] The server receives the user's image and voice data and recognizes emotions using an emotion engine, which analyzes this data and identifies the user's emotional state (e.g., joy, sadness, surprise, etc.).

[0265] Step 6:

[0266] The client device receives the 3D model data sent from the server. The received data is temporarily stored in the device's memory. The device then initializes the AR engine and loads the received 3D model data into the AR scene.

[0267] Step 7:

[0268] The client device dynamically changes the content of the displayed 3D model based on the user's emotional data obtained from the emotion engine. For example, if the user is excited, an effect that emphasizes the color of the building is added.

[0269] Step 8:

[0270] The client device overlays the loaded 3D model onto the real world. When the user looks at the real world through the device, the 3D model of the building is displayed appropriately positioned based on the location information.

[0271] Step 9:

[0272] Through AR devices, users can view 3D models that dynamically change depending on their emotional state, enhancing the fusion of reality and digital and enriching the user experience.

[0273] Example 2

[0274] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0275] Conventional systems were able to obtain information from existing APIs and generate 3D models, but they did not support dynamic model display according to the user's emotions. Therefore, in order to improve the user experience, there was a need for technology that not only displays 3D models generated based on obtained information in real space, but also dynamically changes the display content according to the user's emotions.

[0276] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for acquiring information from an existing API, means for analyzing the acquired information, means for generating a 3D model based on the analyzed information, means for transmitting the generated 3D model to the client terminal, means for analyzing the user's emotions, and means for dynamically changing the display content of the 3D model based on the user's emotions. This makes it possible to dynamically change the display content of the 3D model in accordance with the user's emotions.

[0277] "Existing API" refers to a publicly available application program interface, a set of protocols and tools for accessing a particular service or data.

[0278] "Means of obtaining information" refers to the ability to execute a process that uses an existing API to request data and receive it as a response.

[0279] "Means for analyzing information" refers to the function for structuring acquired data and executing a process to extract necessary information.

[0280] "Means for generating a 3D model" refers to a function that executes a process to create a three-dimensional digital object based on the analyzed information.

[0281] "Means for sending to the client terminal" refers to the function for sending the 3D model data generated on the server side to the client side via the network.

[0282] "Client terminal" refers to a device that processes data received from a server and interacts with a user.

[0283] "Means for displaying 3D models in real space" refers to technology for displaying three-dimensional digital objects overlaid on the physical environment on a client device.

[0284] "Means for analyzing emotions" refers to a function that executes a process to identify the emotional state of a user based on voice and image data collected from the user.

[0285] "Means for dynamically changing display content based on emotions" refers to the function of receiving the results of emotion analysis and changing the display format of the 3D model in real time according to the results.

[0286] This invention relates to a system that acquires information from an API and generates a 3D model using AI. The system consists of a server and a client terminal. The server acquires and analyzes API data, generates a 3D model, recognizes emotions, and transmits the model. The client terminal receives the generated 3D model and displays it in AR according to the user's emotions.

[0287] Server Configuration

[0288] API data retrieval:

[0289] The server requests the necessary data from the specified API. For example, to obtain information about a city's buildings, the server sends an HTTP GET request to the API endpoint and obtains data in JSON format. Specifically, it accesses https: / / api.citydata.com / buildings and obtains information such as the location, size, and shape of the buildings.

[0290] Data analysis:

[0291] The server analyzes the acquired JSON data and extracts the information necessary to generate a 3D model. For example, it uses the Python json library to convert the data into a dictionary format and extracts the location, size, shape, and other information of each building.

[0292] 3D model generation:

[0293] The server uses an AI model to generate a 3D model based on the analyzed data. Deep learning algorithms such as TensorFlow and PyTorch are used here. For example, building shape data is input into the AI ​​model to generate a 3D model.

[0294] Convert and serve the model:

[0295] The generated 3D model is converted to GLTF format and sent to the client device as an HTTP response. This conversion is performed using libraries such as pytree. The converted model data is then sent to the client device.

[0296] Emotion Engine in action:

[0297] The server recognizes emotions based on the image and audio data sent by the user. The emotion recognition engine uses algorithms such as EmotionAI to analyze the emotional data. For example, if the user is laughing, the server analyzes the emotional data and recognizes it as excitement or joy.

[0298] Client terminal configuration

[0299] Receiving and displaying models:

[0300] The client device receives the 3D model data sent from the server and uses an AR engine to overlay the model in real space, specifically using frameworks such as ARKit or ARCore.

[0301] Initialize the AR engine:

[0302] The client device initializes a dedicated AR engine for AR display. For example, on an iOS device, it initializes using ARKit and starts an AR session.

[0303] Emotion-aware display changes:

[0304] The client terminal dynamically changes the content of the displayed 3D model based on the emotional data obtained from the emotion engine. If the user is excited, the color or effects of the building model will be changed.

[0305] Specific examples

[0306] For example, if a user wants to view city building data in AR:

[0307] 1. API data acquisition:

[0308] The server retrieves JSON-formatted data from the city building data API, which includes information on the location, size, and shape of buildings.

[0309] 2. Data Analysis:

[0310] The server analyzes the acquired data and extracts the necessary information (location, size, shape) for each building.

[0311] 3. Generate 3D model:

[0312] The server uses AI models to generate 3D models based on the extracted information, and deep learning algorithms enable high-precision reproduction of even complex shapes.

[0313] 4. Convert and send the model:

[0314] The server converts the generated 3D model into GLTF format and sends it to the client terminal as an HTTP response.

[0315] 5. Display by client terminal:

[0316] The client device uses an AR engine to overlay the received 3D model onto real space.

[0317] 6. Leveraging the Emotion Engine:

[0318] The client terminal acquires the user's emotional data from the emotion engine and dynamically changes the display content, for example, by changing the color or effects of the building if the user is excited.

[0319] An example of a prompt is, "Obtain city building data from the API and generate a 3D model using an AI model based on the analyzed data. Convert the generated 3D model into GLTF format so that the client device can display it using an AR engine. Also, recognize the user's emotions based on their image and voice, and change the display accordingly."

[0320] This system is a technology that improves the user experience by efficiently generating 3D models from API data and incorporating advanced emotion recognition functionality, thereby reducing development costs and enabling the provision of services quickly and efficiently.

[0321] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0322] Step 1:

[0323] Retrieving API data

[0324] The server sends an HTTP GET request to an API endpoint. The input is the API endpoint URL (e.g., https: / / api.citydata.com / buildings). The output is JSON-formatted data containing building information for the city. Specifically, the server sends the request and stores the received response data in storage.

[0325] Step 2:

[0326] Data analysis

[0327] The server parses the acquired JSON data and extracts the necessary information. The input is the JSON data obtained in step 1. The output is the parsed information (e.g., building location, size, and shape data). Specifically, the server uses Python's JSON library to convert the data into a dictionary format and extracts each field.

[0328] Step 3:

[0329] 3D model generation

[0330] The server uses an AI model based on the analysis data to generate a 3D model. The input is the analysis data obtained in step 2. The output is the generated 3D model. Specifically, the server uses a deep learning framework such as TensorFlow or PyTorch to input the analysis data into the AI ​​model and generate the 3D model. For example, it performs the process building_model = ai_model.predict(parsed_data).

[0331] Step 4:

[0332] Converting and serving models

[0333] The server converts the generated 3D model into GLTF format and sends it to the client device. The input is the 3D model generated in step 3. The output is the format-converted GLTF data. Specifically, the server converts the model data using the pytree library and sends the converted GLTF data to the client device as an HTTP response.

[0334] Step 5:

[0335] Emotion Engine Operation

[0336] The server analyzes emotions based on image and audio data sent by the user. The input is the image and audio data sent by the user from their device. The output is the analyzed emotional data. Specifically, the server analyzes the data using an emotion recognition engine and performs the process emotion_result = emotion_engine.analyze(image_data).

[0337] Step 6:

[0338] Receiving and displaying the model

[0339] The device receives 3D model data sent from the server. The input is GLTF data sent from the server. The output is a 3D model loaded on the device. Specifically, the device analyzes the received data and loads it into the AR engine.

[0340] Step 7:

[0341] Initializing the AR engine

[0342] The device initializes the AR engine for AR display. There is no input. The output is an initialized AR session. Specifically, the device uses ARKit, for example, and sets arView = ARView(frame: .zero) and arView.session.run(configuration).

[0343] Step 8:

[0344] Display change based on emotion recognition

[0345] The device dynamically changes the display content of the 3D model based on the emotion data obtained from the server. The input is emotion data. The output is the changed 3D model display. For example, to change the color of a building based on emotion data, set model.materials = [bright_color_material].

[0346] Step 9:

[0347] Adding to an AR scene

[0348] The device adds the 3D model to the AR scene and displays it as an overlay in real space. The input is the 3D model loaded in step 6. The output is the overlayed 3D model. Specifically, the device performs the process arView.scene.add(anchorEntity).

[0349] (Application example 2)

[0350] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0351] In recent years, the demand for augmented reality (AR) technology in real space has been increasing, but existing systems have faced challenges in that it is difficult to change the user experience based on dynamic emotion recognition. In particular, it has been difficult to change the displayed content in real time according to the user's emotions. Furthermore, existing systems often have difficulty efficiently analyzing data obtained from APIs and generating 3D models, which reduces usability.

[0352] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for acquiring information from an existing API, means for analyzing the acquired information, means for generating a 3D model based on the analyzed information, means for transmitting the generated 3D model to a client terminal, means for the client terminal to display the 3D model in real space and dynamically change the content according to the user's emotions, means for the client terminal to recognize the user's emotions and transmit the emotion data to the server, and means for the server to analyze the emotion data and provide the analysis results to the client terminal. This enables efficient analysis of API data and generation of a 3D model, and real-time display changes according to the user's emotions.

[0353] An "existing API" is a defined interface for accessing external data and functions.

[0354] A "means of obtaining information" is a function for sending a request to a specified API endpoint and receiving data.

[0355] "Means for analyzing information" refers to a function for processing acquired data and extracting necessary information.

[0356] "Means for generating a 3D model" refers to a function for creating a three-dimensional object based on the extracted data.

[0357] A "client terminal" is a device that processes data received from a server and displays it to the user.

[0358] "Means for displaying in real space" refers to the function for overlaying a 3D model on a physical environment.

[0359] The "means for dynamically changing content according to the user's emotions" is a function for changing the display content in real time based on the user's emotional data.

[0360] "Means for recognizing user emotions" is a function for determining emotions from the user's image and voice data.

[0361] The "means for transmitting emotion data to the server" is a function for transferring the user's emotion information to the server.

[0362] "Means for analyzing emotional data" refers to a function for processing and analyzing emotional data received on the server side.

[0363] The "means for providing the analysis results to the client terminal" is a function for transmitting the analysis results of emotion data to the client terminal.

[0364] An "AR engine" is a software component that realizes augmented reality technology.

[0365] "Means for dynamic change" is a function for changing the display content in real time.

[0366] The system for implementing this invention consists of a server and a client terminal. The server acquires information from an existing API, analyzes the acquired information, generates a 3D model based on the analyzed information, and sends it to the client terminal. The server also has an emotion engine that recognizes the user's emotions, analyzes the user's emotion data, and provides the results to the client terminal.

[0367] The client device has the ability to display the 3D model sent from the server in real space and dynamically change the content according to the user's emotions. Furthermore, the client device recognizes the user's emotions and sends that emotional data to the server. These functions allow the display content to change in real time according to the user's emotions, enabling an interactive AR experience.

[0368] Hardware and software used

[0369] Server: A typical server computer

[0370] Server-side software: Python, Flask (for building API servers), TensorFlow or PyTorch (deep learning libraries)

[0371] 3D model generation tools: Blender (3D modeling software), Three.js (JavaScript(registered trademark) 3D library)

[0372] Emotion recognition system: OpenCV (computer vision library), Dlib (machine learning library)

[0373] Client terminals: smart glasses and head-mounted displays

[0374] Client-side software: Unity (game development engine), ARCore or ARKit (AR development framework)

[0375] Server-side program processing

[0376] The server first sends an HTTP request to the specified API endpoint to obtain information. It then analyzes the obtained JSON-formatted data and extracts necessary information (such as the product's size, shape, and location). It then uses a deep learning algorithm to generate a 3D model based on the extracted information, converts the generated 3D model to GLTF format, and sends it to the client device. The server then recognizes emotions using the user's image and voice data sent from the client device and provides the analysis results to the client device.

[0377] Client terminal program processing

[0378] The client device receives 3D model data from the server, initializes the AR engine, and displays the 3D model in real space. It uses a camera and microphone to recognize emotions and collects the user's emotional data, which it then sends to the server. It also dynamically changes the displayed 3D model and effects based on the acquired emotional data and the analysis results from the server. For example, when a user is looking at a specific product, it displays the product's 3D model, price, and promotion information.

[0379] Specific examples

[0380] For example, imagine a situation where a user is using smart glasses to look at products in a physical store. The server retrieves product data from the product information API, analyzes it, and generates a 3D model. The generated 3D model is displayed through the smart glasses' AR engine. If the user shows interest in a particular product, promotional information and detailed product information for that product are displayed. Furthermore, if the user expresses surprise or delight, the displayed effects change depending on the user's emotion.

[0381] Prompt Sentence Examples

[0382] Create a Unity program that recognizes the user's emotions based on their image data, and changes the effects of the 3D model displayed if the user is excited.

[0383] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0384] Step 1:

[0385] The server sends a request to the API endpoint. The server sends an HTTP request to the specified API endpoint and obtains data in JSON format. It receives API endpoint information as input and obtains JSON format data as output.

[0386] Step 2:

[0387] The server analyzes the acquired JSON data. The server parses the acquired JSON data and extracts necessary information (such as product size, shape, and location information). It receives the acquired JSON data as input and outputs product attribute information as the analysis result.

[0388] Step 3:

[0389] The server generates a 3D model using a deep learning algorithm based on the analyzed information. This uses the generative AI model we defined. It receives product attribute information as input and obtains the generated 3D model data as output.

[0390] Step 4:

[0391] The server sends the generated 3D model to the client terminal. The server converts the generated 3D model into GLTF format and sends it to the client terminal as an HTTP response. The server receives the generated 3D model data as input and obtains a GLTF file to send to the client terminal as output.

[0392] Step 5:

[0393] The client device receives the 3D model data. The device receives the GLTF file sent from the server and prepares it for use in the next step. It receives the GLTF file received from the server as input and obtains the received 3D model data as output.

[0394] Step 6:

[0395] The client device initializes the AR engine. The device reads the received 3D model data and starts the AR engine to prepare for overlaying the model in real space. The device receives the received 3D model data and AR engine initialization information as input, and obtains the initialized AR engine as output.

[0396] Step 7:

[0397] The client device displays the 3D model in real space. The device uses the AR engine to overlay the received 3D model in real space. The device receives the initialized AR engine and the received 3D model data as input, and obtains the 3D model overlaid in real space as output.

[0398] Step 8:

[0399] The client device recognizes the user's emotions. The device uses a camera and microphone to collect the user's image and voice data, which it then inputs into an emotion recognition algorithm (emotion engine). The device receives the user's image and voice data as input, and obtains recognized emotion data as output.

[0400] Step 9:

[0401] The client device sends emotional data to the server. The device then sends the recognized emotional data to the server, and analysis begins on the server side. The recognized emotional data is received as input, and the emotional data sent to the server is obtained as output.

[0402] Step 10:

[0403] The server analyzes the emotion data and provides the results to the client terminal. The server analyzes the received emotion data and sends the analysis results to the client terminal. The server receives the recognized emotion data as input and sends the analysis results to the terminal as output.

[0404] Step 11:

[0405] The client terminal dynamically changes the displayed content. The client terminal dynamically changes the 3D model and effects displayed based on the analysis results from the server. The analysis results from the server are received as input, and the changed display content is obtained as output.

[0406] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0407] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0408] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0409] [Second embodiment]

[0410] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0411] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0412] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0413] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0414] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0415] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0416] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0417] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0418] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0419] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0420] In the smart glasses 214, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0421] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0422] This invention relates to a system that analyzes information obtained from an API, generates a 3D model using AI, and displays it in AR on a client device. This system is primarily composed of a server and a client device, where the server obtains and analyzes API data and generates a 3D model, and the client device receives the generated 3D model and displays it in AR.

[0423] System configuration

[0424] 1. Server

[0425] Retrieving API data: The server requests the required data from the specified API. For example, to retrieve building information for a city, the server sends an HTTP request to the API endpoint and retrieves the data in JSON format.

[0426] Data analysis: The server analyzes the acquired JSON data and extracts the information necessary to generate a 3D model, such as the location, size, and shape of the building.

[0427] 3D model generation: The server uses the AI ​​model to generate a 3D model based on the analyzed data. Here, a deep learning algorithm is used.

[0428] Model conversion and provision: The generated 3D model is converted into a standard file format (e.g., GLTF format) and sent to the client terminal.

[0429] 2. Client Terminal

[0430] Receiving and displaying the model: The client device receives the 3D model sent from the server and uses the AR engine to overlay the model in real space.

[0431] Initializing the AR engine: The client device initializes a dedicated AR engine for AR display, for example, using a framework such as ARKit or ARCore.

[0432] Program processing

[0433] Server-side processing

[0434] The server first sends a request to an API endpoint to retrieve the required data.

[0435] The acquired data is analyzed to extract necessary information (e.g., position, size, shape, etc.).

[0436] Based on the extracted data, a 3D model is generated using an AI model, which is based on deep learning algorithms and has advanced shape recognition and generation capabilities.

[0437] The generated 3D model is converted into a standard format such as GLTF and sent to the client terminal as an HTTP response.

[0438] Processing on the client terminal side

[0439] The client terminal receives the 3D model from the server.

[0440] The device initializes the AR engine and loads the received 3D model data.

[0441] The loaded 3D model is overlaid in real space, providing visual feedback to the user.

[0442] Specific examples

[0443] Take the example of AR display of city building data.

[0444] 1. Acquiring API data

[0445] The server retrieves JSON-formatted data from the urban architecture data API, including information on the location, size, and shape of each building.

[0446] 2. Data Analysis

[0447] The server analyzes this data and extracts the location, size, and shape of each building.

[0448] 3. Generating 3D models

[0449] The server uses the extracted information to generate a 3D model of the building using an AI model that uses deep learning technology to reproduce even complex shapes with high accuracy.

[0450] 4. Convert and send the model

[0451] The generated 3D model is converted into GLTF format and sent to the client terminal as an HTTP response.

[0452] 5. Display on the client terminal

[0453] The client device uses an AR engine to overlay the received 3D model onto the real world, allowing users to see the digital building model superimposed on the real cityscape.

[0454] In this way, the present invention effectively utilizes existing API data to efficiently generate advanced 3D models and integrate them into real space, significantly reducing the cost and time required for developing 3D content and enabling rapid service provision.

[0455] The processing flow will be explained below.

[0456] Step 1:

[0457] The server sends an HTTP request to the specified API endpoint to retrieve the required data. This request may include authentication information and filtering conditions as needed. When the API returns a response to the request, the server receives the response data in JSON format.

[0458] Step 2:

[0459] The server analyzes the received JSON data. Specifically, it extracts necessary information from the response data, such as the building's location, size, shape, and color. The extracted data is then organized into an appropriate internal data structure as a preprocessing step for generating a 3D model.

[0460] Step 3:

[0461] The server uses an AI model to generate a 3D model based on the organized data. First, it uses a deep learning algorithm to input the data and run a process to generate a 3D model. At this time, the AI ​​model is pre-trained within the server.

[0462] Step 4:

[0463] The server converts the generated 3D model into a standard file format, such as GLTF or OBJ, which makes it easier for the client device to receive it. After the conversion process is complete, the file is sent to the client device as an HTTP response.

[0464] Step 5:

[0465] The device receives the 3D model data sent from the server. The received data is stored on the device and prepared for AR display. The device then sends this data to the AR engine and loads the 3D model.

[0466] Step 6:

[0467] The device initializes its AR engine, which includes calibrating the camera and scanning the environment, preparing it to accurately display 3D models in real space.

[0468] Step 7:

[0469] The device overlays the loaded 3D model onto the real world. This includes rendering to ensure the 3D model appears in the correct position when the user looks at the real world through the AR device. Once rendering is complete, the user can enjoy a visual experience in which the digital 3D model blends into the real world.

[0470] Through this series of steps, users can easily experience the fusion of real space and digital content.

[0471] Example 1

[0472] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0473] Conventional technology has the problem of being difficult to analyze information obtained from existing APIs, generate 3D models efficiently and with high accuracy based on that information, and display them in real space. It has also been pointed out that the accuracy and realism of the generated 3D models are not sufficient. To solve these problems, an advanced analysis and generation system utilizing AI technology is required.

[0474] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0475] In this invention, the server includes means for acquiring data from existing information sources, means for analyzing the acquired data, means for generating a 3D model based on the analyzed data, means for transmitting the generated 3D model to a customer terminal, and means for displaying the 3D model in real space by the customer terminal, thereby enabling the generation of a highly accurate 3D model and its overlay display in real space.

[0476] "Existing sources" refer to external systems or interfaces that provide data that already exists, such as databases or online services accessible through APIs.

[0477] "Data" refers to information such as specific numbers, characters, images, etc. that are the subject of calculation, analysis, and processing.

[0478] "Analyzing" refers to examining acquired data in detail and processing it to understand its structure and meaning.

[0479] A "solid model" refers to a computer-generated object that has three-dimensional (3D) geometric information, such as 3D data of a building or object.

[0480] "Customer terminal" refers to a computer device used by a user, including, for example, a smartphone, tablet, or personal computer.

[0481] "Displaying in real space" refers to using augmented reality technology to display digital data overlaid on the physical real world.

[0482] "Generative AI" refers to AI techniques that have the ability to generate new data based on specified input data, including deep learning models and neural networks.

[0483] An "artificial neural network" is an algorithm that mimics the activity of nerve cells in the human brain, and refers to a technology that learns from large amounts of data to recognize patterns and generate data.

[0484] This invention relates to a system that acquires data from existing information sources, analyzes the data, generates a highly accurate 3D model using AI technology, transmits the model to a customer's terminal, and displays it in real space. This system is mainly composed of a server and a customer's terminal.

[0485] Server program processing description

[0486] 1. Data Acquisition:

[0487] The server requests the required data from an existing information source (such as an API). It sends an HTTP request to an API endpoint and retrieves the data in JSON format. For example, to retrieve building information for a city, it uses a URL like "https: / / api.example.com / buildings / tokyo".

[0488] 2. Data Analysis:

[0489] The server parses the received JSON data and extracts information such as the building's location, size, shape, etc. The extracted data has the following structure:

[0490] Building 1: Latitude 35.6895, Longitude 139.6917, Height 150, Shape Rectangle

[0491] Building 2: Latitude 35.6581, Longitude 139.7516, Height 200, Shape Circle

[0492] 3. Generate 3D model:

[0493] The server uses the extracted data to provide prompts to a generative artificial intelligence (AI) model, which is trained using deep learning frameworks such as TensorFlow and PyTorch.

[0494] For example, enter the following prompt:

[0495] "latitude": 35.6895, "longitude": 139.6917, "height": 150, "shape": "rectangle"

[0496] The server uses an AI model to generate a 3D model from this data.

[0497] 4. Convert and send the model:

[0498] Convert the generated 3D model into the standard GLTF format. For example, save it as "model1.gltf".

[0499] The server sends this model data to the client terminal as an HTTP response.

[0500] Explanation of the client terminal program processing

[0501] 1. Receiving the model:

[0502] The client device receives the GLTF format 3D model sent from the server and saves it as temporary data, for example, "model1.gltf."

[0503] 2. Initialize the AR engine:

[0504] The customer device will initialize a dedicated AR engine (such as Apple's ARKit or Google's ARCore), and at this time, the settings will be configured to allow use of the camera and location information.

[0505] 3. Viewing the 3D model:

[0506] The customer's device loads the received GLTF model into its AR engine and displays it as an overlay in real space. A 3D model of the building is displayed in the direction the user points the camera.

[0507] Examples:

[0508] Taking the example of AR display of city building data, the server retrieves JSON-formatted building information data from "https: / / api.example.com / buildings / tokyo," extracts the location, size, and shape of each building, and supplies this to the AI ​​model. Using TensorFlow, the AI ​​model generates a highly accurate 3D model in response to a prompt, such as "latitude: 35.6895, longitude: 139.6917, height: 150, shape: rectangle," and sends it to the client device as "model1.gltf" in GLTF format. The client device receives this and uses ARKit to overlay the 3D model on the real world. This allows users to enjoy an AR experience where digital building models are overlaid on the Tokyo cityscape.

[0509] This system makes it possible to generate highly accurate 3D models and overlay them in real space.

[0510] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0511] Step 1:

[0512] The server retrieves data from an existing information source (API). Specifically, it sends an HTTP GET request to an API endpoint and receives JSON-formatted data as a response. For example, it retrieves city building information from "https: / / api.example.com / buildings / tokyo."

[0513] Input: API endpoint

[0514] Output: API response data in JSON format

[0515] Specific operation: Send a GET request to "https: / / api.example.com / buildings / tokyo" and receive JSON data containing information such as the building's location, size, and shape.

[0516] Step 2:

[0517] The server analyzes the received JSON data using a JSON parser to extract necessary information such as the location, size, and shape of each building.

[0518] Input: API response data in JSON format

[0519] Output: Information on the location, size, and shape of the extracted buildings

[0520] Specific behavior: Parse the received JSON data and convert it into the following structure:

[0521] Building 1: Latitude 35.6895, Longitude 139.6917, Height 150, Shape Rectangle

[0522] Building 2: Latitude 35.6581, Longitude 139.7516, Height 200, Shape Circle

[0523] Step 3:

[0524] The server inputs prompts into the AI ​​model based on the extracted data to generate a 3D model. Specifically, it inputs prompts into the AI ​​model (a deep learning model trained with TensorFlow or PyTorch) to generate 3D model data.

[0525] Input: Information on the location, size, and shape of the extracted buildings

[0526] Output: Generated 3D model

[0527] Specific behavior:

[0528] Prompt: "latitude": 35.6895, "longitude": 139.6917, "height": 150, "shape": "rectangle"

[0529] is input into the AI ​​model to generate a three-dimensional model.

[0530] Step 4:

[0531] The server converts the generated 3D model into the standard GLTF format and sends it to the client terminal. Specifically, it converts it into the GLTF format using a model conversion tool and sends it to the client terminal as an HTTP response.

[0532] Input: Generated 3D model

[0533] Output: 3D model data in GLTF format

[0534] Specific operation: The generated 3D model is saved as "model1.gltf" and sent to the client terminal.

[0535] Step 5:

[0536] The client device receives the GLTF format 3D model sent from the server and initializes the AR engine, specifically, a dedicated AR engine such as ARKit or ARCore, and makes the camera and location information available.

[0537] Input: 3D model data in GLTF format

[0538] Output: Initialized AR Engine

[0539] Specific operation: The client device receives "model1.gltf", initializes ARKit, and obtains camera access permission.

[0540] Step 6:

[0541] The client device loads the received GLTF model into the AR engine and displays it as an overlay in real space. Specifically, the GLTF model is loaded into the AR engine and displayed in the direction the user points the camera.

[0542] Input: Initialized AR engine, 3D model data in GLTF format

[0543] Output: 3D model displayed in real space

[0544] Specific operation: The client device loads "model1.gltf" into ARKit and displays a 3D model of the building in real space in the direction the user is pointing the camera.

[0545] This allows users to visually experience a digital three-dimensional model superimposed on a real-world scene.

[0546] (Application example 1)

[0547] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0548] In existing brick-and-mortar stores, customers often have to spend a lot of time finding specific products, and detailed product information is difficult to obtain. Stores also face the challenge of having few ways to communicate product layout changes and inventory status to customers in real time. To solve these problems, there is a need for technology that can easily obtain product location information within a store and present it to users in a visually easy-to-understand manner.

[0549] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0550] In this invention, the server includes means for acquiring information from an existing API, means for analyzing the acquired information, means for generating a 3D model based on the analyzed information, means for transmitting the generated 3D model to a client terminal, and means for the client terminal to provide product location information and detailed information to a user using the 3D model displayed in real space. This enables customers to quickly and accurately find specific products in a physical store and obtain detailed product information in real time.

[0551] "Means of obtaining information from an API" refers to a mechanism for requesting and receiving necessary information from external data providers via the Internet.

[0552] "Means for analyzing information" refers to a program or method for interpreting acquired data according to certain rules and extracting or converting necessary items.

[0553] "Means for generating a 3D model" refers to a technology or process for creating a digital model with a three-dimensional shape based on analyzed information.

[0554] "Means for transmitting the generated 3D model to the client terminal" refers to a mechanism for distributing the 3D model created on the server side to the client terminal via the Internet.

[0555] "Means for displaying a 3D model in real space using a client terminal" refers to a technology in which the client terminal uses a camera and display to visually display a 3D model superimposed on the real environment.

[0556] "A means by which a client terminal provides users with product location information and detailed information using a 3D model displayed in real space" is a mechanism that allows users to visually obtain product location and detailed information via a 3D model displayed in AR.

[0557] "Means for generating 3D models using AI" refers to a method that utilizes artificial intelligence technology to automatically create 3D models from analyzed information.

[0558] "Means using deep learning algorithms" refers to the process of using deep learning technology, a type of artificial intelligence, to analyze complex data patterns and generate 3D models.

[0559] This invention is a system that acquires information from an existing API, generates a 3D model based on that information, and displays it in real space on a client terminal, providing users with product location information and detailed information. Here we will explain the specific system configuration and how to implement it.

[0560] System configuration

[0561] 1. Server

[0562] The server sends a request to an external data provision service (API) via the Internet to obtain the required data.

[0563] The acquired data is analyzed to extract necessary information such as the product's position, shape, and size.

[0564] Based on the extracted information, a 3D model is generated using AI technology, using machine learning frameworks such as TensorFlow.

[0565] The generated 3D model is converted into GLTF format and sent to the client device.

[0566] 2. Client Terminal

[0567] The client terminal receives the 3D model sent from the server.

[0568] Using an AR engine (e.g., ARCore or ARKit), the received 3D model is overlaid onto the real world.

[0569] When a user holds the camera over a product, the product's location and detailed information are visually displayed through AR.

[0570] Specific processing details

[0571] 1. Data acquisition by the server

[0572] The server sends an HTTP request to the specified API to obtain JSON formatted data. For example, it uses an API to obtain the location and detailed information of a store's products.

[0573] 2. Data Analysis and 3D Model Generation

[0574] The acquired data is analyzed to extract the position, size, and shape of each product.

[0575] Based on the extracted information, a 3D model is generated using a machine learning model, using a framework such as TensorFlow.

[0576] 3. Convert and send 3D models

[0577] The generated 3D model is converted into GLTF format and sent to the client terminal as an HTTP response.

[0578] 4. AR display on client device

[0579] The client device loads the 3D model received from the server and initializes the AR engine.

[0580] A 3D model is overlaid onto the real space where the camera is pointed, providing product location information and detailed information.

[0581] Specific examples

[0582] As a specific scenario, the following prompt sentence is used:

[0583] "This application displays the location information of products in the store in real time, allowing you to easily search and view the location of specific products. It also uses AR to display detailed information about products (price, reviews, stock, etc.). ARCore is used for AR display, so accurate location display is possible."

[0584] This allows users to efficiently search for products in the store using their smartphones and easily obtain detailed information.

[0585] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0586] Step 1:

[0587] The server sends an HTTP request to an existing API via the Internet to obtain the required data. Specifically, it receives JSON-formatted data that includes the location, size, shape, etc. of in-store products. The input is the API request, and the output is the JSON data returned as the API response.

[0588] Step 2:

[0589] The server analyzes the acquired JSON data and extracts information about the product's position, size, and shape. Specifically, it parses the JSON data and extracts the parameters of the target product. The input is JSON data, and the output is a list of parsed product information.

[0590] Step 3:

[0591] The server uses an AI model to generate a 3D model based on the extracted product information, applying a deep learning algorithm using a machine learning framework such as TensorFlow. The input is a list of product information, and the output is the generated 3D model.

[0592] Step 4:

[0593] The server converts the generated 3D model into GLTF format and sends it to the client device. Specifically, it encodes the 3D model into a standard file format (GLTF) and sends it as an HTTP response. The input is the 3D model, and the output is a GLTF file.

[0594] Step 5:

[0595] The device loads the GLTF format 3D model received from the server. Specifically, it receives the HTTP response, parses the GLTF file, and reads the data. The input is the GLTF file, and the output is the read 3D model data.

[0596] Step 6:

[0597] The device initializes the AR engine (ARCore or ARKit) and overlays the loaded 3D model in real space. When the user points the camera at the product, the product's location and detailed information are displayed. Specifically, the AR engine is started and the 3D model is overlaid on the device's camera image. The input is the loaded 3D model data, and the output is the user's visual interface.

[0598] Step 7:

[0599] Users use their smartphones to explore products in the store and check the product's location and detailed information displayed in AR. Specifically, they move the smartphone's camera to compare the real world with the 3D model to find the product. The input is a video of the real world and a 3D model display, and the output is visual product location information and detailed information.

[0600] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0601] This invention relates to a system that analyzes information obtained from an API, generates 3D models using AI, and combines it with an emotion engine that recognizes user emotions. This system consists of a server and a client terminal. The server obtains and analyzes API data, generates 3D models, recognizes emotions, and transmits the models. The client terminal receives the generated 3D models and displays them in AR according to the user's emotions.

[0602] System configuration

[0603] 1. Server

[0604] Retrieving API data: The server requests the required data from a specified API. For example, to retrieve building information for a city, the server sends an HTTP request to the API endpoint and retrieves the data in JSON format.

[0605] Data analysis: The server analyzes the acquired JSON data and extracts the information necessary for generating a 3D model, such as the location, size, and shape of the building.

[0606] 3D model generation: The server uses the analyzed data to generate a 3D model using an AI model, which uses a deep learning algorithm.

[0607] Model conversion and serving: The generated 3D model is converted into a standard file format (e.g., GLTF format) and sent to the client terminal as an HTTP response.

[0608] Operation of the emotion engine: The server is equipped with an emotion engine that recognizes emotions from the user's image and voice data, and analyzes the acquired emotion data.

[0609] 2. Client Terminal

[0610] Receiving and displaying the model: The client device receives the 3D model sent from the server and uses the AR engine to overlay the model in real space.

[0611] Initializing the AR engine: The client device initializes a dedicated AR engine for AR display, for example, using a framework such as ARKit or ARCore.

[0612] Display change based on emotion recognition: The client device dynamically changes the content of the displayed 3D model based on the emotion data obtained from the emotion engine.

[0613] Program processing

[0614] Server-side processing

[0615] The server first sends a request to an API endpoint to retrieve the required data.

[0616] The acquired data is analyzed to extract necessary information (e.g., position, size, shape, etc.).

[0617] The server generates 3D objects using an AI model that uses deep learning based on the analysis data.

[0618] The generated 3D model is converted into a standard format such as GLTF and sent to the client terminal as an HTTP response.

[0619] The server receives the user's emotional data and analyzes it using the emotion engine. The emotional data is based on the user's image and voice.

[0620] Processing on the client terminal side

[0621] The client terminal acquires the 3D model data received from the server.

[0622] The device initializes the AR engine and loads the received 3D model data.

[0623] The device dynamically changes the content of the 3D model displayed based on the user's emotion recognition results. As emotions change, corresponding objects and effects are displayed.

[0624] The client device adds 3D models to the AR scene and overlays them in real space, allowing users to experience the visual experience of digital objects superimposed on real-world scenery.

[0625] Specific examples

[0626] For example, if a user wants to view city building data in AR:

[0627] 1. Acquiring API data

[0628] The server retrieves JSON-formatted data from the city building data API, including information on the location, size, and shape of each building.

[0629] 2. Data Analysis

[0630] The server analyzes this data and compiles the necessary information for each building, including its location, size, and shape.

[0631] 3. Generating 3D models

[0632] The server uses the organized information to generate a 3D model using an AI model that uses a deep learning algorithm to reproduce even complex shapes with high accuracy.

[0633] 4. Convert and send the model

[0634] The generated 3D model is converted into GLTF format and sent to the client terminal as an HTTP response.

[0635] 5. Display on the client terminal

[0636] The client device uses an AR engine to overlay the received 3D model onto real space.

[0637] 6. Leveraging Emotional Engines

[0638] The client terminal receives the user's emotional data from the emotion engine and dynamically changes the display content, for example, by changing the color or effects of the building if the user is excited.

[0639] As such, this invention is a technology that efficiently generates 3D models from API data and incorporates advanced emotion recognition functionality to enhance the user experience, thereby reducing development costs and enabling services to be provided quickly and efficiently.

[0640] The processing flow will be explained below.

[0641] Step 1:

[0642] The server sends an HTTP request to a specified API endpoint to retrieve the required data, which may include authentication information and filtering criteria as needed. For example, it retrieves data about the location, size, and shape of buildings from a city building data API.

[0643] Step 2:

[0644] The server analyzes the received JSON data. Specifically, it extracts necessary information from the response data, such as the building's location, size, shape, and color. The extracted data is then organized into an appropriate internal data structure as a preprocessing step for generating a 3D model.

[0645] Step 3:

[0646] The server uses an AI model to generate a 3D model based on the organized data. First, it preprocesses the data using a deep learning algorithm and uses it as input for the AI ​​model. The model predicts the 3D shape of the building and generates it in digital form.

[0647] Step 4:

[0648] The server converts the generated 3D model into a standard file format, such as GLTF or OBJ, which makes it easier for the client device to receive it. After the conversion process is complete, the server sends the file to the client device as an HTTP response.

[0649] Step 5:

[0650] The server receives the user's image and voice data and recognizes emotions using an emotion engine, which analyzes this data and identifies the user's emotional state (e.g., joy, sadness, surprise, etc.).

[0651] Step 6:

[0652] The client device receives the 3D model data sent from the server. The received data is temporarily stored in the device's memory. The device then initializes the AR engine and loads the received 3D model data into the AR scene.

[0653] Step 7:

[0654] The client device dynamically changes the content of the displayed 3D model based on the user's emotional data obtained from the emotion engine. For example, if the user is excited, an effect that emphasizes the color of the building is added.

[0655] Step 8:

[0656] The client device overlays the loaded 3D model onto the real world. When the user looks at the real world through the device, the 3D model of the building is displayed appropriately positioned based on the location information.

[0657] Step 9:

[0658] Through AR devices, users can view 3D models that dynamically change depending on their emotional state, enhancing the fusion of reality and digital and enriching the user experience.

[0659] Example 2

[0660] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0661] Conventional systems were able to obtain information from existing APIs and generate 3D models, but they did not support dynamic model display according to the user's emotions. Therefore, in order to improve the user experience, there was a need for technology that not only displays 3D models generated based on obtained information in real space, but also dynamically changes the display content according to the user's emotions.

[0662] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for acquiring information from an existing API, means for analyzing the acquired information, means for generating a 3D model based on the analyzed information, means for transmitting the generated 3D model to the client terminal, means for analyzing the user's emotions, and means for dynamically changing the display content of the 3D model based on the user's emotions. This makes it possible to dynamically change the display content of the 3D model in accordance with the user's emotions.

[0663] "Existing API" refers to a publicly available application program interface, a set of protocols and tools for accessing a particular service or data.

[0664] "Means of obtaining information" refers to the ability to execute a process that uses an existing API to request data and receive it as a response.

[0665] "Means for analyzing information" refers to the function for structuring acquired data and executing a process to extract necessary information.

[0666] "Means for generating a 3D model" refers to a function that executes a process to create a three-dimensional digital object based on the analyzed information.

[0667] "Means for sending to the client terminal" refers to the function for sending the 3D model data generated on the server side to the client side via the network.

[0668] "Client terminal" refers to a device that processes data received from a server and interacts with a user.

[0669] "Means for displaying 3D models in real space" refers to technology for displaying three-dimensional digital objects overlaid on the physical environment on a client device.

[0670] "Means for analyzing emotions" refers to a function that executes a process to identify the emotional state of a user based on voice and image data collected from the user.

[0671] "Means for dynamically changing display content based on emotions" refers to the function of receiving the results of emotion analysis and changing the display format of the 3D model in real time according to the results.

[0672] This invention relates to a system that acquires information from an API and generates a 3D model using AI. The system consists of a server and a client terminal. The server acquires and analyzes API data, generates a 3D model, recognizes emotions, and transmits the model. The client terminal receives the generated 3D model and displays it in AR according to the user's emotions.

[0673] Server Configuration

[0674] API data retrieval:

[0675] The server requests the necessary data from the specified API. For example, to obtain information about a city's buildings, the server sends an HTTP GET request to the API endpoint and obtains data in JSON format. Specifically, it accesses https: / / api.citydata.com / buildings and obtains information such as the location, size, and shape of the buildings.

[0676] Data analysis:

[0677] The server analyzes the acquired JSON data and extracts the information necessary to generate a 3D model. For example, it uses the Python json library to convert the data into a dictionary format and extracts the location, size, shape, and other information of each building.

[0678] 3D model generation:

[0679] The server uses an AI model to generate a 3D model based on the analyzed data. Deep learning algorithms such as TensorFlow and PyTorch are used here. For example, building shape data is input into the AI ​​model to generate a 3D model.

[0680] Convert and serve the model:

[0681] The generated 3D model is converted to GLTF format and sent to the client device as an HTTP response. This conversion is performed using libraries such as pytree. The converted model data is then sent to the client device.

[0682] Emotion Engine in action:

[0683] The server recognizes emotions based on the image and audio data sent by the user. The emotion recognition engine uses algorithms such as EmotionAI to analyze the emotional data. For example, if the user is laughing, the server analyzes the emotional data and recognizes it as excitement or joy.

[0684] Client terminal configuration

[0685] Receiving and displaying models:

[0686] The client device receives the 3D model data sent from the server and uses an AR engine to overlay the model in real space, specifically using frameworks such as ARKit or ARCore.

[0687] Initialize the AR engine:

[0688] The client device initializes a dedicated AR engine for AR display. For example, on an iOS device, it initializes using ARKit and starts an AR session.

[0689] Emotion-aware display changes:

[0690] The client terminal dynamically changes the content of the displayed 3D model based on the emotional data obtained from the emotion engine. If the user is excited, the color or effects of the building model will be changed.

[0691] Specific examples

[0692] For example, if a user wants to view city building data in AR:

[0693] 1. API data acquisition:

[0694] The server retrieves JSON-formatted data from the city building data API, which includes information on the location, size, and shape of buildings.

[0695] 2. Data Analysis:

[0696] The server analyzes the acquired data and extracts the necessary information (location, size, shape) for each building.

[0697] 3. Generate 3D model:

[0698] The server uses AI models to generate 3D models based on the extracted information, and deep learning algorithms enable high-precision reproduction of even complex shapes.

[0699] 4. Convert and send the model:

[0700] The server converts the generated 3D model into GLTF format and sends it to the client terminal as an HTTP response.

[0701] 5. Display by client terminal:

[0702] The client device uses an AR engine to overlay the received 3D model onto real space.

[0703] 6. Leveraging the Emotion Engine:

[0704] The client terminal acquires the user's emotional data from the emotion engine and dynamically changes the display content, for example, by changing the color or effects of the building if the user is excited.

[0705] An example of a prompt is, "Obtain city building data from the API and generate a 3D model using an AI model based on the analyzed data. Convert the generated 3D model into GLTF format so that the client device can display it using an AR engine. Also, recognize the user's emotions based on their image and voice, and change the display accordingly."

[0706] This system is a technology that improves the user experience by efficiently generating 3D models from API data and incorporating advanced emotion recognition functionality, thereby reducing development costs and enabling the provision of services quickly and efficiently.

[0707] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0708] Step 1:

[0709] Retrieving API data

[0710] The server sends an HTTP GET request to an API endpoint. The input is the API endpoint URL (e.g., https: / / api.citydata.com / buildings). The output is JSON-formatted data containing building information for the city. Specifically, the server sends the request and stores the received response data in storage.

[0711] Step 2:

[0712] Data analysis

[0713] The server parses the acquired JSON data and extracts the necessary information. The input is the JSON data obtained in step 1. The output is the parsed information (e.g., building location, size, and shape data). Specifically, the server uses Python's JSON library to convert the data into a dictionary format and extracts each field.

[0714] Step 3:

[0715] 3D model generation

[0716] The server uses an AI model based on the analysis data to generate a 3D model. The input is the analysis data obtained in step 2. The output is the generated 3D model. Specifically, the server uses a deep learning framework such as TensorFlow or PyTorch to input the analysis data into the AI ​​model and generate the 3D model. For example, it performs the process building_model = ai_model.predict(parsed_data).

[0717] Step 4:

[0718] Converting and serving models

[0719] The server converts the generated 3D model into GLTF format and sends it to the client device. The input is the 3D model generated in step 3. The output is the format-converted GLTF data. Specifically, the server converts the model data using the pytree library and sends the converted GLTF data to the client device as an HTTP response.

[0720] Step 5:

[0721] Emotion Engine Operation

[0722] The server analyzes emotions based on image and audio data sent by the user. The input is the image and audio data sent by the user from their device. The output is the analyzed emotional data. Specifically, the server analyzes the data using an emotion recognition engine and performs the process emotion_result = emotion_engine.analyze(image_data).

[0723] Step 6:

[0724] Receiving and displaying the model

[0725] The device receives 3D model data sent from the server. The input is GLTF data sent from the server. The output is a 3D model loaded on the device. Specifically, the device analyzes the received data and loads it into the AR engine.

[0726] Step 7:

[0727] Initializing the AR engine

[0728] The device initializes the AR engine for AR display. There is no input. The output is an initialized AR session. Specifically, the device uses ARKit, for example, and sets arView = ARView(frame: .zero) and arView.session.run(configuration).

[0729] Step 8:

[0730] Display change based on emotion recognition

[0731] The device dynamically changes the display content of the 3D model based on the emotion data obtained from the server. The input is emotion data. The output is the changed 3D model display. For example, to change the color of a building based on emotion data, set model.materials = [bright_color_material].

[0732] Step 9:

[0733] Adding to an AR scene

[0734] The device adds the 3D model to the AR scene and displays it as an overlay in real space. The input is the 3D model loaded in step 6. The output is the overlayed 3D model. Specifically, the device performs the process arView.scene.add(anchorEntity).

[0735] (Application example 2)

[0736] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0737] In recent years, the demand for augmented reality (AR) technology in real space has been increasing, but existing systems have faced challenges in that it is difficult to change the user experience based on dynamic emotion recognition. In particular, it has been difficult to change the displayed content in real time according to the user's emotions. Furthermore, existing systems often have difficulty efficiently analyzing data obtained from APIs and generating 3D models, which reduces usability.

[0738] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for acquiring information from an existing API, means for analyzing the acquired information, means for generating a 3D model based on the analyzed information, means for transmitting the generated 3D model to a client terminal, means for the client terminal to display the 3D model in real space and dynamically change the content according to the user's emotions, means for the client terminal to recognize the user's emotions and transmit the emotion data to the server, and means for the server to analyze the emotion data and provide the analysis results to the client terminal. This enables efficient analysis of API data and generation of a 3D model, and real-time display changes according to the user's emotions.

[0739] An "existing API" is a defined interface for accessing external data and functions.

[0740] A "means of obtaining information" is a function for sending a request to a specified API endpoint and receiving data.

[0741] "Means for analyzing information" refers to a function for processing acquired data and extracting necessary information.

[0742] "Means for generating a 3D model" refers to a function for creating a three-dimensional object based on the extracted data.

[0743] A "client terminal" is a device that processes data received from a server and displays it to the user.

[0744] "Means for displaying in real space" refers to the function for overlaying a 3D model on a physical environment.

[0745] The "means for dynamically changing content according to the user's emotions" is a function for changing the display content in real time based on the user's emotional data.

[0746] "Means for recognizing user emotions" is a function for determining emotions from the user's image and voice data.

[0747] The "means for transmitting emotion data to the server" is a function for transferring the user's emotion information to the server.

[0748] "Means for analyzing emotional data" refers to a function for processing and analyzing emotional data received on the server side.

[0749] The "means for providing the analysis results to the client terminal" is a function for transmitting the analysis results of emotion data to the client terminal.

[0750] An "AR engine" is a software component that realizes augmented reality technology.

[0751] "Means for dynamic change" is a function for changing the display content in real time.

[0752] The system for implementing this invention consists of a server and a client terminal. The server acquires information from an existing API, analyzes the acquired information, generates a 3D model based on the analyzed information, and sends it to the client terminal. The server also has an emotion engine that recognizes the user's emotions, analyzes the user's emotion data, and provides the results to the client terminal.

[0753] The client device has the ability to display the 3D model sent from the server in real space and dynamically change the content according to the user's emotions. Furthermore, the client device recognizes the user's emotions and sends that emotional data to the server. These functions allow the display content to change in real time according to the user's emotions, enabling an interactive AR experience.

[0754] Hardware and software used

[0755] Server: A typical server computer

[0756] Server-side software: Python, Flask (for building API servers), TensorFlow or PyTorch (deep learning libraries)

[0757] 3D model generation tools: Blender (3D modeling software), Three.js (JavaScript 3D library)

[0758] Emotion recognition system: OpenCV (computer vision library), Dlib (machine learning library)

[0759] Client terminals: smart glasses and head-mounted displays

[0760] Client-side software: Unity (game development engine), ARCore or ARKit (AR development framework)

[0761] Server-side program processing

[0762] The server first sends an HTTP request to the specified API endpoint to obtain information. It then analyzes the obtained JSON-formatted data and extracts necessary information (such as the product's size, shape, and location). It then uses a deep learning algorithm to generate a 3D model based on the extracted information, converts the generated 3D model to GLTF format, and sends it to the client device. The server then recognizes emotions using the user's image and voice data sent from the client device and provides the analysis results to the client device.

[0763] Client terminal program processing

[0764] The client device receives 3D model data from the server, initializes the AR engine, and displays the 3D model in real space. It uses a camera and microphone to recognize emotions and collects the user's emotional data, which it then sends to the server. It also dynamically changes the displayed 3D model and effects based on the acquired emotional data and the analysis results from the server. For example, when a user is looking at a specific product, it displays the product's 3D model, price, and promotion information.

[0765] Specific examples

[0766] For example, imagine a situation where a user is using smart glasses to look at products in a physical store. The server retrieves product data from the product information API, analyzes it, and generates a 3D model. The generated 3D model is displayed through the smart glasses' AR engine. If the user shows interest in a particular product, promotional information and detailed product information for that product are displayed. Furthermore, if the user expresses surprise or delight, the displayed effects change depending on the user's emotion.

[0767] Prompt Sentence Examples

[0768] Create a Unity program that recognizes the user's emotions based on their image data, and changes the effects of the 3D model displayed if the user is excited.

[0769] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0770] Step 1:

[0771] The server sends a request to the API endpoint. The server sends an HTTP request to the specified API endpoint and obtains data in JSON format. It receives API endpoint information as input and obtains JSON format data as output.

[0772] Step 2:

[0773] The server analyzes the acquired JSON data. The server parses the acquired JSON data and extracts necessary information (such as product size, shape, and location information). It receives the acquired JSON data as input and outputs product attribute information as the analysis result.

[0774] Step 3:

[0775] The server generates a 3D model using a deep learning algorithm based on the analyzed information. This uses the generative AI model we defined. It receives product attribute information as input and obtains the generated 3D model data as output.

[0776] Step 4:

[0777] The server sends the generated 3D model to the client terminal. The server converts the generated 3D model into GLTF format and sends it to the client terminal as an HTTP response. The server receives the generated 3D model data as input and obtains a GLTF file to send to the client terminal as output.

[0778] Step 5:

[0779] The client device receives the 3D model data. The device receives the GLTF file sent from the server and prepares it for use in the next step. It receives the GLTF file received from the server as input and obtains the received 3D model data as output.

[0780] Step 6:

[0781] The client device initializes the AR engine. The device reads the received 3D model data and starts the AR engine to prepare for overlaying the model in real space. The device receives the received 3D model data and AR engine initialization information as input, and obtains the initialized AR engine as output.

[0782] Step 7:

[0783] The client device displays the 3D model in real space. The device uses the AR engine to overlay the received 3D model in real space. The device receives the initialized AR engine and the received 3D model data as input, and obtains the 3D model overlaid in real space as output.

[0784] Step 8:

[0785] The client device recognizes the user's emotions. The device uses a camera and microphone to collect the user's image and voice data, which it then inputs into an emotion recognition algorithm (emotion engine). The device receives the user's image and voice data as input, and obtains recognized emotion data as output.

[0786] Step 9:

[0787] The client device sends emotional data to the server. The device then sends the recognized emotional data to the server, and analysis begins on the server side. The recognized emotional data is received as input, and the emotional data sent to the server is obtained as output.

[0788] Step 10:

[0789] The server analyzes the emotion data and provides the results to the client terminal. The server analyzes the received emotion data and sends the analysis results to the client terminal. The server receives the recognized emotion data as input and sends the analysis results to the terminal as output.

[0790] Step 11:

[0791] The client terminal dynamically changes the displayed content. The client terminal dynamically changes the 3D model and effects displayed based on the analysis results from the server. The analysis results from the server are received as input, and the changed display content is obtained as output.

[0792] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0793] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0794] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0795] [Third embodiment]

[0796] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0797] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0798] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0799] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0800] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0801] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0802] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0803] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0804] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0805] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0806] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0807] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0808] This invention relates to a system that analyzes information obtained from an API, generates a 3D model using AI, and displays it in AR on a client device. This system is primarily composed of a server and a client device, where the server obtains and analyzes API data and generates a 3D model, and the client device receives the generated 3D model and displays it in AR.

[0809] System configuration

[0810] 1. Server

[0811] Retrieving API data: The server requests the required data from the specified API. For example, to retrieve building information for a city, the server sends an HTTP request to the API endpoint and retrieves the data in JSON format.

[0812] Data analysis: The server analyzes the acquired JSON data and extracts the information necessary to generate a 3D model, such as the location, size, and shape of the building.

[0813] 3D model generation: The server uses the AI ​​model to generate a 3D model based on the analyzed data. Here, a deep learning algorithm is used.

[0814] Model conversion and provision: The generated 3D model is converted into a standard file format (e.g., GLTF format) and sent to the client terminal.

[0815] 2. Client Terminal

[0816] Receiving and displaying the model: The client device receives the 3D model sent from the server and uses the AR engine to overlay the model in real space.

[0817] Initializing the AR engine: The client device initializes a dedicated AR engine for AR display, for example, using a framework such as ARKit or ARCore.

[0818] Program processing

[0819] Server-side processing

[0820] The server first sends a request to an API endpoint to retrieve the required data.

[0821] The acquired data is analyzed to extract necessary information (e.g., position, size, shape, etc.).

[0822] Based on the extracted data, a 3D model is generated using an AI model, which is based on deep learning algorithms and has advanced shape recognition and generation capabilities.

[0823] The generated 3D model is converted into a standard format such as GLTF and sent to the client terminal as an HTTP response.

[0824] Processing on the client terminal side

[0825] The client terminal receives the 3D model from the server.

[0826] The device initializes the AR engine and loads the received 3D model data.

[0827] The loaded 3D model is overlaid in real space, providing visual feedback to the user.

[0828] Specific examples

[0829] Take the example of AR display of city building data.

[0830] 1. Acquiring API data

[0831] The server retrieves JSON-formatted data from the urban architecture data API, including information on the location, size, and shape of each building.

[0832] 2. Data Analysis

[0833] The server analyzes this data and extracts the location, size, and shape of each building.

[0834] 3. Generating 3D models

[0835] The server uses the extracted information to generate a 3D model of the building using an AI model that uses deep learning technology to reproduce even complex shapes with high accuracy.

[0836] 4. Convert and send the model

[0837] The generated 3D model is converted into GLTF format and sent to the client terminal as an HTTP response.

[0838] 5. Display on the client terminal

[0839] The client device uses an AR engine to overlay the received 3D model onto the real world, allowing users to see the digital building model superimposed on the real cityscape.

[0840] In this way, the present invention effectively utilizes existing API data to efficiently generate advanced 3D models and integrate them into real space, significantly reducing the cost and time required for developing 3D content and enabling rapid service provision.

[0841] The processing flow will be explained below.

[0842] Step 1:

[0843] The server sends an HTTP request to the specified API endpoint to retrieve the required data. This request may include authentication information and filtering conditions as needed. When the API returns a response to the request, the server receives the response data in JSON format.

[0844] Step 2:

[0845] The server analyzes the received JSON data. Specifically, it extracts necessary information from the response data, such as the building's location, size, shape, and color. The extracted data is then organized into an appropriate internal data structure as a preprocessing step for generating a 3D model.

[0846] Step 3:

[0847] The server uses an AI model to generate a 3D model based on the organized data. First, it uses a deep learning algorithm to input the data and run a process to generate a 3D model. At this time, the AI ​​model is pre-trained within the server.

[0848] Step 4:

[0849] The server converts the generated 3D model into a standard file format, such as GLTF or OBJ, which makes it easier for the client device to receive it. After the conversion process is complete, the file is sent to the client device as an HTTP response.

[0850] Step 5:

[0851] The device receives the 3D model data sent from the server. The received data is stored on the device and prepared for AR display. The device then sends this data to the AR engine and loads the 3D model.

[0852] Step 6:

[0853] The device initializes its AR engine, which includes calibrating the camera and scanning the environment, preparing it to accurately display 3D models in real space.

[0854] Step 7:

[0855] The device overlays the loaded 3D model onto the real world. This includes rendering to ensure the 3D model appears in the correct position when the user looks at the real world through the AR device. Once rendering is complete, the user can enjoy a visual experience in which the digital 3D model blends into the real world.

[0856] Through this series of steps, users can easily experience the fusion of real space and digital content.

[0857] Example 1

[0858] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0859] Conventional technology has the problem of being difficult to analyze information obtained from existing APIs, generate 3D models efficiently and with high accuracy based on that information, and display them in real space. It has also been pointed out that the accuracy and realism of the generated 3D models are not sufficient. To solve these problems, an advanced analysis and generation system utilizing AI technology is required.

[0860] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0861] In this invention, the server includes means for acquiring data from existing information sources, means for analyzing the acquired data, means for generating a 3D model based on the analyzed data, means for transmitting the generated 3D model to a customer terminal, and means for displaying the 3D model in real space by the customer terminal, thereby enabling the generation of a highly accurate 3D model and its overlay display in real space.

[0862] "Existing sources" refer to external systems or interfaces that provide data that already exists, such as databases or online services accessible through APIs.

[0863] "Data" refers to information such as specific numbers, characters, images, etc. that are the subject of calculation, analysis, and processing.

[0864] "Analyzing" refers to examining acquired data in detail and processing it to understand its structure and meaning.

[0865] A "solid model" refers to a computer-generated object that has three-dimensional (3D) geometric information, such as 3D data of a building or object.

[0866] "Customer terminal" refers to a computer device used by a user, including, for example, a smartphone, tablet, or personal computer.

[0867] "Displaying in real space" refers to using augmented reality technology to display digital data overlaid on the physical real world.

[0868] "Generative AI" refers to AI techniques that have the ability to generate new data based on specified input data, including deep learning models and neural networks.

[0869] An "artificial neural network" is an algorithm that mimics the activity of nerve cells in the human brain, and refers to a technology that learns from large amounts of data to recognize patterns and generate data.

[0870] This invention relates to a system that acquires data from existing information sources, analyzes the data, generates a highly accurate 3D model using AI technology, transmits the model to a customer's terminal, and displays it in real space. This system is mainly composed of a server and a customer's terminal.

[0871] Server program processing description

[0872] 1. Data Acquisition:

[0873] The server requests the required data from an existing information source (such as an API). It sends an HTTP request to an API endpoint and retrieves the data in JSON format. For example, to retrieve building information for a city, it uses a URL like "https: / / api.example.com / buildings / tokyo".

[0874] 2. Data Analysis:

[0875] The server parses the received JSON data and extracts information such as the building's location, size, shape, etc. The extracted data has the following structure:

[0876] Building 1: Latitude 35.6895, Longitude 139.6917, Height 150, Shape Rectangle

[0877] Building 2: Latitude 35.6581, Longitude 139.7516, Height 200, Shape Circle

[0878] 3. Generate 3D model:

[0879] The server uses the extracted data to provide prompts to a generative artificial intelligence (AI) model, which is trained using deep learning frameworks such as TensorFlow and PyTorch.

[0880] For example, enter the following prompt:

[0881] "latitude": 35.6895, "longitude": 139.6917, "height": 150, "shape": "rectangle"

[0882] The server uses an AI model to generate a 3D model from this data.

[0883] 4. Convert and send the model:

[0884] Convert the generated 3D model into the standard GLTF format. For example, save it as "model1.gltf".

[0885] The server sends this model data to the client terminal as an HTTP response.

[0886] Explanation of the client terminal program processing

[0887] 1. Receiving the model:

[0888] The client device receives the GLTF format 3D model sent from the server and saves it as temporary data, for example, "model1.gltf."

[0889] 2. Initialize the AR engine:

[0890] The customer device will initialize a dedicated AR engine (such as Apple's ARKit or Google's ARCore), and at this time, the settings will be configured to allow use of the camera and location information.

[0891] 3. Viewing the 3D model:

[0892] The customer's device loads the received GLTF model into its AR engine and displays it as an overlay in real space. A 3D model of the building is displayed in the direction the user points the camera.

[0893] Examples:

[0894] Taking the example of AR display of city building data, the server retrieves JSON-formatted building information data from "https: / / api.example.com / buildings / tokyo," extracts the location, size, and shape of each building, and supplies this to the AI ​​model. Using TensorFlow, the AI ​​model generates a highly accurate 3D model in response to a prompt, such as "latitude: 35.6895, longitude: 139.6917, height: 150, shape: rectangle," and sends it to the client device as "model1.gltf" in GLTF format. The client device receives this and uses ARKit to overlay the 3D model on the real world. This allows users to enjoy an AR experience where digital building models are overlaid on the Tokyo cityscape.

[0895] This system makes it possible to generate highly accurate 3D models and overlay them in real space.

[0896] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0897] Step 1:

[0898] The server retrieves data from an existing information source (API). Specifically, it sends an HTTP GET request to an API endpoint and receives JSON-formatted data as a response. For example, it retrieves city building information from "https: / / api.example.com / buildings / tokyo."

[0899] Input: API endpoint

[0900] Output: API response data in JSON format

[0901] Specific operation: Send a GET request to "https: / / api.example.com / buildings / tokyo" and receive JSON data containing information such as the building's location, size, and shape.

[0902] Step 2:

[0903] The server analyzes the received JSON data using a JSON parser to extract necessary information such as the location, size, and shape of each building.

[0904] Input: API response data in JSON format

[0905] Output: Information on the location, size, and shape of the extracted buildings

[0906] Specific behavior: Parse the received JSON data and convert it into the following structure:

[0907] Building 1: Latitude 35.6895, Longitude 139.6917, Height 150, Shape Rectangle

[0908] Building 2: Latitude 35.6581, Longitude 139.7516, Height 200, Shape Circle

[0909] Step 3:

[0910] The server inputs prompts into the AI ​​model based on the extracted data to generate a 3D model. Specifically, it inputs prompts into the AI ​​model (a deep learning model trained with TensorFlow or PyTorch) to generate 3D model data.

[0911] Input: Information on the location, size, and shape of the extracted buildings

[0912] Output: Generated 3D model

[0913] Specific behavior:

[0914] Prompt: "latitude": 35.6895, "longitude": 139.6917, "height": 150, "shape": "rectangle"

[0915] is input into the AI ​​model to generate a three-dimensional model.

[0916] Step 4:

[0917] The server converts the generated 3D model into the standard GLTF format and sends it to the client terminal. Specifically, it converts it into the GLTF format using a model conversion tool and sends it to the client terminal as an HTTP response.

[0918] Input: Generated 3D model

[0919] Output: 3D model data in GLTF format

[0920] Specific operation: The generated 3D model is saved as "model1.gltf" and sent to the client terminal.

[0921] Step 5:

[0922] The client device receives the GLTF format 3D model sent from the server and initializes the AR engine, specifically, a dedicated AR engine such as ARKit or ARCore, and makes the camera and location information available.

[0923] Input: 3D model data in GLTF format

[0924] Output: Initialized AR Engine

[0925] Specific operation: The client device receives "model1.gltf", initializes ARKit, and obtains camera access permission.

[0926] Step 6:

[0927] The client device loads the received GLTF model into the AR engine and displays it as an overlay in real space. Specifically, the GLTF model is loaded into the AR engine and displayed in the direction the user points the camera.

[0928] Input: Initialized AR engine, 3D model data in GLTF format

[0929] Output: 3D model displayed in real space

[0930] Specific operation: The client device loads "model1.gltf" into ARKit and displays a 3D model of the building in real space in the direction the user is pointing the camera.

[0931] This allows users to visually experience a digital three-dimensional model superimposed on a real-world scene.

[0932] (Application example 1)

[0933] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0934] In existing brick-and-mortar stores, customers often have to spend a lot of time finding specific products, and detailed product information is difficult to obtain. Stores also face the challenge of having few ways to communicate product layout changes and inventory status to customers in real time. To solve these problems, there is a need for technology that can easily obtain product location information within a store and present it to users in a visually easy-to-understand manner.

[0935] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0936] In this invention, the server includes means for acquiring information from an existing API, means for analyzing the acquired information, means for generating a 3D model based on the analyzed information, means for transmitting the generated 3D model to a client terminal, and means for the client terminal to provide product location information and detailed information to a user using the 3D model displayed in real space. This enables customers to quickly and accurately find specific products in a physical store and obtain detailed product information in real time.

[0937] "Means of obtaining information from an API" refers to a mechanism for requesting and receiving necessary information from external data providers via the Internet.

[0938] "Means for analyzing information" refers to a program or method for interpreting acquired data according to certain rules and extracting or converting necessary items.

[0939] "Means for generating a 3D model" refers to a technology or process for creating a digital model with a three-dimensional shape based on analyzed information.

[0940] "Means for transmitting the generated 3D model to the client terminal" refers to a mechanism for distributing the 3D model created on the server side to the client terminal via the Internet.

[0941] "Means for displaying a 3D model in real space using a client terminal" refers to a technology in which the client terminal uses a camera and display to visually display a 3D model superimposed on the real environment.

[0942] "A means by which a client terminal provides users with product location information and detailed information using a 3D model displayed in real space" is a mechanism that allows users to visually obtain product location and detailed information via a 3D model displayed in AR.

[0943] "Means for generating 3D models using AI" refers to a method that utilizes artificial intelligence technology to automatically create 3D models from analyzed information.

[0944] "Means using deep learning algorithms" refers to the process of using deep learning technology, a type of artificial intelligence, to analyze complex data patterns and generate 3D models.

[0945] This invention is a system that acquires information from an existing API, generates a 3D model based on that information, and displays it in real space on a client terminal, providing users with product location information and detailed information. Here we will explain the specific system configuration and how to implement it.

[0946] System configuration

[0947] 1. Server

[0948] The server sends a request to an external data provision service (API) via the Internet to obtain the required data.

[0949] The acquired data is analyzed to extract necessary information such as the product's position, shape, and size.

[0950] Based on the extracted information, a 3D model is generated using AI technology, using machine learning frameworks such as TensorFlow.

[0951] The generated 3D model is converted into GLTF format and sent to the client device.

[0952] 2. Client Terminal

[0953] The client terminal receives the 3D model sent from the server.

[0954] Using an AR engine (e.g., ARCore or ARKit), the received 3D model is overlaid onto the real world.

[0955] When a user holds the camera over a product, the product's location and detailed information are visually displayed through AR.

[0956] Specific processing details

[0957] 1. Data acquisition by the server

[0958] The server sends an HTTP request to the specified API to obtain JSON formatted data. For example, it uses an API to obtain the location and detailed information of a store's products.

[0959] 2. Data Analysis and 3D Model Generation

[0960] The acquired data is analyzed to extract the position, size, and shape of each product.

[0961] Based on the extracted information, a 3D model is generated using a machine learning model, using a framework such as TensorFlow.

[0962] 3. Convert and send 3D models

[0963] The generated 3D model is converted into GLTF format and sent to the client terminal as an HTTP response.

[0964] 4. AR display on client device

[0965] The client device loads the 3D model received from the server and initializes the AR engine.

[0966] A 3D model is overlaid onto the real space where the camera is pointed, providing product location information and detailed information.

[0967] Specific examples

[0968] As a specific scenario, the following prompt sentence is used:

[0969] "This application displays the location information of products in the store in real time, allowing you to easily search and view the location of specific products. It also uses AR to display detailed information about products (price, reviews, stock, etc.). ARCore is used for AR display, so accurate location display is possible."

[0970] This allows users to efficiently search for products in the store using their smartphones and easily obtain detailed information.

[0971] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0972] Step 1:

[0973] The server sends an HTTP request to an existing API via the Internet to obtain the required data. Specifically, it receives JSON-formatted data that includes the location, size, shape, etc. of in-store products. The input is the API request, and the output is the JSON data returned as the API response.

[0974] Step 2:

[0975] The server analyzes the acquired JSON data and extracts information about the product's position, size, and shape. Specifically, it parses the JSON data and extracts the parameters of the target product. The input is JSON data, and the output is a list of parsed product information.

[0976] Step 3:

[0977] The server uses an AI model to generate a 3D model based on the extracted product information, applying a deep learning algorithm using a machine learning framework such as TensorFlow. The input is a list of product information, and the output is the generated 3D model.

[0978] Step 4:

[0979] The server converts the generated 3D model into GLTF format and sends it to the client device. Specifically, it encodes the 3D model into a standard file format (GLTF) and sends it as an HTTP response. The input is the 3D model, and the output is a GLTF file.

[0980] Step 5:

[0981] The device loads the GLTF format 3D model received from the server. Specifically, it receives the HTTP response, parses the GLTF file, and reads the data. The input is the GLTF file, and the output is the read 3D model data.

[0982] Step 6:

[0983] The device initializes the AR engine (ARCore or ARKit) and overlays the loaded 3D model in real space. When the user points the camera at the product, the product's location and detailed information are displayed. Specifically, the AR engine is started and the 3D model is overlaid on the device's camera image. The input is the loaded 3D model data, and the output is the user's visual interface.

[0984] Step 7:

[0985] Users use their smartphones to explore products in the store and check the product's location and detailed information displayed in AR. Specifically, they move the smartphone's camera to compare the real world with the 3D model to find the product. The input is a video of the real world and a 3D model display, and the output is visual product location information and detailed information.

[0986] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0987] This invention relates to a system that analyzes information obtained from an API, generates 3D models using AI, and combines it with an emotion engine that recognizes user emotions. This system consists of a server and a client terminal. The server obtains and analyzes API data, generates 3D models, recognizes emotions, and transmits the models. The client terminal receives the generated 3D models and displays them in AR according to the user's emotions.

[0988] System configuration

[0989] 1. Server

[0990] Retrieving API data: The server requests the required data from a specified API. For example, to retrieve building information for a city, the server sends an HTTP request to the API endpoint and retrieves the data in JSON format.

[0991] Data analysis: The server analyzes the acquired JSON data and extracts the information necessary for generating a 3D model, such as the location, size, and shape of the building.

[0992] 3D model generation: The server uses the analyzed data to generate a 3D model using an AI model, which uses a deep learning algorithm.

[0993] Model conversion and serving: The generated 3D model is converted into a standard file format (e.g., GLTF format) and sent to the client terminal as an HTTP response.

[0994] Operation of the emotion engine: The server is equipped with an emotion engine that recognizes emotions from the user's image and voice data, and analyzes the acquired emotion data.

[0995] 2. Client Terminal

[0996] Receiving and displaying the model: The client device receives the 3D model sent from the server and uses the AR engine to overlay the model in real space.

[0997] Initializing the AR engine: The client device initializes a dedicated AR engine for AR display, for example, using a framework such as ARKit or ARCore.

[0998] Display change based on emotion recognition: The client device dynamically changes the content of the displayed 3D model based on the emotion data obtained from the emotion engine.

[0999] Program processing

[1000] Server-side processing

[1001] The server first sends a request to an API endpoint to retrieve the required data.

[1002] The acquired data is analyzed to extract necessary information (e.g., position, size, shape, etc.).

[1003] The server generates 3D objects using an AI model that uses deep learning based on the analysis data.

[1004] The generated 3D model is converted into a standard format such as GLTF and sent to the client terminal as an HTTP response.

[1005] The server receives the user's emotional data and analyzes it using the emotion engine. The emotional data is based on the user's image and voice.

[1006] Processing on the client terminal side

[1007] The client terminal acquires the 3D model data received from the server.

[1008] The device initializes the AR engine and loads the received 3D model data.

[1009] The device dynamically changes the content of the 3D model displayed based on the user's emotion recognition results. As emotions change, corresponding objects and effects are displayed.

[1010] The client device adds 3D models to the AR scene and overlays them in real space, allowing users to experience the visual experience of digital objects superimposed on real-world scenery.

[1011] Specific examples

[1012] For example, if a user wants to view city building data in AR:

[1013] 1. Acquiring API data

[1014] The server retrieves JSON-formatted data from the city building data API, including information on the location, size, and shape of each building.

[1015] 2. Data Analysis

[1016] The server analyzes this data and compiles the necessary information for each building, including its location, size, and shape.

[1017] 3. Generating 3D models

[1018] The server uses the organized information to generate a 3D model using an AI model that uses a deep learning algorithm to reproduce even complex shapes with high accuracy.

[1019] 4. Convert and send the model

[1020] The generated 3D model is converted into GLTF format and sent to the client terminal as an HTTP response.

[1021] 5. Display on the client terminal

[1022] The client device uses an AR engine to overlay the received 3D model onto real space.

[1023] 6. Leveraging Emotional Engines

[1024] The client terminal receives the user's emotional data from the emotion engine and dynamically changes the display content, for example, by changing the color or effects of the building if the user is excited.

[1025] As such, this invention is a technology that efficiently generates 3D models from API data and incorporates advanced emotion recognition functionality to enhance the user experience, thereby reducing development costs and enabling services to be provided quickly and efficiently.

[1026] The processing flow will be explained below.

[1027] Step 1:

[1028] The server sends an HTTP request to a specified API endpoint to retrieve the required data, which may include authentication information and filtering criteria as needed. For example, it retrieves data about the location, size, and shape of buildings from a city building data API.

[1029] Step 2:

[1030] The server analyzes the received JSON data. Specifically, it extracts necessary information from the response data, such as the building's location, size, shape, and color. The extracted data is then organized into an appropriate internal data structure as a preprocessing step for generating a 3D model.

[1031] Step 3:

[1032] The server uses an AI model to generate a 3D model based on the organized data. First, it preprocesses the data using a deep learning algorithm and uses it as input for the AI ​​model. The model predicts the 3D shape of the building and generates it in digital form.

[1033] Step 4:

[1034] The server converts the generated 3D model into a standard file format, such as GLTF or OBJ, which makes it easier for the client device to receive it. After the conversion process is complete, the server sends the file to the client device as an HTTP response.

[1035] Step 5:

[1036] The server receives the user's image and voice data and recognizes emotions using an emotion engine, which analyzes this data and identifies the user's emotional state (e.g., joy, sadness, surprise, etc.).

[1037] Step 6:

[1038] The client device receives the 3D model data sent from the server. The received data is temporarily stored in the device's memory. The device then initializes the AR engine and loads the received 3D model data into the AR scene.

[1039] Step 7:

[1040] The client device dynamically changes the content of the displayed 3D model based on the user's emotional data obtained from the emotion engine. For example, if the user is excited, an effect that emphasizes the color of the building is added.

[1041] Step 8:

[1042] The client device overlays the loaded 3D model onto the real world. When the user looks at the real world through the device, the 3D model of the building is displayed appropriately positioned based on the location information.

[1043] Step 9:

[1044] Through AR devices, users can view 3D models that dynamically change depending on their emotional state, enhancing the fusion of reality and digital and enriching the user experience.

[1045] Example 2

[1046] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1047] Conventional systems were able to obtain information from existing APIs and generate 3D models, but they did not support dynamic model display according to the user's emotions. Therefore, in order to improve the user experience, there was a need for technology that not only displays 3D models generated based on obtained information in real space, but also dynamically changes the display content according to the user's emotions.

[1048] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for acquiring information from an existing API, means for analyzing the acquired information, means for generating a 3D model based on the analyzed information, means for transmitting the generated 3D model to the client terminal, means for analyzing the user's emotions, and means for dynamically changing the display content of the 3D model based on the user's emotions. This makes it possible to dynamically change the display content of the 3D model in accordance with the user's emotions.

[1049] "Existing API" refers to a publicly available application program interface, a set of protocols and tools for accessing a particular service or data.

[1050] "Means of obtaining information" refers to the ability to execute a process that uses an existing API to request data and receive it as a response.

[1051] "Means for analyzing information" refers to the function for structuring acquired data and executing a process to extract necessary information.

[1052] "Means for generating a 3D model" refers to a function that executes a process to create a three-dimensional digital object based on the analyzed information.

[1053] "Means for sending to the client terminal" refers to the function for sending the 3D model data generated on the server side to the client side via the network.

[1054] "Client terminal" refers to a device that processes data received from a server and interacts with a user.

[1055] "Means for displaying 3D models in real space" refers to technology for displaying three-dimensional digital objects overlaid on the physical environment on a client device.

[1056] "Means for analyzing emotions" refers to a function that executes a process to identify the emotional state of a user based on voice and image data collected from the user.

[1057] "Means for dynamically changing display content based on emotions" refers to the function of receiving the results of emotion analysis and changing the display format of the 3D model in real time according to the results.

[1058] This invention relates to a system that acquires information from an API and generates a 3D model using AI. The system consists of a server and a client terminal. The server acquires and analyzes API data, generates a 3D model, recognizes emotions, and transmits the model. The client terminal receives the generated 3D model and displays it in AR according to the user's emotions.

[1059] Server Configuration

[1060] API data retrieval:

[1061] The server requests the necessary data from the specified API. For example, to obtain information about a city's buildings, the server sends an HTTP GET request to the API endpoint and obtains data in JSON format. Specifically, it accesses https: / / api.citydata.com / buildings and obtains information such as the location, size, and shape of the buildings.

[1062] Data analysis:

[1063] The server analyzes the acquired JSON data and extracts the information necessary to generate a 3D model. For example, it uses the Python json library to convert the data into a dictionary format and extracts the location, size, shape, and other information of each building.

[1064] 3D model generation:

[1065] The server uses an AI model to generate a 3D model based on the analyzed data. Deep learning algorithms such as TensorFlow and PyTorch are used here. For example, building shape data is input into the AI ​​model to generate a 3D model.

[1066] Convert and serve the model:

[1067] The generated 3D model is converted to GLTF format and sent to the client device as an HTTP response. This conversion is performed using libraries such as pytree. The converted model data is then sent to the client device.

[1068] Emotion Engine in action:

[1069] The server recognizes emotions based on the image and audio data sent by the user. The emotion recognition engine uses algorithms such as EmotionAI to analyze the emotional data. For example, if the user is laughing, the server analyzes the emotional data and recognizes it as excitement or joy.

[1070] Client terminal configuration

[1071] Receiving and displaying models:

[1072] The client device receives the 3D model data sent from the server and uses an AR engine to overlay the model in real space, specifically using frameworks such as ARKit or ARCore.

[1073] Initialize the AR engine:

[1074] The client device initializes a dedicated AR engine for AR display. For example, on an iOS device, it initializes using ARKit and starts an AR session.

[1075] Emotion-aware display changes:

[1076] The client terminal dynamically changes the content of the displayed 3D model based on the emotional data obtained from the emotion engine. If the user is excited, the color or effects of the building model will be changed.

[1077] Specific examples

[1078] For example, if a user wants to view city building data in AR:

[1079] 1. API data acquisition:

[1080] The server retrieves JSON-formatted data from the city building data API, which includes information on the location, size, and shape of buildings.

[1081] 2. Data Analysis:

[1082] The server analyzes the acquired data and extracts the necessary information (location, size, shape) for each building.

[1083] 3. Generate 3D model:

[1084] The server uses AI models to generate 3D models based on the extracted information, and deep learning algorithms enable high-precision reproduction of even complex shapes.

[1085] 4. Convert and send the model:

[1086] The server converts the generated 3D model into GLTF format and sends it to the client terminal as an HTTP response.

[1087] 5. Display by client terminal:

[1088] The client device uses an AR engine to overlay the received 3D model onto real space.

[1089] 6. Leveraging the Emotion Engine:

[1090] The client terminal acquires the user's emotional data from the emotion engine and dynamically changes the display content, for example, by changing the color or effects of the building if the user is excited.

[1091] An example of a prompt is, "Obtain city building data from the API and generate a 3D model using an AI model based on the analyzed data. Convert the generated 3D model into GLTF format so that the client device can display it using an AR engine. Also, recognize the user's emotions based on their image and voice, and change the display accordingly."

[1092] This system is a technology that improves the user experience by efficiently generating 3D models from API data and incorporating advanced emotion recognition functionality, thereby reducing development costs and enabling the provision of services quickly and efficiently.

[1093] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1094] Step 1:

[1095] Retrieving API data

[1096] The server sends an HTTP GET request to an API endpoint. The input is the API endpoint URL (e.g., https: / / api.citydata.com / buildings). The output is JSON-formatted data containing building information for the city. Specifically, the server sends the request and stores the received response data in storage.

[1097] Step 2:

[1098] Data analysis

[1099] The server parses the acquired JSON data and extracts the necessary information. The input is the JSON data obtained in step 1. The output is the parsed information (e.g., building location, size, and shape data). Specifically, the server uses Python's JSON library to convert the data into a dictionary format and extracts each field.

[1100] Step 3:

[1101] 3D model generation

[1102] The server uses an AI model based on the analysis data to generate a 3D model. The input is the analysis data obtained in step 2. The output is the generated 3D model. Specifically, the server uses a deep learning framework such as TensorFlow or PyTorch to input the analysis data into the AI ​​model and generate the 3D model. For example, it performs the process building_model = ai_model.predict(parsed_data).

[1103] Step 4:

[1104] Converting and serving models

[1105] The server converts the generated 3D model into GLTF format and sends it to the client device. The input is the 3D model generated in step 3. The output is the format-converted GLTF data. Specifically, the server converts the model data using the pytree library and sends the converted GLTF data to the client device as an HTTP response.

[1106] Step 5:

[1107] Emotion Engine Operation

[1108] The server analyzes emotions based on image and audio data sent by the user. The input is the image and audio data sent by the user from their device. The output is the analyzed emotional data. Specifically, the server analyzes the data using an emotion recognition engine and performs the process emotion_result = emotion_engine.analyze(image_data).

[1109] Step 6:

[1110] Receiving and displaying the model

[1111] The device receives 3D model data sent from the server. The input is GLTF data sent from the server. The output is a 3D model loaded on the device. Specifically, the device analyzes the received data and loads it into the AR engine.

[1112] Step 7:

[1113] Initializing the AR engine

[1114] The device initializes the AR engine for AR display. There is no input. The output is an initialized AR session. Specifically, the device uses ARKit, for example, and sets arView = ARView(frame: .zero) and arView.session.run(configuration).

[1115] Step 8:

[1116] Display change based on emotion recognition

[1117] The device dynamically changes the display content of the 3D model based on the emotion data obtained from the server. The input is emotion data. The output is the changed 3D model display. For example, to change the color of a building based on emotion data, set model.materials = [bright_color_material].

[1118] Step 9:

[1119] Adding to an AR scene

[1120] The device adds the 3D model to the AR scene and displays it as an overlay in real space. The input is the 3D model loaded in step 6. The output is the overlayed 3D model. Specifically, the device performs the process arView.scene.add(anchorEntity).

[1121] (Application example 2)

[1122] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1123] In recent years, the demand for augmented reality (AR) technology in real space has been increasing, but existing systems have faced challenges in that it is difficult to change the user experience based on dynamic emotion recognition. In particular, it has been difficult to change the displayed content in real time according to the user's emotions. Furthermore, existing systems often have difficulty efficiently analyzing data obtained from APIs and generating 3D models, which reduces usability.

[1124] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for acquiring information from an existing API, means for analyzing the acquired information, means for generating a 3D model based on the analyzed information, means for transmitting the generated 3D model to a client terminal, means for the client terminal to display the 3D model in real space and dynamically change the content according to the user's emotions, means for the client terminal to recognize the user's emotions and transmit the emotion data to the server, and means for the server to analyze the emotion data and provide the analysis results to the client terminal. This enables efficient analysis of API data and generation of a 3D model, and real-time display changes according to the user's emotions.

[1125] An "existing API" is a defined interface for accessing external data and functions.

[1126] A "means of obtaining information" is a function for sending a request to a specified API endpoint and receiving data.

[1127] "Means for analyzing information" refers to a function for processing acquired data and extracting necessary information.

[1128] "Means for generating a 3D model" refers to a function for creating a three-dimensional object based on the extracted data.

[1129] A "client terminal" is a device that processes data received from a server and displays it to the user.

[1130] "Means for displaying in real space" refers to the function for overlaying a 3D model on a physical environment.

[1131] The "means for dynamically changing content according to the user's emotions" is a function for changing the display content in real time based on the user's emotional data.

[1132] "Means for recognizing user emotions" is a function for determining emotions from the user's image and voice data.

[1133] The "means for transmitting emotion data to the server" is a function for transferring the user's emotion information to the server.

[1134] "Means for analyzing emotional data" refers to a function for processing and analyzing emotional data received on the server side.

[1135] The "means for providing the analysis results to the client terminal" is a function for transmitting the analysis results of emotion data to the client terminal.

[1136] An "AR engine" is a software component that realizes augmented reality technology.

[1137] "Means for dynamic change" is a function for changing the display content in real time.

[1138] The system for implementing this invention consists of a server and a client terminal. The server acquires information from an existing API, analyzes the acquired information, generates a 3D model based on the analyzed information, and sends it to the client terminal. The server also has an emotion engine that recognizes the user's emotions, analyzes the user's emotion data, and provides the results to the client terminal.

[1139] The client device has the ability to display the 3D model sent from the server in real space and dynamically change the content according to the user's emotions. Furthermore, the client device recognizes the user's emotions and sends that emotional data to the server. These functions allow the display content to change in real time according to the user's emotions, enabling an interactive AR experience.

[1140] Hardware and software used

[1141] Server: A typical server computer

[1142] Server-side software: Python, Flask (for building API servers), TensorFlow or PyTorch (deep learning libraries)

[1143] 3D model generation tools: Blender (3D modeling software), Three.js (JavaScript 3D library)

[1144] Emotion recognition system: OpenCV (computer vision library), Dlib (machine learning library)

[1145] Client terminals: smart glasses and head-mounted displays

[1146] Client-side software: Unity (game development engine), ARCore or ARKit (AR development framework)

[1147] Server-side program processing

[1148] The server first sends an HTTP request to the specified API endpoint to obtain information. It then analyzes the obtained JSON-formatted data and extracts necessary information (such as the product's size, shape, and location). It then uses a deep learning algorithm to generate a 3D model based on the extracted information, converts the generated 3D model to GLTF format, and sends it to the client device. The server then recognizes emotions using the user's image and voice data sent from the client device and provides the analysis results to the client device.

[1149] Client terminal program processing

[1150] The client device receives 3D model data from the server, initializes the AR engine, and displays the 3D model in real space. It uses a camera and microphone to recognize emotions and collects the user's emotional data, which it then sends to the server. It also dynamically changes the displayed 3D model and effects based on the acquired emotional data and the analysis results from the server. For example, when a user is looking at a specific product, it displays the product's 3D model, price, and promotion information.

[1151] Specific examples

[1152] For example, imagine a situation where a user is using smart glasses to look at products in a physical store. The server retrieves product data from the product information API, analyzes it, and generates a 3D model. The generated 3D model is displayed through the smart glasses' AR engine. If the user shows interest in a particular product, promotional information and detailed product information for that product are displayed. Furthermore, if the user expresses surprise or delight, the displayed effects change depending on the user's emotion.

[1153] Prompt Sentence Examples

[1154] Create a Unity program that recognizes the user's emotions based on their image data, and changes the effects of the 3D model displayed if the user is excited.

[1155] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1156] Step 1:

[1157] The server sends a request to the API endpoint. The server sends an HTTP request to the specified API endpoint and obtains data in JSON format. It receives API endpoint information as input and obtains JSON format data as output.

[1158] Step 2:

[1159] The server analyzes the acquired JSON data. The server parses the acquired JSON data and extracts necessary information (such as product size, shape, and location information). It receives the acquired JSON data as input and outputs product attribute information as the analysis result.

[1160] Step 3:

[1161] The server generates a 3D model using a deep learning algorithm based on the analyzed information. This uses the generative AI model we defined. It receives product attribute information as input and obtains the generated 3D model data as output.

[1162] Step 4:

[1163] The server sends the generated 3D model to the client terminal. The server converts the generated 3D model into GLTF format and sends it to the client terminal as an HTTP response. The server receives the generated 3D model data as input and obtains a GLTF file to send to the client terminal as output.

[1164] Step 5:

[1165] The client device receives the 3D model data. The device receives the GLTF file sent from the server and prepares it for use in the next step. It receives the GLTF file received from the server as input and obtains the received 3D model data as output.

[1166] Step 6:

[1167] The client device initializes the AR engine. The device reads the received 3D model data and starts the AR engine to prepare for overlaying the model in real space. The device receives the received 3D model data and AR engine initialization information as input, and obtains the initialized AR engine as output.

[1168] Step 7:

[1169] The client device displays the 3D model in real space. The device uses the AR engine to overlay the received 3D model in real space. The device receives the initialized AR engine and the received 3D model data as input, and obtains the 3D model overlaid in real space as output.

[1170] Step 8:

[1171] The client device recognizes the user's emotions. The device uses a camera and microphone to collect the user's image and voice data, which it then inputs into an emotion recognition algorithm (emotion engine). The device receives the user's image and voice data as input, and obtains recognized emotion data as output.

[1172] Step 9:

[1173] The client device sends emotional data to the server. The device then sends the recognized emotional data to the server, and analysis begins on the server side. The recognized emotional data is received as input, and the emotional data sent to the server is obtained as output.

[1174] Step 10:

[1175] The server analyzes the emotion data and provides the results to the client terminal. The server analyzes the received emotion data and sends the analysis results to the client terminal. The server receives the recognized emotion data as input and sends the analysis results to the terminal as output.

[1176] Step 11:

[1177] The client terminal dynamically changes the displayed content. The client terminal dynamically changes the 3D model and effects displayed based on the analysis results from the server. The analysis results from the server are received as input, and the changed display content is obtained as output.

[1178] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1179] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1180] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1181] [Fourth embodiment]

[1182] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1183] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1184] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1185] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1186] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1187] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1188] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1189] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1190] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1191] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1192] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1193] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1194] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1195] This invention relates to a system that analyzes information obtained from an API, generates a 3D model using AI, and displays it in AR on a client device. This system is primarily composed of a server and a client device, where the server obtains and analyzes API data and generates a 3D model, and the client device receives the generated 3D model and displays it in AR.

[1196] System configuration

[1197] 1. Server

[1198] Retrieving API data: The server requests the required data from the specified API. For example, to retrieve building information for a city, the server sends an HTTP request to the API endpoint and retrieves the data in JSON format.

[1199] Data analysis: The server analyzes the acquired JSON data and extracts the information necessary to generate a 3D model, such as the location, size, and shape of the building.

[1200] 3D model generation: The server uses the AI ​​model to generate a 3D model based on the analyzed data. Here, a deep learning algorithm is used.

[1201] Model conversion and provision: The generated 3D model is converted into a standard file format (e.g., GLTF format) and sent to the client terminal.

[1202] 2. Client Terminal

[1203] Receiving and displaying the model: The client device receives the 3D model sent from the server and uses the AR engine to overlay the model in real space.

[1204] Initializing the AR engine: The client device initializes a dedicated AR engine for AR display, for example, using a framework such as ARKit or ARCore.

[1205] Program processing

[1206] Server-side processing

[1207] The server first sends a request to an API endpoint to retrieve the required data.

[1208] The acquired data is analyzed to extract necessary information (e.g., position, size, shape, etc.).

[1209] Based on the extracted data, a 3D model is generated using an AI model, which is based on deep learning algorithms and has advanced shape recognition and generation capabilities.

[1210] The generated 3D model is converted into a standard format such as GLTF and sent to the client terminal as an HTTP response.

[1211] Processing on the client terminal side

[1212] The client terminal receives the 3D model from the server.

[1213] The device initializes the AR engine and loads the received 3D model data.

[1214] The loaded 3D model is overlaid in real space, providing visual feedback to the user.

[1215] Specific examples

[1216] Take the example of AR display of city building data.

[1217] 1. Acquiring API data

[1218] The server retrieves JSON-formatted data from the urban architecture data API, including information on the location, size, and shape of each building.

[1219] 2. Data Analysis

[1220] The server analyzes this data and extracts the location, size, and shape of each building.

[1221] 3. Generating 3D models

[1222] The server uses the extracted information to generate a 3D model of the building using an AI model that uses deep learning technology to reproduce even complex shapes with high accuracy.

[1223] 4. Convert and send the model

[1224] The generated 3D model is converted into GLTF format and sent to the client terminal as an HTTP response.

[1225] 5. Display on the client terminal

[1226] The client device uses an AR engine to overlay the received 3D model onto the real world, allowing users to see the digital building model superimposed on the real cityscape.

[1227] In this way, the present invention effectively utilizes existing API data to efficiently generate advanced 3D models and integrate them into real space, significantly reducing the cost and time required for developing 3D content and enabling rapid service provision.

[1228] The processing flow will be explained below.

[1229] Step 1:

[1230] The server sends an HTTP request to the specified API endpoint to retrieve the required data. This request may include authentication information and filtering conditions as needed. When the API returns a response to the request, the server receives the response data in JSON format.

[1231] Step 2:

[1232] The server analyzes the received JSON data. Specifically, it extracts necessary information from the response data, such as the building's location, size, shape, and color. The extracted data is then organized into an appropriate internal data structure as a preprocessing step for generating a 3D model.

[1233] Step 3:

[1234] The server uses an AI model to generate a 3D model based on the organized data. First, it uses a deep learning algorithm to input the data and run a process to generate a 3D model. At this time, the AI ​​model is pre-trained within the server.

[1235] Step 4:

[1236] The server converts the generated 3D model into a standard file format, such as GLTF or OBJ, which makes it easier for the client device to receive it. After the conversion process is complete, the file is sent to the client device as an HTTP response.

[1237] Step 5:

[1238] The device receives the 3D model data sent from the server. The received data is stored on the device and prepared for AR display. The device then sends this data to the AR engine and loads the 3D model.

[1239] Step 6:

[1240] The device initializes its AR engine, which includes calibrating the camera and scanning the environment, preparing it to accurately display 3D models in real space.

[1241] Step 7:

[1242] The device overlays the loaded 3D model onto the real world. This includes rendering to ensure the 3D model appears in the correct position when the user looks at the real world through the AR device. Once rendering is complete, the user can enjoy a visual experience in which the digital 3D model blends into the real world.

[1243] Through this series of steps, users can easily experience the fusion of real space and digital content.

[1244] Example 1

[1245] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1246] Conventional technology has the problem of being difficult to analyze information obtained from existing APIs, generate 3D models efficiently and with high accuracy based on that information, and display them in real space. It has also been pointed out that the accuracy and realism of the generated 3D models are not sufficient. To solve these problems, an advanced analysis and generation system utilizing AI technology is required.

[1247] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1248] In this invention, the server includes means for acquiring data from existing information sources, means for analyzing the acquired data, means for generating a 3D model based on the analyzed data, means for transmitting the generated 3D model to a customer terminal, and means for displaying the 3D model in real space by the customer terminal, thereby enabling the generation of a highly accurate 3D model and its overlay display in real space.

[1249] "Existing sources" refer to external systems or interfaces that provide data that already exists, such as databases or online services accessible through APIs.

[1250] "Data" refers to information such as specific numbers, characters, images, etc. that are the subject of calculation, analysis, and processing.

[1251] "Analyzing" refers to examining acquired data in detail and processing it to understand its structure and meaning.

[1252] A "solid model" refers to a computer-generated object that has three-dimensional (3D) geometric information, such as 3D data of a building or object.

[1253] "Customer terminal" refers to a computer device used by a user, including, for example, a smartphone, tablet, or personal computer.

[1254] "Displaying in real space" refers to using augmented reality technology to display digital data overlaid on the physical real world.

[1255] "Generative AI" refers to AI techniques that have the ability to generate new data based on specified input data, including deep learning models and neural networks.

[1256] An "artificial neural network" is an algorithm that mimics the activity of nerve cells in the human brain, and refers to a technology that learns from large amounts of data to recognize patterns and generate data.

[1257] This invention relates to a system that acquires data from existing information sources, analyzes the data, generates a highly accurate 3D model using AI technology, transmits the model to a customer's terminal, and displays it in real space. This system is mainly composed of a server and a customer's terminal.

[1258] Server program processing description

[1259] 1. Data Acquisition:

[1260] The server requests the required data from an existing information source (such as an API). It sends an HTTP request to an API endpoint and retrieves the data in JSON format. For example, to retrieve building information for a city, it uses a URL like "https: / / api.example.com / buildings / tokyo".

[1261] 2. Data Analysis:

[1262] The server parses the received JSON data and extracts information such as the building's location, size, shape, etc. The extracted data has the following structure:

[1263] Building 1: Latitude 35.6895, Longitude 139.6917, Height 150, Shape Rectangle

[1264] Building 2: Latitude 35.6581, Longitude 139.7516, Height 200, Shape Circle

[1265] 3. Generate 3D model:

[1266] The server uses the extracted data to provide prompts to a generative artificial intelligence (AI) model, which is trained using deep learning frameworks such as TensorFlow and PyTorch.

[1267] For example, enter the following prompt:

[1268] "latitude": 35.6895, "longitude": 139.6917, "height": 150, "shape": "rectangle"

[1269] The server uses an AI model to generate a 3D model from this data.

[1270] 4. Convert and send the model:

[1271] Convert the generated 3D model into the standard GLTF format. For example, save it as "model1.gltf".

[1272] The server sends this model data to the client terminal as an HTTP response.

[1273] Explanation of the client terminal program processing

[1274] 1. Receiving the model:

[1275] The client device receives the GLTF format 3D model sent from the server and saves it as temporary data, for example, "model1.gltf."

[1276] 2. Initialize the AR engine:

[1277] The customer device will initialize a dedicated AR engine (such as Apple's ARKit or Google's ARCore), and at this time, the settings will be configured to allow use of the camera and location information.

[1278] 3. Viewing the 3D model:

[1279] The customer's device loads the received GLTF model into its AR engine and displays it as an overlay in real space. A 3D model of the building is displayed in the direction the user points the camera.

[1280] Examples:

[1281] Taking the example of AR display of city building data, the server retrieves JSON-formatted building information data from "https: / / api.example.com / buildings / tokyo," extracts the location, size, and shape of each building, and supplies this to the AI ​​model. Using TensorFlow, the AI ​​model generates a highly accurate 3D model in response to a prompt, such as "latitude: 35.6895, longitude: 139.6917, height: 150, shape: rectangle," and sends it to the client device as "model1.gltf" in GLTF format. The client device receives this and uses ARKit to overlay the 3D model on the real world. This allows users to enjoy an AR experience where digital building models are overlaid on the Tokyo cityscape.

[1282] This system makes it possible to generate highly accurate 3D models and overlay them in real space.

[1283] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1284] Step 1:

[1285] The server retrieves data from an existing information source (API). Specifically, it sends an HTTP GET request to an API endpoint and receives JSON-formatted data as a response. For example, it retrieves city building information from "https: / / api.example.com / buildings / tokyo."

[1286] Input: API endpoint

[1287] Output: API response data in JSON format

[1288] Specific operation: Send a GET request to "https: / / api.example.com / buildings / tokyo" and receive JSON data containing information such as the building's location, size, and shape.

[1289] Step 2:

[1290] The server analyzes the received JSON data using a JSON parser to extract necessary information such as the location, size, and shape of each building.

[1291] Input: API response data in JSON format

[1292] Output: Information on the location, size, and shape of the extracted buildings

[1293] Specific behavior: Parse the received JSON data and convert it into the following structure:

[1294] Building 1: Latitude 35.6895, Longitude 139.6917, Height 150, Shape Rectangle

[1295] Building 2: Latitude 35.6581, Longitude 139.7516, Height 200, Shape Circle

[1296] Step 3:

[1297] The server inputs prompts into the AI ​​model based on the extracted data to generate a 3D model. Specifically, it inputs prompts into the AI ​​model (a deep learning model trained with TensorFlow or PyTorch) to generate 3D model data.

[1298] Input: Information on the location, size, and shape of the extracted buildings

[1299] Output: Generated 3D model

[1300] Specific behavior:

[1301] Prompt: "latitude": 35.6895, "longitude": 139.6917, "height": 150, "shape": "rectangle"

[1302] is input into the AI ​​model to generate a three-dimensional model.

[1303] Step 4:

[1304] The server converts the generated 3D model into the standard GLTF format and sends it to the client terminal. Specifically, it converts it into the GLTF format using a model conversion tool and sends it to the client terminal as an HTTP response.

[1305] Input: Generated 3D model

[1306] Output: 3D model data in GLTF format

[1307] Specific operation: The generated 3D model is saved as "model1.gltf" and sent to the client terminal.

[1308] Step 5:

[1309] The client device receives the GLTF format 3D model sent from the server and initializes the AR engine, specifically, a dedicated AR engine such as ARKit or ARCore, and makes the camera and location information available.

[1310] Input: 3D model data in GLTF format

[1311] Output: Initialized AR Engine

[1312] Specific operation: The client device receives "model1.gltf", initializes ARKit, and obtains camera access permission.

[1313] Step 6:

[1314] The client device loads the received GLTF model into the AR engine and displays it as an overlay in real space. Specifically, the GLTF model is loaded into the AR engine and displayed in the direction the user points the camera.

[1315] Input: Initialized AR engine, 3D model data in GLTF format

[1316] Output: 3D model displayed in real space

[1317] Specific operation: The client device loads "model1.gltf" into ARKit and displays a 3D model of the building in real space in the direction the user is pointing the camera.

[1318] This allows users to visually experience a digital three-dimensional model superimposed on a real-world scene.

[1319] (Application example 1)

[1320] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1321] In existing brick-and-mortar stores, customers often have to spend a lot of time finding specific products, and detailed product information is difficult to obtain. Stores also face the challenge of having few ways to communicate product layout changes and inventory status to customers in real time. To solve these problems, there is a need for technology that can easily obtain product location information within a store and present it to users in a visually easy-to-understand manner.

[1322] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1323] In this invention, the server includes means for acquiring information from an existing API, means for analyzing the acquired information, means for generating a 3D model based on the analyzed information, means for transmitting the generated 3D model to a client terminal, and means for the client terminal to provide product location information and detailed information to a user using the 3D model displayed in real space. This enables customers to quickly and accurately find specific products in a physical store and obtain detailed product information in real time.

[1324] "Means of obtaining information from an API" refers to a mechanism for requesting and receiving necessary information from external data providers via the Internet.

[1325] "Means for analyzing information" refers to a program or method for interpreting acquired data according to certain rules and extracting or converting necessary items.

[1326] "Means for generating a 3D model" refers to a technology or process for creating a digital model with a three-dimensional shape based on analyzed information.

[1327] "Means for transmitting the generated 3D model to the client terminal" refers to a mechanism for distributing the 3D model created on the server side to the client terminal via the Internet.

[1328] "Means for displaying a 3D model in real space using a client terminal" refers to a technology in which the client terminal uses a camera and display to visually display a 3D model superimposed on the real environment.

[1329] "A means by which a client terminal provides users with product location information and detailed information using a 3D model displayed in real space" is a mechanism that allows users to visually obtain product location and detailed information via a 3D model displayed in AR.

[1330] "Means for generating 3D models using AI" refers to a method that utilizes artificial intelligence technology to automatically create 3D models from analyzed information.

[1331] "Means using deep learning algorithms" refers to the process of using deep learning technology, a type of artificial intelligence, to analyze complex data patterns and generate 3D models.

[1332] This invention is a system that acquires information from an existing API, generates a 3D model based on that information, and displays it in real space on a client terminal, providing users with product location information and detailed information. Here we will explain the specific system configuration and how to implement it.

[1333] System configuration

[1334] 1. Server

[1335] The server sends a request to an external data provision service (API) via the Internet to obtain the required data.

[1336] The acquired data is analyzed to extract necessary information such as the product's position, shape, and size.

[1337] Based on the extracted information, a 3D model is generated using AI technology, using machine learning frameworks such as TensorFlow.

[1338] The generated 3D model is converted into GLTF format and sent to the client device.

[1339] 2. Client Terminal

[1340] The client terminal receives the 3D model sent from the server.

[1341] Using an AR engine (e.g., ARCore or ARKit), the received 3D model is overlaid onto the real world.

[1342] When a user holds the camera over a product, the product's location and detailed information are visually displayed through AR.

[1343] Specific processing details

[1344] 1. Data acquisition by the server

[1345] The server sends an HTTP request to the specified API to obtain JSON formatted data. For example, it uses an API to obtain the location and detailed information of a store's products.

[1346] 2. Data Analysis and 3D Model Generation

[1347] The acquired data is analyzed to extract the position, size, and shape of each product.

[1348] Based on the extracted information, a 3D model is generated using a machine learning model, using a framework such as TensorFlow.

[1349] 3. Convert and send 3D models

[1350] The generated 3D model is converted into GLTF format and sent to the client terminal as an HTTP response.

[1351] 4. AR display on client device

[1352] The client device loads the 3D model received from the server and initializes the AR engine.

[1353] A 3D model is overlaid onto the real space where the camera is pointed, providing product location information and detailed information.

[1354] Specific examples

[1355] As a specific scenario, the following prompt sentence is used:

[1356] "This application displays the location information of products in the store in real time, allowing you to easily search and view the location of specific products. It also uses AR to display detailed information about products (price, reviews, stock, etc.). ARCore is used for AR display, so accurate location display is possible."

[1357] This allows users to efficiently search for products in the store using their smartphones and easily obtain detailed information.

[1358] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1359] Step 1:

[1360] The server sends an HTTP request to an existing API via the Internet to obtain the required data. Specifically, it receives JSON-formatted data that includes the location, size, shape, etc. of in-store products. The input is the API request, and the output is the JSON data returned as the API response.

[1361] Step 2:

[1362] The server analyzes the acquired JSON data and extracts information about the product's position, size, and shape. Specifically, it parses the JSON data and extracts the parameters of the target product. The input is JSON data, and the output is a list of parsed product information.

[1363] Step 3:

[1364] The server uses an AI model to generate a 3D model based on the extracted product information, applying a deep learning algorithm using a machine learning framework such as TensorFlow. The input is a list of product information, and the output is the generated 3D model.

[1365] Step 4:

[1366] The server converts the generated 3D model into GLTF format and sends it to the client device. Specifically, it encodes the 3D model into a standard file format (GLTF) and sends it as an HTTP response. The input is the 3D model, and the output is a GLTF file.

[1367] Step 5:

[1368] The device loads the GLTF format 3D model received from the server. Specifically, it receives the HTTP response, parses the GLTF file, and reads the data. The input is the GLTF file, and the output is the read 3D model data.

[1369] Step 6:

[1370] The device initializes the AR engine (ARCore or ARKit) and overlays the loaded 3D model in real space. When the user points the camera at the product, the product's location and detailed information are displayed. Specifically, the AR engine is started and the 3D model is overlaid on the device's camera image. The input is the loaded 3D model data, and the output is the user's visual interface.

[1371] Step 7:

[1372] Users use their smartphones to explore products in the store and check the product's location and detailed information displayed in AR. Specifically, they move the smartphone's camera to compare the real world with the 3D model to find the product. The input is a video of the real world and a 3D model display, and the output is visual product location information and detailed information.

[1373] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1374] This invention relates to a system that analyzes information obtained from an API, generates 3D models using AI, and combines it with an emotion engine that recognizes user emotions. This system consists of a server and a client terminal. The server obtains and analyzes API data, generates 3D models, recognizes emotions, and transmits the models. The client terminal receives the generated 3D models and displays them in AR according to the user's emotions.

[1375] System configuration

[1376] 1. Server

[1377] Retrieving API data: The server requests the required data from a specified API. For example, to retrieve building information for a city, the server sends an HTTP request to the API endpoint and retrieves the data in JSON format.

[1378] Data analysis: The server analyzes the acquired JSON data and extracts the information necessary for generating a 3D model, such as the location, size, and shape of the building.

[1379] 3D model generation: The server uses the analyzed data to generate a 3D model using an AI model, which uses a deep learning algorithm.

[1380] Model conversion and serving: The generated 3D model is converted into a standard file format (e.g., GLTF format) and sent to the client terminal as an HTTP response.

[1381] Operation of the emotion engine: The server is equipped with an emotion engine that recognizes emotions from the user's image and voice data, and analyzes the acquired emotion data.

[1382] 2. Client Terminal

[1383] Receiving and displaying the model: The client device receives the 3D model sent from the server and uses the AR engine to overlay the model in real space.

[1384] Initializing the AR engine: The client device initializes a dedicated AR engine for AR display, for example, using a framework such as ARKit or ARCore.

[1385] Display change based on emotion recognition: The client device dynamically changes the content of the displayed 3D model based on the emotion data obtained from the emotion engine.

[1386] Program processing

[1387] Server-side processing

[1388] The server first sends a request to an API endpoint to retrieve the required data.

[1389] The acquired data is analyzed to extract necessary information (e.g., position, size, shape, etc.).

[1390] The server generates 3D objects using an AI model that uses deep learning based on the analysis data.

[1391] The generated 3D model is converted into a standard format such as GLTF and sent to the client terminal as an HTTP response.

[1392] The server receives the user's emotional data and analyzes it using the emotion engine. The emotional data is based on the user's image and voice.

[1393] Processing on the client terminal side

[1394] The client terminal acquires the 3D model data received from the server.

[1395] The device initializes the AR engine and loads the received 3D model data.

[1396] The device dynamically changes the content of the 3D model displayed based on the user's emotion recognition results. As emotions change, corresponding objects and effects are displayed.

[1397] The client device adds 3D models to the AR scene and overlays them in real space, allowing users to experience the visual experience of digital objects superimposed on real-world scenery.

[1398] Specific examples

[1399] For example, if a user wants to view city building data in AR:

[1400] 1. Acquiring API data

[1401] The server retrieves JSON-formatted data from the city building data API, including information on the location, size, and shape of each building.

[1402] 2. Data Analysis

[1403] The server analyzes this data and compiles the necessary information for each building, including its location, size, and shape.

[1404] 3. Generating 3D models

[1405] The server uses the organized information to generate a 3D model using an AI model that uses a deep learning algorithm to reproduce even complex shapes with high accuracy.

[1406] 4. Convert and send the model

[1407] The generated 3D model is converted into GLTF format and sent to the client terminal as an HTTP response.

[1408] 5. Display on the client terminal

[1409] The client device uses an AR engine to overlay the received 3D model onto real space.

[1410] 6. Leveraging Emotional Engines

[1411] The client terminal receives the user's emotional data from the emotion engine and dynamically changes the display content, for example, by changing the color or effects of the building if the user is excited.

[1412] As such, this invention is a technology that efficiently generates 3D models from API data and incorporates advanced emotion recognition functionality to enhance the user experience, thereby reducing development costs and enabling services to be provided quickly and efficiently.

[1413] The processing flow will be explained below.

[1414] Step 1:

[1415] The server sends an HTTP request to a specified API endpoint to retrieve the required data, which may include authentication information and filtering criteria as needed. For example, it retrieves data about the location, size, and shape of buildings from a city building data API.

[1416] Step 2:

[1417] The server analyzes the received JSON data. Specifically, it extracts necessary information from the response data, such as the building's location, size, shape, and color. The extracted data is then organized into an appropriate internal data structure as a preprocessing step for generating a 3D model.

[1418] Step 3:

[1419] The server uses an AI model to generate a 3D model based on the organized data. First, it preprocesses the data using a deep learning algorithm and uses it as input for the AI ​​model. The model predicts the 3D shape of the building and generates it in digital form.

[1420] Step 4:

[1421] The server converts the generated 3D model into a standard file format, such as GLTF or OBJ, which makes it easier for the client device to receive it. After the conversion process is complete, the server sends the file to the client device as an HTTP response.

[1422] Step 5:

[1423] The server receives the user's image and voice data and recognizes emotions using an emotion engine, which analyzes this data and identifies the user's emotional state (e.g., joy, sadness, surprise, etc.).

[1424] Step 6:

[1425] The client device receives the 3D model data sent from the server. The received data is temporarily stored in the device's memory. The device then initializes the AR engine and loads the received 3D model data into the AR scene.

[1426] Step 7:

[1427] The client device dynamically changes the content of the displayed 3D model based on the user's emotional data obtained from the emotion engine. For example, if the user is excited, an effect that emphasizes the color of the building is added.

[1428] Step 8:

[1429] The client device overlays the loaded 3D model onto the real world. When the user looks at the real world through the device, the 3D model of the building is displayed appropriately positioned based on the location information.

[1430] Step 9:

[1431] Through AR devices, users can view 3D models that dynamically change depending on their emotional state, enhancing the fusion of reality and digital and enriching the user experience.

[1432] Example 2

[1433] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1434] Conventional systems were able to obtain information from existing APIs and generate 3D models, but they did not support dynamic model display according to the user's emotions. Therefore, in order to improve the user experience, there was a need for technology that not only displays 3D models generated based on obtained information in real space, but also dynamically changes the display content according to the user's emotions.

[1435] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for acquiring information from an existing API, means for analyzing the acquired information, means for generating a 3D model based on the analyzed information, means for transmitting the generated 3D model to the client terminal, means for analyzing the user's emotions, and means for dynamically changing the display content of the 3D model based on the user's emotions. This makes it possible to dynamically change the display content of the 3D model in accordance with the user's emotions.

[1436] "Existing API" refers to a publicly available application program interface, a set of protocols and tools for accessing a particular service or data.

[1437] "Means of obtaining information" refers to the ability to execute a process that uses an existing API to request data and receive it as a response.

[1438] "Means for analyzing information" refers to the function for structuring acquired data and executing a process to extract necessary information.

[1439] "Means for generating a 3D model" refers to a function that executes a process to create a three-dimensional digital object based on the analyzed information.

[1440] "Means for sending to the client terminal" refers to the function for sending the 3D model data generated on the server side to the client side via the network.

[1441] "Client terminal" refers to a device that processes data received from a server and interacts with a user.

[1442] "Means for displaying 3D models in real space" refers to technology for displaying three-dimensional digital objects overlaid on the physical environment on a client device.

[1443] "Means for analyzing emotions" refers to a function that executes a process to identify the emotional state of a user based on voice and image data collected from the user.

[1444] "Means for dynamically changing display content based on emotions" refers to the function of receiving the results of emotion analysis and changing the display format of the 3D model in real time according to the results.

[1445] This invention relates to a system that acquires information from an API and generates a 3D model using AI. The system consists of a server and a client terminal. The server acquires and analyzes API data, generates a 3D model, recognizes emotions, and transmits the model. The client terminal receives the generated 3D model and displays it in AR according to the user's emotions.

[1446] Server Configuration

[1447] API data retrieval:

[1448] The server requests the necessary data from the specified API. For example, to obtain information about a city's buildings, the server sends an HTTP GET request to the API endpoint and obtains data in JSON format. Specifically, it accesses https: / / api.citydata.com / buildings and obtains information such as the location, size, and shape of the buildings.

[1449] Data analysis:

[1450] The server analyzes the acquired JSON data and extracts the information necessary to generate a 3D model. For example, it uses the Python json library to convert the data into a dictionary format and extracts the location, size, shape, and other information of each building.

[1451] 3D model generation:

[1452] The server uses an AI model to generate a 3D model based on the analyzed data. Deep learning algorithms such as TensorFlow and PyTorch are used here. For example, building shape data is input into the AI ​​model to generate a 3D model.

[1453] Convert and serve the model:

[1454] The generated 3D model is converted to GLTF format and sent to the client device as an HTTP response. This conversion is performed using libraries such as pytree. The converted model data is then sent to the client device.

[1455] Emotion Engine in action:

[1456] The server recognizes emotions based on the image and audio data sent by the user. The emotion recognition engine uses algorithms such as EmotionAI to analyze the emotional data. For example, if the user is laughing, the server analyzes the emotional data and recognizes it as excitement or joy.

[1457] Client terminal configuration

[1458] Receiving and displaying models:

[1459] The client device receives the 3D model data sent from the server and uses an AR engine to overlay the model in real space, specifically using frameworks such as ARKit or ARCore.

[1460] Initialize the AR engine:

[1461] The client device initializes a dedicated AR engine for AR display. For example, on an iOS device, it initializes using ARKit and starts an AR session.

[1462] Emotion-aware display changes:

[1463] The client terminal dynamically changes the content of the displayed 3D model based on the emotional data obtained from the emotion engine. If the user is excited, the color or effects of the building model will be changed.

[1464] Specific examples

[1465] For example, if a user wants to view city building data in AR:

[1466] 1. API data acquisition:

[1467] The server retrieves JSON-formatted data from the city building data API, which includes information on the location, size, and shape of buildings.

[1468] 2. Data Analysis:

[1469] The server analyzes the acquired data and extracts the necessary information (location, size, shape) for each building.

[1470] 3. Generate 3D model:

[1471] The server uses AI models to generate 3D models based on the extracted information, and deep learning algorithms enable high-precision reproduction of even complex shapes.

[1472] 4. Convert and send the model:

[1473] The server converts the generated 3D model into GLTF format and sends it to the client terminal as an HTTP response.

[1474] 5. Display by client terminal:

[1475] The client device uses an AR engine to overlay the received 3D model onto real space.

[1476] 6. Leveraging the Emotion Engine:

[1477] The client terminal acquires the user's emotional data from the emotion engine and dynamically changes the display content, for example, by changing the color or effects of the building if the user is excited.

[1478] An example of a prompt is, "Obtain city building data from the API and generate a 3D model using an AI model based on the analyzed data. Convert the generated 3D model into GLTF format so that the client device can display it using an AR engine. Also, recognize the user's emotions based on their image and voice, and change the display accordingly."

[1479] This system is a technology that improves the user experience by efficiently generating 3D models from API data and incorporating advanced emotion recognition functionality, thereby reducing development costs and enabling the provision of services quickly and efficiently.

[1480] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1481] Step 1:

[1482] Retrieving API data

[1483] The server sends an HTTP GET request to an API endpoint. The input is the API endpoint URL (e.g., https: / / api.citydata.com / buildings). The output is JSON-formatted data containing building information for the city. Specifically, the server sends the request and stores the received response data in storage.

[1484] Step 2:

[1485] Data analysis

[1486] The server parses the acquired JSON data and extracts the necessary information. The input is the JSON data obtained in step 1. The output is the parsed information (e.g., building location, size, and shape data). Specifically, the server uses Python's JSON library to convert the data into a dictionary format and extracts each field.

[1487] Step 3:

[1488] 3D model generation

[1489] The server uses an AI model based on the analysis data to generate a 3D model. The input is the analysis data obtained in step 2. The output is the generated 3D model. Specifically, the server uses a deep learning framework such as TensorFlow or PyTorch to input the analysis data into the AI ​​model and generate the 3D model. For example, it performs the process building_model = ai_model.predict(parsed_data).

[1490] Step 4:

[1491] Converting and serving models

[1492] The server converts the generated 3D model into GLTF format and sends it to the client device. The input is the 3D model generated in step 3. The output is the format-converted GLTF data. Specifically, the server converts the model data using the pytree library and sends the converted GLTF data to the client device as an HTTP response.

[1493] Step 5:

[1494] Emotion Engine Operation

[1495] The server analyzes emotions based on image and audio data sent by the user. The input is the image and audio data sent by the user from their device. The output is the analyzed emotional data. Specifically, the server analyzes the data using an emotion recognition engine and performs the process emotion_result = emotion_engine.analyze(image_data).

[1496] Step 6:

[1497] Receiving and displaying the model

[1498] The device receives 3D model data sent from the server. The input is GLTF data sent from the server. The output is a 3D model loaded on the device. Specifically, the device analyzes the received data and loads it into the AR engine.

[1499] Step 7:

[1500] Initializing the AR engine

[1501] The device initializes the AR engine for AR display. There is no input. The output is an initialized AR session. Specifically, the device uses ARKit, for example, and sets arView = ARView(frame: .zero) and arView.session.run(configuration).

[1502] Step 8:

[1503] Display change based on emotion recognition

[1504] The device dynamically changes the display content of the 3D model based on the emotion data obtained from the server. The input is emotion data. The output is the changed 3D model display. For example, to change the color of a building based on emotion data, set model.materials = [bright_color_material].

[1505] Step 9:

[1506] Adding to an AR scene

[1507] The device adds the 3D model to the AR scene and displays it as an overlay in real space. The input is the 3D model loaded in step 6. The output is the overlayed 3D model. Specifically, the device performs the process arView.scene.add(anchorEntity).

[1508] (Application example 2)

[1509] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1510] In recent years, the demand for augmented reality (AR) technology in real space has been increasing, but existing systems have faced challenges in that it is difficult to change the user experience based on dynamic emotion recognition. In particular, it has been difficult to change the displayed content in real time according to the user's emotions. Furthermore, existing systems often have difficulty efficiently analyzing data obtained from APIs and generating 3D models, which reduces usability.

[1511] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for acquiring information from an existing API, means for analyzing the acquired information, means for generating a 3D model based on the analyzed information, means for transmitting the generated 3D model to a client terminal, means for the client terminal to display the 3D model in real space and dynamically change the content according to the user's emotions, means for the client terminal to recognize the user's emotions and transmit the emotion data to the server, and means for the server to analyze the emotion data and provide the analysis results to the client terminal. This enables efficient analysis of API data and generation of a 3D model, and real-time display changes according to the user's emotions.

[1512] An "existing API" is a defined interface for accessing external data and functions.

[1513] A "means of obtaining information" is a function for sending a request to a specified API endpoint and receiving data.

[1514] "Means for analyzing information" refers to a function for processing acquired data and extracting necessary information.

[1515] "Means for generating a 3D model" refers to a function for creating a three-dimensional object based on the extracted data.

[1516] A "client terminal" is a device that processes data received from a server and displays it to the user.

[1517] "Means for displaying in real space" refers to the function for overlaying a 3D model on a physical environment.

[1518] The "means for dynamically changing content according to the user's emotions" is a function for changing the display content in real time based on the user's emotional data.

[1519] "Means for recognizing user emotions" is a function for determining emotions from the user's image and voice data.

[1520] The "means for transmitting emotion data to the server" is a function for transferring the user's emotion information to the server.

[1521] "Means for analyzing emotional data" refers to a function for processing and analyzing emotional data received on the server side.

[1522] The "means for providing the analysis results to the client terminal" is a function for transmitting the analysis results of emotion data to the client terminal.

[1523] An "AR engine" is a software component that realizes augmented reality technology.

[1524] "Means for dynamic change" is a function for changing the display content in real time.

[1525] The system for implementing this invention consists of a server and a client terminal. The server acquires information from an existing API, analyzes the acquired information, generates a 3D model based on the analyzed information, and sends it to the client terminal. The server also has an emotion engine that recognizes the user's emotions, analyzes the user's emotion data, and provides the results to the client terminal.

[1526] The client device has the ability to display the 3D model sent from the server in real space and dynamically change the content according to the user's emotions. Furthermore, the client device recognizes the user's emotions and sends that emotional data to the server. These functions allow the display content to change in real time according to the user's emotions, enabling an interactive AR experience.

[1527] Hardware and software used

[1528] Server: A typical server computer

[1529] Server-side software: Python, Flask (for building API servers), TensorFlow or PyTorch (deep learning libraries)

[1530] 3D model generation tools: Blender (3D modeling software), Three.js (JavaScript 3D library)

[1531] Emotion recognition system: OpenCV (computer vision library), Dlib (machine learning library)

[1532] Client terminals: smart glasses and head-mounted displays

[1533] Client-side software: Unity (game development engine), ARCore or ARKit (AR development framework)

[1534] Server-side program processing

[1535] The server first sends an HTTP request to the specified API endpoint to obtain information. It then analyzes the obtained JSON-formatted data and extracts necessary information (such as the product's size, shape, and location). It then uses a deep learning algorithm to generate a 3D model based on the extracted information, converts the generated 3D model to GLTF format, and sends it to the client device. The server then recognizes emotions using the user's image and voice data sent from the client device and provides the analysis results to the client device.

[1536] Client terminal program processing

[1537] The client device receives 3D model data from the server, initializes the AR engine, and displays the 3D model in real space. It uses a camera and microphone to recognize emotions and collects the user's emotional data, which it then sends to the server. It also dynamically changes the displayed 3D model and effects based on the acquired emotional data and the analysis results from the server. For example, when a user is looking at a specific product, it displays the product's 3D model, price, and promotion information.

[1538] Specific examples

[1539] For example, imagine a situation where a user is using smart glasses to look at products in a physical store. The server retrieves product data from the product information API, analyzes it, and generates a 3D model. The generated 3D model is displayed through the smart glasses' AR engine. If the user shows interest in a particular product, promotional information and detailed product information for that product are displayed. Furthermore, if the user expresses surprise or delight, the displayed effects change depending on the user's emotion.

[1540] Prompt Sentence Examples

[1541] Create a Unity program that recognizes the user's emotions based on their image data, and changes the effects of the 3D model displayed if the user is excited.

[1542] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1543] Step 1:

[1544] The server sends a request to the API endpoint. The server sends an HTTP request to the specified API endpoint and obtains data in JSON format. It receives API endpoint information as input and obtains JSON format data as output.

[1545] Step 2:

[1546] The server analyzes the acquired JSON data. The server parses the acquired JSON data and extracts necessary information (such as product size, shape, and location information). It receives the acquired JSON data as input and outputs product attribute information as the analysis result.

[1547] Step 3:

[1548] The server generates a 3D model using a deep learning algorithm based on the analyzed information. This uses the generative AI model we defined. It receives product attribute information as input and obtains the generated 3D model data as output.

[1549] Step 4:

[1550] The server sends the generated 3D model to the client terminal. The server converts the generated 3D model into GLTF format and sends it to the client terminal as an HTTP response. The server receives the generated 3D model data as input and obtains a GLTF file to send to the client terminal as output.

[1551] Step 5:

[1552] The client device receives the 3D model data. The device receives the GLTF file sent from the server and prepares it for use in the next step. It receives the GLTF file received from the server as input and obtains the received 3D model data as output.

[1553] Step 6:

[1554] The client device initializes the AR engine. The device reads the received 3D model data and starts the AR engine to prepare for overlaying the model in real space. The device receives the received 3D model data and AR engine initialization information as input, and obtains the initialized AR engine as output.

[1555] Step 7:

[1556] The client device displays the 3D model in real space. The device uses the AR engine to overlay the received 3D model in real space. The device receives the initialized AR engine and the received 3D model data as input, and obtains the 3D model overlaid in real space as output.

[1557] Step 8:

[1558] The client device recognizes the user's emotions. The device uses a camera and microphone to collect the user's image and voice data, which it then inputs into an emotion recognition algorithm (emotion engine). The device receives the user's image and voice data as input, and obtains recognized emotion data as output.

[1559] Step 9:

[1560] The client device sends emotional data to the server. The device then sends the recognized emotional data to the server, and analysis begins on the server side. The recognized emotional data is received as input, and the emotional data sent to the server is obtained as output.

[1561] Step 10:

[1562] The server analyzes the emotion data and provides the results to the client terminal. The server analyzes the received emotion data and sends the analysis results to the client terminal. The server receives the recognized emotion data as input and sends the analysis results to the terminal as output.

[1563] Step 11:

[1564] The client terminal dynamically changes the displayed content. The client terminal dynamically changes the 3D model and effects displayed based on the analysis results from the server. The analysis results from the server are received as input, and the changed display content is obtained as output.

[1565] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1566] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1567] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1568] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1569] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1570] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1571] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1572] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1573] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1574] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1575] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1576] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1577] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1578] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1579] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1580] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1581] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1582] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1583] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1584] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1585] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1586] The following is further disclosed regarding the above embodiment.

[1587] (Claim 1)

[1588] A means of obtaining information from existing APIs,

[1589] means for analyzing the acquired information;

[1590] A means for generating a 3D model based on the analyzed information;

[1591] A means for transmitting the generated 3D model to a client terminal;

[1592] A system including a means for displaying a 3D model in real space by a client terminal.

[1593] (Claim 2)

[1594] 10. The system of claim 1, further comprising means for generating a 3D model using AI based on the acquired information.

[1595] (Claim 3)

[1596] 10. The system of claim 1, further comprising: means for using a deep learning algorithm in converting the analyzed information into the 3D model.

[1597] (Claim 4)

[1598] 10. The system of claim 1, further comprising means for converting the generated 3D model into a standard file format (GLTF or OBJ).

[1599] (Claim 5)

[1600] The system according to claim 1, further comprising means for the client terminal to overlay and display the received 3D model in real space using an AR engine.

[1601] "Example 1"

[1602] (Claim 1)

[1603] a means of obtaining data from existing sources;

[1604] means for analyzing the acquired data;

[1605] a means for generating a three-dimensional model based on the analyzed data;

[1606] means for transmitting the generated three-dimensional model to a customer terminal;

[1607] A system including a means for displaying a three-dimensional model in real space by a customer terminal.

[1608] (Claim 2)

[1609] 10. The system of claim 1, further comprising means for generating a solid model based on the acquired data using generative artificial intelligence.

[1610] (Claim 3)

[1611] 10. The system of claim 1, further comprising means for using an artificial neural network in converting the analyzed data into a solid model.

[1612] "Application Example 1"

[1613] (Claim 1)

[1614] A means of obtaining information from existing APIs,

[1615] means for analyzing the acquired information;

[1616] A means for generating a 3D model based on the analyzed information;

[1617] A means for transmitting the generated 3D model to a client terminal;

[1618] A means for displaying the 3D model in real space by a client terminal;

[1619] A system including a means for a client terminal to provide product location information and detailed information to users using a 3D model displayed in real space.

[1620] (Claim 2)

[1621] 10. The system of claim 1, further comprising means for generating a 3D model using AI based on the acquired information.

[1622] (Claim 3)

[1623] 10. The system of claim 1, further comprising: means for using a deep learning algorithm in converting the analyzed information into the 3D model.

[1624] "Example 2: Combining Emotion Engines"

[1625] (Claim 1)

[1626] A means of obtaining information from existing APIs,

[1627] means for analyzing the acquired information;

[1628] A means for generating a 3D model based on the analyzed information;

[1629] A means for transmitting the generated 3D model to a client terminal;

[1630] A means for displaying the 3D model in real space by a client terminal;

[1631] means for analyzing user emotions;

[1632] A system including a means for dynamically changing the display content of a 3D model based on a user's emotions.

[1633] (Claim 2)

[1634] 10. The system of claim 1, further comprising means for generating a 3D model using AI based on the acquired information.

[1635] (Claim 3)

[1636] 10. The system of claim 1, further comprising: means for using a deep learning algorithm in converting the analyzed information into the 3D model.

[1637] "Application example 2 when combining emotion engines"

[1638] (Claim 1)

[1639] A means of obtaining information from existing APIs,

[1640] means for analyzing the acquired information;

[1641] A means for generating a 3D model based on the analyzed information;

[1642] A means for transmitting the generated 3D model to a client terminal;

[1643] A means for displaying the 3D model in real space by a client terminal and dynamically changing the content according to the user's emotions;

[1644] A means for the client terminal to recognize the user's emotion and transmit the emotion data to the server;

[1645] A means for the server to analyze the emotion data and provide the analysis result to the client terminal;

[1646] A system including:

[1647] (Claim 2)

[1648] The system according to claim 1, further comprising: means for generating a 3D model using AI based on the acquired information; means for starting an AR engine on the client terminal and overlaying the generated 3D model in real space; and means for initializing the AR engine on the client terminal.

[1649] (Claim 3)

[1650] 10. The system of claim 1, further comprising: means for using a deep learning algorithm when converting the analyzed information into a 3D model; and means for dynamically modifying the data at the client terminal. [Explanation of symbols]

[1651] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. A means of obtaining information from existing APIs, means for analyzing the acquired information; A means for generating a 3D model based on the analyzed information; A means for transmitting the generated 3D model to a client terminal; A system including a means for displaying a 3D model in real space by a client terminal.

2. The system of claim 1 , further comprising means for generating a 3D model using AI based on the acquired information.

3. 10. The system of claim 1, further comprising means for using a deep learning algorithm in converting the analyzed information into a 3D model.

4. The system of claim 1 , further comprising means for converting the generated 3D model into a standard file format (GLTF or OBJ).

5. The system according to claim 1 , further comprising means for the client terminal to overlay and display the received 3D model in real space using an AR engine.

Images