System

Abstract

A system is provided.SOLUTION: A system comprising: a terminal including a camera configured to capture a 360 ° image of an object; a server configured to receive a plurality of images captured by the camera, analyze the images, and generate a three dimensional model of the object; a server configured to select block parts for reproducing the object based on the three dimensional model, generate an assembly drawing, and transmit the assembly drawing to the terminal; and a user configured to assemble blocks according to the assembly drawing displayed on the terminal.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Conventional block toys are primarily assembled based on commercialized assembly diagrams, limiting the degree of freedom for creativity. Furthermore, they are primarily used as educational toys, and have not been fully utilized as learning tools for adults. Furthermore, recreating objects using blocks requires advanced design skills, making it difficult for average users. Therefore, there is a demand for technology that offers new ways to enjoy block toys and new learning benefits. [Means for solving the problem]

[0005] To solve the above problems, the present invention provides the following means. First, a terminal equipped with a camera is used to capture 360-degree images of an object. Next, the terminal sends multiple images captured by the camera to a server. The server analyzes the received images and generates a 3D model of the object. Based on the generated 3D model, the server then selects the optimal block parts from a past database and generates an assembly diagram. Finally, the server sends the generated assembly diagram to the terminal, which then displays detailed assembly steps to the user. This allows the user to easily assemble the blocks and recreate the object. The present invention can be used not only as an educational toy but also as a learning tool for adults.

[0006] A "terminal" is an electronic device equipped with a camera for capturing 360° images of an object.

[0007] A "server" is a computer system that receives multiple images sent from a terminal, analyzes them, and generates a three-dimensional model of the object.

[0008] A "three-dimensional model" is digital data that represents the shape and structure of an object in three-dimensional space.

[0009] "Block parts" are multiple building block-like parts used for assembly.

[0010] An "assembly drawing" is a drawing that shows the steps and arrangement for recreating an object using block parts based on a three-dimensional model.

[0011] A "user" is a person who operates a terminal and performs photography and assembly work.

[0012] An "algorithm" is a calculation procedure that the server uses to select the optimal block parts from a past database based on a three-dimensional model.

[0013] The "database" is a digital storage device that stores information about past block part selections.

[0014] "Animation" is a visual representation that dynamically shows each step of an assembly drawing.

[0015] A "three-dimensional view" is a visual representation that displays an assembly drawing three-dimensionally, allowing the user to understand it in a three-dimensional manner. [Brief explanation of the drawings]

[0016] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13]FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0017] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0018] First, the terms used in the following description will be explained.

[0019] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0020] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0021] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0022] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0023] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0024] [First embodiment]

[0025] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0026] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0027] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0028] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0029] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0030] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0031] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0032] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0033] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0034] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0035] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0036] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0037] This invention is a system that aims to capture 360° images of an object, analyze the images, and recreate the object using block parts. Specific embodiments of this system are described in detail below.

[0038] System configuration and operation procedures

[0039] The system consists of three main components: terminals, servers and users.

[0040] 1. Terminal

[0041] The device is an electronic device such as a smartphone or tablet, equipped with a camera capable of 360° photography. The user first starts up the device and launches the application. Next, they operate the camera according to the application's instructions to photograph the object from multiple angles. The device saves the captured images in real time and sends the image data to a server once the capture is complete.

[0042] 2. Server

[0043] The server receives multiple images sent from the device and analyzes them to generate a 3D model of the object. Specifically, the server's AI analysis engine processes the images and analyzes the object's shape and structure. It then selects the optimal block parts from a past database and generates a detailed assembly drawing. Once the assembly drawing is complete, the server sends the data back to the device.

[0044] 3. Users

[0045] The user operates the terminal to display the assembly diagram sent and assemble the object using the block parts. The assembly diagram includes animations and three-dimensional views that show each step in detail, making it easy for the user to understand. The user completes the steps according to the assembly progress and finally recreates the object.

[0046] Program processing and specific examples

[0047] 1. Initialization and configuration

[0048] The user launches the app on their smartphone and grants permission to use the camera. The device then performs the necessary initial setup and displays the main screen.

[0049] 2. Photographing the subject

[0050] The user takes photos of an object, such as a coffee cup, from multiple angles, and the device checks the quality of the images in real time, moving on to the next step once the required number of photos have been taken.

[0051] 3. Image analysis and data transmission

[0052] The device sends the captured images to a server, which receives them and analyzes them to generate a 3D model.

[0053] 4. Block part selection and assembly drawing generation

[0054] Based on the analysis results, the server selects the optimal block parts from the database and generates detailed assembly drawings, which are then sent to the terminal.

[0055] 5. Displaying assembly diagrams and assembly

[0056] The terminal displays the received assembly diagram to the user, showing each step in detail. The user assembles the block parts according to the assembly diagram and recreates the object.

[0057] Specific examples

[0058] For example, if a user wants to recreate a coffee cup at home, he or she would follow the steps below.

[0059] 1. The user launches the app and takes photos of a coffee cup from multiple angles.

[0060] 2. The device sends the captured image to the server.

[0061] 3. The server analyzes the image and generates a 3D model of the coffee cup.

[0062] 4. The server references the past database, selects the optimal block parts, and generates an assembly drawing.

[0063] 5. The assembly drawing is sent to the terminal, and the user recreates the coffee cup using the block parts according to the assembly drawing.

[0064] Thus, the present invention provides a system that allows users to easily recreate objects in their home using blocks.

[0065] The processing flow will be explained below.

[0066] Step 1:

[0067] The user launches the application. The device performs an initial setup check and asks the user for permission to access the camera.

[0068] Step 2:

[0069] The device confirms camera access permission and displays the main screen, where the user selects the option to take a photo of the object.

[0070] Step 3:

[0071] The user takes photos of an object from multiple angles, and the device displays guidelines during the photo shoot to help ensure each angle is captured accurately.

[0072] Step 4:

[0073] The device saves the images in real time, checks the image quality (resolution, brightness, focus, etc.), and displays a notification that the image has been taken once the minimum number of images have been taken.

[0074] Step 5:

[0075] The device compresses the captured image and sends it to the server. The progress of the sending process is displayed to the user in real time and a notification is sent when it is complete.

[0076] Step 6:

[0077] The server receives the image data and checks its consistency. If there are no problems, it starts image analysis.

[0078] Step 7:

[0079] The server's AI analysis engine analyzes the images and recognizes the shape and structure of the object, then builds a 3D model based on the generated data.

[0080] Step 8:

[0081] The server runs an algorithm to select the optimal block parts from a database based on the 3D model, and generates an assembly drawing using the selected parts.

[0082] Step 9:

[0083] The server sends the assembly drawing to the terminal. The terminal checks the progress of the sending process and notifies the user when it is complete.

[0084] Step 10:

[0085] The terminal unpacks the received assembly drawing and checks its consistency. If there are no problems, the assembly drawing is displayed to the user.

[0086] Step 11:

[0087] The user assembles the block parts according to the assembly diagram. A progress check button is provided on the terminal to check the progress of each step.

[0088] Step 12:

[0089] As the user completes each step, they press a progress check button and the terminal displays the next assembly step.

[0090] Step 13:

[0091] After the device is finished assembling, it will display a pop-up asking the user for feedback, which will help improve the algorithm for the next time.

[0092] In this way, each step works in conjunction with the entire system, allowing the user to easily recreate an object using blocks.

[0093] Example 1

[0094] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0095] Conventional methods require specialized knowledge and skills to recreate an object using block parts, making them difficult for average users. Furthermore, generating a 3D model from images taken from multiple angles, selecting the block parts, and creating an assembly diagram all take time. The objective of this invention is to solve these problems and provide a system that allows users to easily recreate an object using block parts.

[0096] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0097] In this invention, the server includes a photographing means for allowing a user to photograph an object in 360°, an analysis means for receiving and analyzing multiple images photographed by the photographing means to generate a 3D model of the object, and a generation means for selecting block parts for reproducing the object based on the 3D model, generating an assembly drawing, and presenting the drawing to the user. This enables a user to easily reproduce an object using block parts without requiring specialized knowledge.

[0098] "Photographing means" refers to a device such as a camera or smartphone that allows a user to take 360-degree photographs of an object.

[0099] The "analysis means" refers to a device or system that receives multiple captured images, analyzes them, and generates a three-dimensional model of the object.

[0100] "Generation means" refers to a device or system that selects block parts to reproduce an object based on a three-dimensional model, generates an assembly drawing, and presents it to the user.

[0101] The "assembly means" is a system having operations for a user to assemble block parts according to an assembly drawing and a function to support this.

[0102] A "block part" is an individual part used to recreate an object based on a three-dimensional model.

[0103] An "assembly diagram" is a detailed drawing or animation that shows the user how to assemble building blocks in the correct order.

[0104] A "three-dimensional model" is data that digitizes the shape and structure of an object in three dimensions.

[0105] An "algorithm" is a set of computational steps or processes used by an analyzing means or generating means in its operation.

[0106] "Animation" refers to moving images or videos that detail each step in the user's assembly of building blocks.

[0107] A "3D view" is a visual representation of an object displayed in three dimensions, allowing a user to see the object from all directions.

[0108] This invention is a system that allows users to take 360° photographs of an object, analyze the photographed images, and recreate the object using block parts. It is designed to automatically perform everything from photographing the object to analyzing it, selecting block parts, and generating assembly drawings, allowing users to easily recreate the object.

[0109] System configuration

[0110] The system consists of three main components: terminals, servers, and users.

[0111] Terminal

[0112] The device is a smartphone, tablet, or other device equipped with a camera capable of 360° photography. The user operates the device to photograph an object from multiple angles. The application is installed, and after the necessary initial settings are made, the main screen is displayed.

[0113] server

[0114] The server is equipped with an AI analysis engine that receives multiple images sent from the device, analyzes them, and generates a 3D model of the object. Based on the generated 3D model, it selects the optimal block parts from a database and generates a detailed assembly drawing.

[0115] User

[0116] The user operates the device and assembles the blocks while viewing the assembly diagram sent to them. The assembly diagram is shown in detail, step by step, and provides animations and three-dimensional views, making it easy for the user to understand.

[0117] Example

[0118] Initialization and Configuration

[0119] The user launches the app on their smartphone and grants permission to use the camera. The device then performs the necessary initial setup and displays the main screen.

[0120] Photographing the subject

[0121] Users take photos of an object, such as a coffee cup, from multiple angles. The application displays a shooting guide and instructs the user on the optimal angle. The captured images are saved to the device in real time.

[0122] Image analysis and data transmission

[0123] The device sends the stored images to a server, which receives them and uses an AI analysis engine to generate a 3D model.

[0124] Block part selection and assembly drawing generation

[0125] The server selects the optimal block parts from a database based on the 3D model and generates a detailed assembly drawing, which is then sent to the terminal.

[0126] Displaying assembly drawings and assembly

[0127] Users assemble the blocks while viewing the assembly diagram displayed on their device. The assembly diagram includes animations and 3D views, and each step is shown in detail, making it easy for users to understand and proceed with the assembly.

[0128] Specific examples

[0129] For example, if a user wants to recreate a coffee cup they have at home, they can follow these steps:

[0130] 1. The user launches the app and takes photos of a coffee cup from multiple angles.

[0131] 2. The device sends the captured image to the server.

[0132] 3. The server analyzes the image and generates a 3D model of the coffee cup.

[0133] 4. The server references the database, selects the optimal block parts, and generates an assembly drawing.

[0134] 5. The assembly drawing is sent to the terminal, and the user recreates the coffee cup using the block parts according to the assembly drawing.

[0135] Prompt Sentence Examples

[0136] "This system takes 360-degree photographs of an object, sends the images to a server, generates a 3D model, and recreates it using block components. Please explain in detail the steps, including how the user operates the camera, how the server processes the images, and how the assembly drawing is generated."

[0137] In this way, this system is designed to enable users to easily recreate objects using block parts. By using an AI analysis engine, it is possible to quickly and accurately generate 3D models and select block parts, significantly reducing the burden on users.

[0138] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0139] Step 1: Initialization and Configuration

[0140] Terminal

[0141] Input: A user launches an app on their smartphone.

[0142] What it does: The application displays a popup requesting camera permission.

[0143] Output: If the user grants permission, the application performs some initial setup and displays the main screen.

[0144] Step 2: Photograph the object

[0145] User

[0146] Input: A user takes photos of an object, such as a coffee cup, from multiple angles.

[0147] What it does: The application displays a shooting guide and instructs the user on the best shooting angle.

[0148] Output: The captured image is saved on the device.

[0149] Terminal

[0150] Input: Multiple captured images.

[0151] How it works: The device's camera checks the quality of the images in real time, and once the required number of images are collected, it moves on to the next step.

[0152] Output: High quality image data in the required quantity.

[0153] Step 3: Image analysis and data transmission

[0154] Terminal

[0155] Input: Multiple high-quality stored image data.

[0156] Operation: Image data is compressed and encrypted before being sent to the server.

[0157] Output: The transmitted image data.

[0158] server

[0159] Input: Image data sent from the device.

[0160] Operation: The server passes the image data to the AI ​​analysis engine, which then starts the process of generating a 3D model. An image recognition algorithm using deep learning technology is then put into operation.

[0161] Output: The generated 3D model.

[0162] Step 4: Selecting block parts and generating assembly drawings

[0163] server

[0164] Input: The generated 3D model.

[0165] Operation: The server selects the optimal block parts from the database based on the analysis results of the 3D model. A detailed assembly drawing is generated based on the selected block parts.

[0166] Output: Assembly drawing data.

[0167] Step 5: Send and view assembly drawings

[0168] server

[0169] Input: Assembly drawing data.

[0170] Operation: The server sends assembly drawing data to the terminal.

[0171] Output: The assembly drawing data sent.

[0172] Terminal

[0173] Input: Submitted assembly drawing data.

[0174] How it works: The device displays an assembly diagram to the user, including animations and 3D views to help the user understand each step in detail.

[0175] Output: Visually detailed assembly drawings.

[0176] Step 6: Assembling the Block Components

[0177] User

[0178] Input: A visually detailed assembly diagram displayed on a terminal.

[0179] Operation: The user assembles the blocks according to the assembly diagram. The user manipulates the blocks at hand and follows the instructions to assemble the blocks.

[0180] Output: The completed assembly.

[0181] In this way, the system works by having the user take a 360-degree photograph of the object, the server analyzes the image and generates a three-dimensional model, then selects the most suitable block parts and generates an assembly drawing, and finally the user recreates the object using the block parts.

[0182] (Application example 1)

[0183] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0184] In modern brick-and-mortar stores, users often have difficulty purchasing or using products because they are not provided with sufficient information to understand how to assemble or recreate the product. Furthermore, when users recreate products using block parts, the assembly process is complex and time-consuming, resulting in a poor user experience. Therefore, there is a need for a system that allows users to easily understand and recreate products.

[0185] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0186] In this invention, the server includes an information terminal equipped with a camera for capturing 360-degree images of an object, an information processing device that receives and analyzes multiple images captured by the camera to generate a 3D shape model of the object, an information processing device that selects components for reproducing the object based on the 3D shape model, generates an assembly drawing and transmits it to the information terminal, and a user who assembles the components according to the assembly drawing displayed on the information terminal. This enables support for users to easily reproduce and use products in physical stores.

[0187] An "information terminal" is an electronic device operated by a user, and is a terminal equipped with a camera capable of 360° photography.

[0188] A "photography device" is a device that is built into or connected to an information terminal and is used to take 360-degree photographs of an object.

[0189] An "information processing device" is a device that receives image data sent from an information terminal, analyzes it, generates a three-dimensional shape model, and creates an assembly drawing.

[0190] A "three-dimensional shape model" is a model that reproduces the three-dimensional structure of an object based on multiple images taken by a photographing device.

[0191] A "component" is a part used to reproduce an object based on a three-dimensional shape model.

[0192] An "assembly drawing" is information that includes detailed drawings and instructions for a user to recreate an object using component parts.

[0193] A "user" is a person who operates an information terminal and assembles components according to the displayed assembly diagram.

[0194] "Brick and Mortar Store" means a physical store that a User can visit in person, where Products are displayed and sold.

[0195] The "database" is an information system that stores and manages three-dimensional shape models and the information necessary for selecting optimal component parts.

[0196] An "algorithm" refers to a calculation method or processing procedure for selecting optimal components from a database based on a three-dimensional shape model.

[0197] "Animated display" is a display method that includes animations and visual effects to visually explain assembly drawings to users.

[0198] "Three-dimensional view display" is a display method that displays assembly drawings from a three-dimensional perspective, helping users to understand them in three dimensions.

[0199] This invention is a system that takes 360° images of an object, analyzes the images, and recreates the object using block parts. Specific embodiments of this system are described in detail below.

[0200] System configuration and operation procedures

[0201] 1. Information terminal

[0202] An information terminal is an electronic device operated by a user and equipped with a camera capable of 360° photography. The user activates the device and takes photos of an object from multiple angles. Examples of such devices include smartphones and tablets.

[0203] 2. Imaging equipment

[0204] The image capturing device is a device that is built into or connected to an information terminal and captures 360-degree images of an object, and is used when a user takes a picture of an object (such as furniture) in a store.

[0205] 3. Information Processing Device

[0206] The information processing device receives image data sent from the information terminal and generates a three-dimensional shape model using an AI analysis engine. Specifically, it uses image analysis software such as TENSORFLOW (registered trademark). Based on the generated model, it also selects the optimal components from a database and creates an assembly drawing. MySQL (registered trademark) is used as the database software.

[0207] 4. Database

[0208] The database is an information system that stores and manages the information necessary for selecting 3D geometric models and optimal component parts, including past analysis results and part information.

[0209] 5. Algorithms

[0210] An algorithm is a calculation method or procedure for selecting optimal components from a database based on a 3D geometric model. It is optimized using a generative AI model.

[0211] 6. Generating and displaying assembly drawings

[0212] The server generates a detailed assembly drawing based on the analysis results. This drawing includes video and 3D views to instruct the user on each step in detail. For use in brick-and-mortar stores, the system takes 360-degree photos of the product on-site, analyzes the images, and provides instructions for recreating the product using the component parts.

[0213] A natural language description of what the program does

[0214] The server first receives multiple images sent from the information terminal. Next, it analyzes the image data using an AI analysis engine (e.g., TensorFlow) and generates a 3D shape model of the object. Based on this 3D shape model, it selects the optimal components from a database (MySQL) and generates a detailed assembly drawing using an algorithm. The generated assembly drawing is then sent to the information terminal and displayed to the user. The user recreates the object by assembling the components according to the displayed assembly drawing.

[0215] Specific examples

[0216] For example, if a user wants to recreate a chair sold in an interior design shop, the following steps are performed: First, the user starts up their information terminal and takes photos of the chair from multiple angles. Next, the captured images are sent to the server. The server analyzes the images and generates a 3D shape model. After that, it references a database, selects the optimal components, and generates an assembly drawing. Finally, the assembly drawing is sent to the information terminal, and the user recreates the chair according to its contents.

[0217] Example prompts for generative AI models

[0218] Generate a 3D model of the object from 360° images, select the optimal components based on that, and generate assembly drawings.

[0219] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0220] Step 1:

[0221] The user starts up the information terminal and grants permission to access the camera. The terminal performs initial settings and displays the main screen.

[0222] Input: User actions, camera permissions

[0223] Output: Main screen of the information terminal

[0224] Specific operation: The information terminal prepares the camera and sensors and launches the application as instructed by the user.

[0225] Step 2:

[0226] A user uses an information terminal to take a 360-degree image of an object (such as a chair) in a physical store. The terminal checks the quality of the images in real time and proceeds to the next step once the required number of images have been collected.

[0227] Input: User-taken image

[0228] Output: High-quality 360° images

[0229] Specific operation: The information terminal uses an image processing algorithm to analyze the quality of captured images in real time and instructs the user to take a new image if any are insufficient.

[0230] Step 3:

[0231] The terminal transmits the captured image data to the server.

[0232] Input: Captured image data

[0233] Output: Image data received on the server side

[0234] Specific operation: The information terminal compresses and organizes the image data and sends it to a server via the Internet.

[0235] Step 4:

[0236] The server analyzes the received image data and generates a three-dimensional shape model using an AI analysis engine (TensorFlow).

[0237] Input: Received image data

[0238] Output: 3D shape model

[0239] How it works: The server uses an AI analysis engine to analyze the relationships between images and build a 3D shape model. Parallel processing technology is used for the calculations.

[0240] Step 5:

[0241] The server runs an algorithm to select the optimal components from a database (MySQL) based on the generated 3D shape model.

[0242] Input: 3D shape model

[0243] Output: List of components

[0244] Specific operation: The server uses a database search algorithm to quickly search and select the optimal components for the 3D shape model.

[0245] Step 6:

[0246] The server generates detailed assembly drawings based on the selected components.

[0247] Input: List of components

[0248] Output: Assembly drawing

[0249] What it does: The server uses CAD software to generate assembly drawings of how components fit together, and creates animations and 3D views.

[0250] Step 7:

[0251] The server transmits the generated assembly drawing to the terminal, which displays it to the user.

[0252] Input: Assembly drawing data

[0253] Output: Assembly diagram displayed on an information terminal

[0254] Specific operation: The server compresses and organizes the assembly drawing data and sends it to the terminal via the Internet. The information terminal decompresses the received data and displays it on the user interface.

[0255] Step 8:

[0256] The user assembles the object using the component parts according to the displayed assembly drawing.

[0257] Input: Assembly drawing, components

[0258] Output: Finished object

[0259] Specific operation: The user manually assembles the components according to the specific assembly instructions displayed on the terminal. After completing the assembly, the user proceeds to the confirmation step via the terminal.

[0260] Prompt Sentence Examples

[0261] Generate a 3D model of the object from 360° images, select the optimal components based on that, and generate assembly drawings.

[0262] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0263] This invention is a system that aims to capture 360° images of an object, analyze the images, and recreate the object using block parts, and by combining it with an emotion engine, it aims to improve the enjoyment of the toy, its educational effects, and the user experience. Specific embodiments of this system are described in detail below.

[0264] System configuration and operation procedures

[0265] This system consists of four main components: a terminal, a server, an emotion engine, and a user.

[0266] 1. Terminal

[0267] The device is an electronic device such as a smartphone or tablet, equipped with a camera capable of 360° photography. The user first starts up the device and launches the application. Next, they operate the camera according to the application's instructions to photograph the object from multiple angles. The device saves the captured images in real time and sends the image data to a server once the capture is complete.

[0268] 2. Server

[0269] The server receives multiple images sent from the device and analyzes them to generate a 3D model of the object. Specifically, the server's AI analysis engine processes the images and analyzes the object's shape and structure. It then selects the optimal block parts from a past database and generates a detailed assembly drawing. Once the assembly drawing is complete, the server sends the data back to the device.

[0270] 3. Emotion Engine

[0271] The emotion engine has the ability to analyze the user's facial expressions in real time and recognize their emotions. The device's camera captures images of the user during assembly and sends the captured images to the emotion engine. Based on the emotion recognition results, the emotion engine dynamically changes the display content of the assembly diagram and the interactions. It also compares the user's emotion data with a past database to learn the user's preferences and tendencies.

[0272] 4. Users

[0273] The user operates the device to display the assembly diagram sent to them and assemble the object using the block parts. The assembly diagram includes animations and three-dimensional views that show each step in detail, making it easy for the user to understand. The user completes the steps according to the assembly progress and finally recreates the object. The system is equipped with an emotion engine, so the system provides appropriate feedback and assistance depending on the user's facial expressions and emotions.

[0274] Program processing and specific examples

[0275] 1. Initialization and configuration

[0276] The user launches the app on their smartphone and grants permission to access the camera and the emotion engine. The device then performs the necessary initial settings and displays the main screen.

[0277] 2. Photographing the subject

[0278] The user takes photos of an object, such as a coffee cup, from multiple angles, and the device checks the quality of the images in real time, moving on to the next step once the required number of photos have been taken.

[0279] 3. Image analysis and data transmission

[0280] The device sends the captured images to a server, which receives them and analyzes them to generate a 3D model.

[0281] 4. Block part selection and assembly drawing generation

[0282] Based on the analysis results, the server selects the optimal block parts from the database and generates detailed assembly drawings, which are then sent to the terminal.

[0283] 5. Displaying assembly diagrams and assembly

[0284] The terminal displays the received assembly diagram to the user, showing each step in detail. The user assembles the block parts according to the assembly diagram and recreates the object.

[0285] 6. Emotion Recognition and Feedback

[0286] The device's camera captures the user's facial expressions during assembly and sends the image data to the emotion engine, which analyzes the user's emotions and provides appropriate feedback and assistance in real time. For example, if the user is confused, the system will display detailed animations to provide additional explanations.

[0287] Specific examples

[0288] For example, if a user wants to recreate a coffee cup at home, he or she would follow the steps below.

[0289] 1. The user launches the app and takes photos of a coffee cup from multiple angles.

[0290] 2. The device sends the captured image to the server.

[0291] 3. The server analyzes the image and generates a 3D model of the coffee cup.

[0292] 4. The server references the past database, selects the optimal block parts, and generates an assembly drawing.

[0293] 5. The assembly drawing is sent to the terminal, and the user recreates the coffee cup using the block parts according to the assembly drawing.

[0294] 6. During the assembly process, the device's camera captures the user's facial expressions, and the emotion engine analyzes the user's emotions and provides appropriate feedback. For example, if the user is happy, the system will add praise or animation effects.

[0295] In this way, the present invention not only provides a system that allows users to easily recreate objects at home using blocks, but also has the effect of improving the user experience through the emotion engine.

[0296] The processing flow will be explained below.

[0297] Step 1:

[0298] The user launches the application. The device performs an initial setup check and asks the user for permission to use the camera and emotion engine.

[0299] Step 2:

[0300] The device checks the camera and emotion engine permissions and displays the main screen, where the user selects the option to take a photo of the object.

[0301] Step 3:

[0302] The user takes photos of an object from multiple angles, and the device displays guidelines during the photo shoot to help ensure each angle is captured accurately.

[0303] Step 4:

[0304] The device saves the captured images in real time and checks the image quality (resolution, brightness, focus, etc.). Once the minimum number of images has been collected, a notification will be displayed indicating that the capture is complete.

[0305] Step 5:

[0306] The device compresses the captured image and sends it to the server. The progress of the sending process is displayed to the user in real time and a notification is sent when it is complete.

[0307] Step 6:

[0308] The server receives the image data and checks its consistency. If there are no problems, it starts image analysis.

[0309] Step 7:

[0310] The server's AI analysis engine analyzes the images and recognizes the shape and structure of the object, then builds a 3D model based on the generated data.

[0311] Step 8:

[0312] The server runs an algorithm to select the optimal block parts from a database based on the 3D model, and generates an assembly drawing using the selected parts.

[0313] Step 9:

[0314] The server sends the assembly drawing to the terminal. The terminal checks the progress of the sending process and notifies the user when it is complete.

[0315] Step 10:

[0316] The terminal unpacks the received assembly drawing and checks its consistency. If there are no problems, the assembly drawing is displayed to the user.

[0317] Step 11:

[0318] The device's camera captures the user during assembly and sends the captured video data to the emotion engine, which analyzes the user's facial expressions in real time and recognizes their emotions.

[0319] Step 12:

[0320] Based on the analysis results of the emotion engine, the device dynamically changes the content displayed on the assembly diagram and the interaction. For example, if the user is confused, it will display a detailed animation to provide additional explanation.

[0321] Step 13:

[0322] As the user completes each step, they press a progress check button and the terminal displays the next assembly step.

[0323] Step 14:

[0324] After the device is finished assembling, it will display a pop-up asking the user for feedback, which will help improve the algorithm for the next time.

[0325] Specific examples

[0326] For example, if a user wants to recreate a coffee cup at home, he or she would follow the steps below.

[0327] 1. The user launches the app and grants permission to use the camera and emotion engine.

[0328] 2. The user takes photos of a coffee cup from multiple angles.

[0329] 3. The device sends the captured image to the server.

[0330] 4. The server analyzes the image and generates a 3D model of the coffee cup.

[0331] 5. The server selects the optimal block parts and generates an assembly drawing.

[0332] 6. The assembly diagram is sent to the terminal, and the user recreates the coffee cup using the block parts according to the assembly diagram.

[0333] 7. The device's camera captures the user's facial expressions, and the emotion engine provides appropriate feedback based on the analysis results.

[0334] This system not only allows users to easily recreate objects with blocks, but also enables a richer, more interactive experience with dynamic feedback provided by the emotion engine.

[0335] Example 2

[0336] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0337] Conventional block play and educational systems lack support for users to fully utilize their creativity in recreating objects. Furthermore, they lack appropriate feedback and support based on the user's emotions and progress, resulting in insufficient educational benefits and user experience. Furthermore, it can take time for users to select the specific steps and components needed to recreate an object, which can discourage users from learning.

[0338] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0339] In this invention, the server includes means for receiving and analyzing multiple images captured by the imaging device to generate a 3D model of the object, means for selecting optimal components from past information storage based on the 3D model, and means for transmitting the generated assembly procedure to a mobile information terminal. This allows the user to easily recreate the object, and the emotion analysis device can provide feedback and support in real time, improving the educational effect and user experience.

[0340] A "personal digital assistant" is an electronic device that can be carried by a user, and includes devices such as smartphones and tablet terminals equipped with communication and camera functions.

[0341] "Photography equipment" refers to cameras and sensors that can capture 360-degree images of an object, and are capable of capturing high-resolution images from multiple angles.

[0342] The "information processing device" refers to a device such as a server or computer that analyzes received image data and generates a three-dimensional model of the target object.

[0343] "Components" refer to the block parts and components required to reproduce an object, and are the basic units for assembling each part.

[0344] A "three-dimensional model" is digital data that represents an object in three dimensions, and is a model with a three-dimensional structure generated from multiple images.

[0345] An "assembly procedure" is an instruction or guide for recreating an object using components, including detailed instructions and animations for each step.

[0346] The "operator" refers to a user who operates a mobile information terminal, follows an assembly procedure, and assembles block parts to recreate an object.

[0347] An "emotion analysis device" is a device that captures the operator's facial expressions and analyzes their emotional state, and has the function of providing feedback and support based on emotions.

[0348] System configuration

[0349] This invention is a system that captures 360° images of an object, analyzes the images, and recreates the object using block parts. This system is composed of four main components: a mobile information terminal, an information processing device (server), an emotion analysis device, and an operator. Specific embodiments of the system are described in detail below.

[0350] Mobile information terminal

[0351] A mobile information terminal is an electronic device such as a smartphone or tablet, equipped with a camera that supports 360-degree photography. The operator first starts up the mobile information terminal and launches the application. Next, to photograph an object from multiple angles, the operator operates the camera according to the application's instructions. The captured images are saved in real time, and when the photographing is complete, the image data is sent to an information processing device.

[0352] Information processing device (server)

[0353] The information processing device receives multiple images sent from the mobile information terminal and analyzes them to generate a three-dimensional model of the object. Specifically, an AI analysis engine in the server processes the images and analyzes the shape and structure of the object. It then selects the optimal components (block parts) from a database of past information and generates detailed assembly instructions. Once the assembly instructions are complete, the information processing device sends the data back to the mobile information terminal.

[0354] Emotion analysis device

[0355] The emotion analysis device has the ability to analyze the operator's facial expressions in real time and recognize their emotions. The camera in the mobile information terminal captures images of the operator during assembly and sends the captured images to the emotion analysis device. The emotion analysis device dynamically changes the assembly procedure display and interactions based on the emotion recognition results. It also compares the operator's emotional data with a past database to learn the operator's preferences and tendencies.

[0356] operator

[0357] The operator operates the mobile information terminal to display the assembly instructions sent to them and assemble the object using the block parts. The assembly instructions include animations and 3D views that show each step in detail, making them easy for the operator to understand. The operator completes the steps according to the assembly progress and finally recreates the object. The system is equipped with an emotion analysis device, so the system provides appropriate feedback and assistance depending on the operator's facial expressions and emotions.

[0358] Specific examples

[0359] For example, if an operator wants to recreate a coffee cup at home, the operator performs the following steps.

[0360] 1. The operator launches the app and takes photos of a coffee cup from multiple angles.

[0361] 2. The mobile information terminal transmits the captured image to the information processing device.

[0362] 3. The information processing device analyzes the image and generates a three-dimensional model of the coffee cup.

[0363] 4. The information processing device refers to the past information database, selects the optimal components, and generates an assembly procedure.

[0364] 5. The assembly instructions are sent to the mobile information terminal, and the operator follows the assembly instructions to recreate the coffee cup using the block parts.

[0365] 6. During the assembly work, the mobile information terminal's camera captures the operator's facial expressions, and the emotion analyzer analyzes the operator's emotions and provides appropriate feedback. For example, if the operator is happy, the system will add praise or animation effects.

[0366] Prompt Sentence Examples

[0367] "This program should create a system in which a 360-degree image of an object (such as a coffee cup) is taken, the server generates a 3D model, selects the optimal block parts, and displays assembly instructions to the operator. In addition, please include a process in which an emotion analysis device is used to analyze the operator's facial expressions in real time and provide appropriate feedback."

[0368] In this way, the present invention not only provides a system that allows an operator to easily recreate objects at home using blocks, but also has the effect of improving the operator's experience by using an emotion analysis device.

[0369] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0370] Step 1: Initialize and configure the app

[0371] Subject: Terminal

[0372] When the user launches the app, the device displays a dialog requesting permission to access the camera and the emotion analyzer. Once the user grants these permissions, the device performs initial setup and displays the main screen, which displays a "Start Recording" button and a "Settings" button.

[0373] Input: User action (launching an app, granting permission)

[0374] Output: Display of the app's main screen (preparation before shooting begins)

[0375] Step 2: Capture a 360° shot of the object

[0376] Subject: User, Device

[0377] The user presses the "Start Capture" button and takes photos of an object (such as a coffee cup) from multiple angles. The device application then instructs the device to rotate the object 360 degrees, taking photos at each angle. The device checks the image quality in real time and proceeds to the next step once the required number of photos have been taken.

[0378] Input: User operation (taking a photo of an object)

[0379] Output: High-quality multi-angle images taken

[0380] Step 3: Sending image data

[0381] Subject: Terminal

[0382] The device sends the captured images to an information processing device (server). At this time, the device uses a compression algorithm (e.g., JPEG compression) to optimize the data size and shorten the upload time.

[0383] Input: Multiple captured image data

[0384] Output: Compressed and transmitted image data

[0385] Step 4: Image analysis and 3D model generation

[0386] Subject: Server

[0387] The server takes in the received image data and begins image processing using its AI analysis engine. First, it analyzes each image to extract the object's shape information, then integrates them to generate a 3D model. The server then integrates information from different angles of the object to build an accurate 3D model. The generated 3D model is temporarily stored in an internal database.

[0388] Input: Compressed image data

[0389] Output: Generated 3D model of the object

[0390] Step 5: Select components and generate assembly procedures

[0391] Subject: Server

[0392] The server selects the optimal building blocks (block parts) from a database based on the 3D model. It searches for parts that suit the shape and size of the object and selects the optimal set. It then generates detailed assembly instructions using the selected parts. The generated assembly instructions are sent to the terminal.

[0393] Input: 3D model

[0394] Output: Selected components and detailed assembly instructions

[0395] Step 6: Display assembly instructions and assemble

[0396] Subject: Terminal, User

[0397] The terminal receives the assembly instructions sent from the server and displays them to the user. The assembly instructions include detailed animations and 3D views, making them easy for the user to understand. The user follows the assembly instructions displayed and recreates the object using the block parts in the correct order.

[0398] Input: Assembly instructions received from the server

[0399] Output: Assembly steps displayed to the user and the actual assembly work

[0400] Step 7: Emotion Recognition and Feedback

[0401] Subject: Terminal, emotion analysis device

[0402] During assembly, the device's camera captures the user's facial expressions in real time and sends the image data to an emotion analyzer, which analyzes the images and determines the user's emotional state. If the user is confused, the emotion analyzer prompts the display of detailed animations or additional explanations. If the user is happy, compliments or animation effects are added.

[0403] Input: User's facial expression data

[0404] Output: Sentiment analysis results and feedback based on them

[0405] (Application example 2)

[0406] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0407] Conventional building systems lack feedback that takes into account the user's emotions and experiences during the process of building an object. This can lead to confusion and loss of interest during the building process, resulting in a poor experience. This is particularly problematic in situations where educational benefits and maintaining user motivation are required.

[0408] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0409] In this invention, the server includes a terminal equipped with a camera for capturing 360-degree images of an object, means for receiving and analyzing multiple images captured by the camera to generate a 3D model of the object, means for selecting block parts for reproducing the object based on the 3D model, generating an assembly diagram and transmitting the diagram to the terminal, an emotion engine for analyzing a user's emotions and providing feedback, and means for dynamically changing interactions and display content based on the emotion analysis results. This allows the user to receive appropriate feedback according to their emotions during the process of assembling the blocks, resulting in a richer experience and improved educational effectiveness.

[0410] "Object" refers to an object that is captured as an image through 360° photography.

[0411] "Camera" means the photographic equipment required to capture 360° images of an object.

[0412] "Terminal" refers to an electronic device that is equipped with a camera and performs processes such as acquiring and transmitting images.

[0413] "Image" means visual data of an object captured by a camera.

[0414] A "server" is a computer system that receives images sent from a terminal and processes and analyzes them.

[0415] A "three-dimensional model" is a three-dimensional digital model of an object generated based on image data analyzed by the server.

[0416] "Block parts" refer to parts that physically reproduce a three-dimensional model.

[0417] An "assembly drawing" is a design drawing for recreating an object using block parts.

[0418] "User" refers to the person who operates the terminal or system to photograph and assemble the object.

[0419] An "emotion engine" is a system that has the ability to analyze a user's emotions and provide feedback based on the results.

[0420] "Interaction" refers to two-way communication and action between a system and a user.

[0421] "Feedback" refers to the response or assistance provided by the system in response to the user's actions or emotions.

[0422] "Algorithm" refers to a computational procedure for solving a particular problem.

[0423] A "database" is a collection of information in which past data is systematically accumulated.

[0424] "Animation" refers to moving images that express movement or change.

[0425] "View" refers to the visual representation of a three-dimensional model.

[0426] This invention provides a system that allows users to take 360° photos of an object and use the data to recreate it using block parts. Furthermore, it aims to improve the user experience by incorporating an emotion engine that analyzes the user's emotions.

[0427] System configuration

[0428] 1. Terminal

[0429] The device is a smartphone or tablet equipped with a camera that can capture 360-degree images of the object. This device captures the object from multiple angles and sends the image data to a server. It also displays assembly diagrams and emotional feedback.

[0430] 2. Server

[0431] The server is a computer system that receives and analyzes multiple images sent from the device. Specifically, it performs the following processes:

[0432] 3D model generation: Multiple images are analyzed to generate a 3D model of the object. This process utilizes image analysis software and AI technology.

[0433] Block part selection: Based on the 3D model, an algorithm is run to select the optimal block parts from a past database.

[0434] Assembly Drawing Generation: Generate detailed assembly drawings based on selected block parts, including animations and 3D views detailing each step.

[0435] 3. Emotion Engine

[0436] The emotion engine is a system that analyzes the user's facial expressions and provides feedback based on their emotions. It captures the user's facial expressions with the device's camera and performs emotional analysis in real time.

[0437] Program processing

[0438] The process of a program in this system is as follows:

[0439] 1. Activating the device and photographing the object: The user activates their smartphone or tablet and takes a 360° photograph of the object. The device then transmits the captured image data to the server in real time.

[0440] 2. 3D model generation: The server receives the image data, analyzes it, and processes it to generate a 3D model of the object. This process uses an image processing library (e.g., OpenCV) and an AI analysis model.

[0441] 3. Block part selection and assembly drawing generation: The server uses the generated 3D model to select the optimal block parts from the database. Based on the selected block parts, it generates a detailed assembly drawing and sends it to the terminal.

[0442] 4. Displaying and assembling assembly diagrams: Users assemble objects using the building blocks at hand while referring to the assembly diagrams displayed on their devices. The assembly diagrams include animations and 3D views, and instructions are displayed in a way that is easy for users to understand.

[0443] 5. Emotion analysis and feedback: The device's camera captures the user's facial expressions during assembly and sends the footage to the emotion engine. The emotion engine analyzes the user's emotions in real time and provides appropriate feedback based on the results. For example, if the user shows a confused expression, the system will display a detailed animation to assist them.

[0444] Specific examples

[0445] For example, if a user wants to recreate a coffee cup they have at home, they might perform the following steps:

[0446] 1. Photographing an object: The user launches the smartphone app and takes photos of a coffee cup from multiple angles in 360°.

[0447] 2. Sending image data: The device sends the captured image data to the server.

[0448] 3. Model generation: The server analyzes the image and generates a 3D model of the coffee cup.

[0449] 4. Block part selection: The server selects the optimal block parts from the database and generates an assembly drawing.

[0450] 5. Displaying the assembly diagram: The assembly diagram is sent to the terminal, and the user follows the instructions to recreate the coffee cup using blocks.

[0451] 6. Providing feedback: If the user is confused, the system will display detailed animations to assist with assembly.

[0452] This application allows users to have fun recreating objects while receiving emotional feedback.

[0453] Example prompts to input to a generative AI model:

[0454] "Design an application that analyzes images of an object scanned by a user in 360° using a smartphone, generates a 3D model, displays the model in a virtual space using a head-mounted display, and uses an emotion analysis engine to provide real-time feedback based on the user's emotions."

[0455] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0456] Step 1:

[0457] The user turns on their smartphone or tablet and starts the application. Following the application's instructions, they take photos of the object from multiple angles across 360°. The device's camera takes images from each angle and temporarily stores the image data. The input is the image data of the object, and the output is the multiple temporarily stored images.

[0458] Step 2:

[0459] The device checks the quality of the multiple image data taken, and if it is found to be of sufficient quality, it sends it to the server.Specifically, it automatically detects the image resolution, brightness, etc., and if it does not meet the standards, it displays a message urging the user to take another photo.The input is temporarily saved image data, and the output is image data sent to the server.

[0460] Step 3:

[0461] The server receives image data sent from the device and generates a 3D model using an image processing library (e.g., OpenCV) and an AI analysis model. Specific operations include aligning images, extracting feature points, and generating point cloud data. The input is multiple image data, and the output is a 3D model.

[0462] Step 4:

[0463] The server runs an algorithm to select the optimal block parts from a database based on the generated 3D model. Specifically, it uses a shape recognition algorithm to analyze the characteristics of the 3D model and searches the database for corresponding block parts. The input is the 3D model, and the output is information about the selected block parts.

[0464] Step 5:

[0465] The server generates an assembly drawing based on the selected block parts. The assembly drawing includes animations and 3D views that show each step in detail. Specific operations involve using CAD software or 3D modeling tools to visually represent the block part placement and assembly steps. The input is block part information, and the output is a detailed assembly drawing.

[0466] Step 6:

[0467] The server sends the generated assembly drawing to the terminal. The terminal displays the received assembly drawing to the user, who then follows the instructions to assemble the object using the block parts. Specifically, the application that displays the assembly drawing outputs a 3D view and animation to the user's screen. The input is detailed assembly drawing data, and the output is the user's assembly work.

[0468] Step 7:

[0469] The device's camera captures the user's facial expressions in real time during assembly and sends the image data to the emotion engine. The emotion engine analyzes the user's facial expressions and returns the results to the server. The input is the user's facial expression data, and the output is the emotion analysis results.

[0470] Step 8:

[0471] The server provides appropriate feedback and assistance based on the emotion analysis results. For example, if the user is confused, the system displays detailed animations or additional explanations. Specific operations include emotion recognition using an AI model and generating interactions based on that. The input is the emotion analysis results, and the output is dynamically changing user interactions.

[0472] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0473] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0474] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0475] [Second embodiment]

[0476] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0477] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0478] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0479] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0480] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0481] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0482] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0483] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0484] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0485] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0486] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0487] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0488] This invention is a system that aims to capture 360° images of an object, analyze the images, and recreate the object using block parts. Specific embodiments of this system are described in detail below.

[0489] System configuration and operation procedures

[0490] The system consists of three main components: terminals, servers and users.

[0491] 1. Terminal

[0492] The device is an electronic device such as a smartphone or tablet, equipped with a camera capable of 360° photography. The user first starts up the device and launches the application. Next, they operate the camera according to the application's instructions to photograph the object from multiple angles. The device saves the captured images in real time and sends the image data to a server once the capture is complete.

[0493] 2. Server

[0494] The server receives multiple images sent from the device and analyzes them to generate a 3D model of the object. Specifically, the server's AI analysis engine processes the images and analyzes the object's shape and structure. It then selects the optimal block parts from a past database and generates a detailed assembly drawing. Once the assembly drawing is complete, the server sends the data back to the device.

[0495] 3. Users

[0496] The user operates the terminal to display the assembly diagram sent and assemble the object using the block parts. The assembly diagram includes animations and three-dimensional views that show each step in detail, making it easy for the user to understand. The user completes the steps according to the assembly progress and finally recreates the object.

[0497] Program processing and specific examples

[0498] 1. Initialization and configuration

[0499] The user launches the app on their smartphone and grants permission to use the camera. The device then performs the necessary initial setup and displays the main screen.

[0500] 2. Photographing the subject

[0501] The user takes photos of an object, such as a coffee cup, from multiple angles, and the device checks the quality of the images in real time, moving on to the next step once the required number of photos have been taken.

[0502] 3. Image analysis and data transmission

[0503] The device sends the captured images to a server, which receives them and analyzes them to generate a 3D model.

[0504] 4. Block part selection and assembly drawing generation

[0505] Based on the analysis results, the server selects the optimal block parts from the database and generates detailed assembly drawings, which are then sent to the terminal.

[0506] 5. Displaying assembly diagrams and assembly

[0507] The terminal displays the received assembly diagram to the user, showing each step in detail. The user assembles the block parts according to the assembly diagram and recreates the object.

[0508] Specific examples

[0509] For example, if a user wants to recreate a coffee cup at home, he or she would follow the steps below.

[0510] 1. The user launches the app and takes photos of a coffee cup from multiple angles.

[0511] 2. The device sends the captured image to the server.

[0512] 3. The server analyzes the image and generates a 3D model of the coffee cup.

[0513] 4. The server references the past database, selects the optimal block parts, and generates an assembly drawing.

[0514] 5. The assembly drawing is sent to the terminal, and the user recreates the coffee cup using the block parts according to the assembly drawing.

[0515] Thus, the present invention provides a system that allows users to easily recreate objects in their home using blocks.

[0516] The processing flow will be explained below.

[0517] Step 1:

[0518] The user launches the application. The device performs an initial setup check and asks the user for permission to access the camera.

[0519] Step 2:

[0520] The device confirms camera access permission and displays the main screen, where the user selects the option to take a photo of the object.

[0521] Step 3:

[0522] The user takes photos of an object from multiple angles, and the device displays guidelines during the photo shoot to help ensure each angle is captured accurately.

[0523] Step 4:

[0524] The device saves the images in real time, checks the image quality (resolution, brightness, focus, etc.), and displays a notification that the image has been taken once the minimum number of images have been taken.

[0525] Step 5:

[0526] The device compresses the captured image and sends it to the server. The progress of the sending process is displayed to the user in real time and a notification is sent when it is complete.

[0527] Step 6:

[0528] The server receives the image data and checks its consistency. If there are no problems, it starts image analysis.

[0529] Step 7:

[0530] The server's AI analysis engine analyzes the images and recognizes the shape and structure of the object, then builds a 3D model based on the generated data.

[0531] Step 8:

[0532] The server runs an algorithm to select the optimal block parts from a database based on the 3D model, and generates an assembly drawing using the selected parts.

[0533] Step 9:

[0534] The server sends the assembly drawing to the terminal. The terminal checks the progress of the sending process and notifies the user when it is complete.

[0535] Step 10:

[0536] The terminal unpacks the received assembly drawing and checks its consistency. If there are no problems, the assembly drawing is displayed to the user.

[0537] Step 11:

[0538] The user assembles the block parts according to the assembly diagram. A progress check button is provided on the terminal to check the progress of each step.

[0539] Step 12:

[0540] As the user completes each step, they press a progress check button and the terminal displays the next assembly step.

[0541] Step 13:

[0542] After the device is finished assembling, it will display a pop-up asking the user for feedback, which will help improve the algorithm for the next time.

[0543] In this way, each step works in conjunction with the entire system, allowing the user to easily recreate an object using blocks.

[0544] Example 1

[0545] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0546] Conventional methods require specialized knowledge and skills to recreate an object using block parts, making them difficult for average users. Furthermore, generating a 3D model from images taken from multiple angles, selecting the block parts, and creating an assembly diagram all take time. The objective of this invention is to solve these problems and provide a system that allows users to easily recreate an object using block parts.

[0547] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0548] In this invention, the server includes a photographing means for allowing a user to photograph an object in 360°, an analysis means for receiving and analyzing multiple images photographed by the photographing means to generate a 3D model of the object, and a generation means for selecting block parts for reproducing the object based on the 3D model, generating an assembly drawing, and presenting the drawing to the user. This enables a user to easily reproduce an object using block parts without requiring specialized knowledge.

[0549] "Photographing means" refers to a device such as a camera or smartphone that allows a user to take 360-degree photographs of an object.

[0550] The "analysis means" refers to a device or system that receives multiple captured images, analyzes them, and generates a three-dimensional model of the object.

[0551] "Generation means" refers to a device or system that selects block parts to reproduce an object based on a three-dimensional model, generates an assembly drawing, and presents it to the user.

[0552] The "assembly means" is a system having operations for a user to assemble block parts according to an assembly drawing and a function to support this.

[0553] A "block part" is an individual part used to recreate an object based on a three-dimensional model.

[0554] An "assembly diagram" is a detailed drawing or animation that shows the user how to assemble building blocks in the correct order.

[0555] A "three-dimensional model" is data that digitizes the shape and structure of an object in three dimensions.

[0556] An "algorithm" is a set of computational steps or processes used by an analyzing means or generating means in its operation.

[0557] "Animation" refers to moving images or videos that detail each step in the user's assembly of building blocks.

[0558] A "3D view" is a visual representation of an object displayed in three dimensions, allowing a user to see the object from all directions.

[0559] This invention is a system that allows users to take 360° photographs of an object, analyze the photographed images, and recreate the object using block parts. It is designed to automatically perform everything from photographing the object to analyzing it, selecting block parts, and generating assembly drawings, allowing users to easily recreate the object.

[0560] System configuration

[0561] The system consists of three main components: terminals, servers, and users.

[0562] Terminal

[0563] The device is a smartphone, tablet, or other device equipped with a camera capable of 360° photography. The user operates the device to photograph an object from multiple angles. The application is installed, and after the necessary initial settings are made, the main screen is displayed.

[0564] server

[0565] The server is equipped with an AI analysis engine that receives multiple images sent from the device, analyzes them, and generates a 3D model of the object. Based on the generated 3D model, it selects the optimal block parts from a database and generates a detailed assembly drawing.

[0566] User

[0567] The user operates the device and assembles the blocks while viewing the assembly diagram sent to them. The assembly diagram is shown in detail, step by step, and provides animations and three-dimensional views, making it easy for the user to understand.

[0568] Example

[0569] Initialization and Configuration

[0570] The user launches the app on their smartphone and grants permission to use the camera. The device then performs the necessary initial setup and displays the main screen.

[0571] Photographing the subject

[0572] Users take photos of an object, such as a coffee cup, from multiple angles. The application displays a shooting guide and instructs the user on the optimal angle. The captured images are saved to the device in real time.

[0573] Image analysis and data transmission

[0574] The device sends the stored images to a server, which receives them and uses an AI analysis engine to generate a 3D model.

[0575] Block part selection and assembly drawing generation

[0576] The server selects the optimal block parts from a database based on the 3D model and generates a detailed assembly drawing, which is then sent to the terminal.

[0577] Displaying assembly drawings and assembly

[0578] Users assemble the blocks while viewing the assembly diagram displayed on their device. The assembly diagram includes animations and 3D views, and each step is shown in detail, making it easy for users to understand and proceed with the assembly.

[0579] Specific examples

[0580] For example, if a user wants to recreate a coffee cup they have at home, they can follow these steps:

[0581] 1. The user launches the app and takes photos of a coffee cup from multiple angles.

[0582] 2. The device sends the captured image to the server.

[0583] 3. The server analyzes the image and generates a 3D model of the coffee cup.

[0584] 4. The server references the database, selects the optimal block parts, and generates an assembly drawing.

[0585] 5. The assembly drawing is sent to the terminal, and the user recreates the coffee cup using the block parts according to the assembly drawing.

[0586] Prompt Sentence Examples

[0587] "This system takes 360-degree photographs of an object, sends the images to a server, generates a 3D model, and recreates it using block components. Please explain in detail the steps, including how the user operates the camera, how the server processes the images, and how the assembly drawing is generated."

[0588] In this way, this system is designed to enable users to easily recreate objects using block parts. By using an AI analysis engine, it is possible to quickly and accurately generate 3D models and select block parts, significantly reducing the burden on users.

[0589] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0590] Step 1: Initialization and Configuration

[0591] Terminal

[0592] Input: A user launches an app on their smartphone.

[0593] What it does: The application displays a popup requesting camera permission.

[0594] Output: If the user grants permission, the application performs some initial setup and displays the main screen.

[0595] Step 2: Photograph the object

[0596] User

[0597] Input: A user takes photos of an object, such as a coffee cup, from multiple angles.

[0598] What it does: The application displays a shooting guide and instructs the user on the best shooting angle.

[0599] Output: The captured image is saved on the device.

[0600] Terminal

[0601] Input: Multiple captured images.

[0602] How it works: The device's camera checks the quality of the images in real time, and once the required number of images are collected, it moves on to the next step.

[0603] Output: High quality image data in the required quantity.

[0604] Step 3: Image analysis and data transmission

[0605] Terminal

[0606] Input: Multiple high-quality stored image data.

[0607] Operation: Image data is compressed and encrypted before being sent to the server.

[0608] Output: The transmitted image data.

[0609] server

[0610] Input: Image data sent from the device.

[0611] Operation: The server passes the image data to the AI ​​analysis engine, which then starts the process of generating a 3D model. An image recognition algorithm using deep learning technology is then put into operation.

[0612] Output: The generated 3D model.

[0613] Step 4: Selecting block parts and generating assembly drawings

[0614] server

[0615] Input: The generated 3D model.

[0616] Operation: The server selects the optimal block parts from the database based on the analysis results of the 3D model. A detailed assembly drawing is generated based on the selected block parts.

[0617] Output: Assembly drawing data.

[0618] Step 5: Send and view assembly drawings

[0619] server

[0620] Input: Assembly drawing data.

[0621] Operation: The server sends assembly drawing data to the terminal.

[0622] Output: The assembly drawing data sent.

[0623] Terminal

[0624] Input: Submitted assembly drawing data.

[0625] How it works: The device displays an assembly diagram to the user, including animations and 3D views to help the user understand each step in detail.

[0626] Output: Visually detailed assembly drawings.

[0627] Step 6: Assembling the Block Components

[0628] User

[0629] Input: A visually detailed assembly diagram displayed on a terminal.

[0630] Operation: The user assembles the blocks according to the assembly diagram. The user manipulates the blocks at hand and follows the instructions to assemble the blocks.

[0631] Output: The completed assembly.

[0632] In this way, the system works by having the user take a 360-degree photograph of the object, the server analyzes the image and generates a three-dimensional model, then selects the most suitable block parts and generates an assembly drawing, and finally the user recreates the object using the block parts.

[0633] (Application example 1)

[0634] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0635] In modern brick-and-mortar stores, users often have difficulty purchasing or using products because they are not provided with sufficient information to understand how to assemble or recreate the product. Furthermore, when users recreate products using block parts, the assembly process is complex and time-consuming, resulting in a poor user experience. Therefore, there is a need for a system that allows users to easily understand and recreate products.

[0636] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0637] In this invention, the server includes an information terminal equipped with a camera for capturing 360-degree images of an object, an information processing device that receives and analyzes multiple images captured by the camera to generate a 3D shape model of the object, an information processing device that selects components for reproducing the object based on the 3D shape model, generates an assembly drawing and transmits it to the information terminal, and a user who assembles the components according to the assembly drawing displayed on the information terminal. This enables support for users to easily reproduce and use products in physical stores.

[0638] An "information terminal" is an electronic device operated by a user, and is a terminal equipped with a camera capable of 360° photography.

[0639] A "photography device" is a device that is built into or connected to an information terminal and is used to take 360-degree photographs of an object.

[0640] An "information processing device" is a device that receives image data sent from an information terminal, analyzes it, generates a three-dimensional shape model, and creates an assembly drawing.

[0641] A "three-dimensional shape model" is a model that reproduces the three-dimensional structure of an object based on multiple images taken by a photographing device.

[0642] A "component" is a part used to reproduce an object based on a three-dimensional shape model.

[0643] An "assembly drawing" is information that includes detailed drawings and instructions for a user to recreate an object using component parts.

[0644] A "user" is a person who operates an information terminal and assembles components according to the displayed assembly diagram.

[0645] "Brick and Mortar Store" means a physical store that a User can visit in person, where Products are displayed and sold.

[0646] The "database" is an information system that stores and manages three-dimensional shape models and the information necessary for selecting optimal component parts.

[0647] An "algorithm" refers to a calculation method or processing procedure for selecting optimal components from a database based on a three-dimensional shape model.

[0648] "Animated display" is a display method that includes animations and visual effects to visually explain assembly drawings to users.

[0649] "Three-dimensional view display" is a display method that displays assembly drawings from a three-dimensional perspective, helping users to understand them in three dimensions.

[0650] This invention is a system that takes 360° images of an object, analyzes the images, and recreates the object using block parts. Specific embodiments of this system are described in detail below.

[0651] System configuration and operation procedures

[0652] 1. Information terminal

[0653] An information terminal is an electronic device operated by a user and equipped with a camera capable of 360° photography. The user activates the device and takes photos of an object from multiple angles. Examples of such devices include smartphones and tablets.

[0654] 2. Imaging equipment

[0655] The image capturing device is a device that is built into or connected to an information terminal and captures 360-degree images of an object, and is used when a user takes a picture of an object (such as furniture) in a store.

[0656] 3. Information Processing Device

[0657] The information processing device receives image data sent from the information terminal and generates a 3D shape model using an AI analysis engine. Specifically, it uses image analysis software such as TensorFlow. Based on the generated model, it also selects the optimal components from a database and creates an assembly drawing. MySQL is used as the database software.

[0658] 4. Database

[0659] The database is an information system that stores and manages the information necessary for selecting 3D geometric models and optimal component parts, including past analysis results and part information.

[0660] 5. Algorithms

[0661] An algorithm is a calculation method or procedure for selecting optimal components from a database based on a 3D geometric model. It is optimized using a generative AI model.

[0662] 6. Generating and displaying assembly drawings

[0663] The server generates a detailed assembly drawing based on the analysis results. This drawing includes video and 3D views to instruct the user on each step in detail. For use in brick-and-mortar stores, the system takes 360-degree photos of the product on-site, analyzes the images, and provides instructions for recreating the product using the component parts.

[0664] A natural language description of what the program does

[0665] The server first receives multiple images sent from the information terminal. Next, it analyzes the image data using an AI analysis engine (e.g., TensorFlow) and generates a 3D shape model of the object. Based on this 3D shape model, it selects the optimal components from a database (MySQL) and generates a detailed assembly drawing using an algorithm. The generated assembly drawing is then sent to the information terminal and displayed to the user. The user recreates the object by assembling the components according to the displayed assembly drawing.

[0666] Specific examples

[0667] For example, if a user wants to recreate a chair sold in an interior design shop, the following steps are performed: First, the user starts up their information terminal and takes photos of the chair from multiple angles. Next, the captured images are sent to the server. The server analyzes the images and generates a 3D shape model. After that, it references a database, selects the optimal components, and generates an assembly drawing. Finally, the assembly drawing is sent to the information terminal, and the user recreates the chair according to its contents.

[0668] Example prompts for generative AI models

[0669] Generate a 3D model of the object from 360° images, select the optimal components based on that, and generate assembly drawings.

[0670] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0671] Step 1:

[0672] The user starts up the information terminal and grants permission to access the camera. The terminal performs initial settings and displays the main screen.

[0673] Input: User actions, camera permissions

[0674] Output: Main screen of the information terminal

[0675] Specific operation: The information terminal prepares the camera and sensors and launches the application as instructed by the user.

[0676] Step 2:

[0677] A user uses an information terminal to take a 360-degree image of an object (such as a chair) in a physical store. The terminal checks the quality of the images in real time and proceeds to the next step once the required number of images have been collected.

[0678] Input: User-taken image

[0679] Output: High-quality 360° images

[0680] Specific operation: The information terminal uses an image processing algorithm to analyze the quality of captured images in real time and instructs the user to take a new image if any are insufficient.

[0681] Step 3:

[0682] The terminal transmits the captured image data to the server.

[0683] Input: Captured image data

[0684] Output: Image data received on the server side

[0685] Specific operation: The information terminal compresses and organizes the image data and sends it to a server via the Internet.

[0686] Step 4:

[0687] The server analyzes the received image data and generates a three-dimensional shape model using an AI analysis engine (TensorFlow).

[0688] Input: Received image data

[0689] Output: 3D shape model

[0690] How it works: The server uses an AI analysis engine to analyze the relationships between images and build a 3D shape model. Parallel processing technology is used for the calculations.

[0691] Step 5:

[0692] The server runs an algorithm to select the optimal components from a database (MySQL) based on the generated 3D shape model.

[0693] Input: 3D shape model

[0694] Output: List of components

[0695] Specific operation: The server uses a database search algorithm to quickly search and select the optimal components for the 3D shape model.

[0696] Step 6:

[0697] The server generates detailed assembly drawings based on the selected components.

[0698] Input: List of components

[0699] Output: Assembly drawing

[0700] What it does: The server uses CAD software to generate assembly drawings of how components fit together, and creates animations and 3D views.

[0701] Step 7:

[0702] The server transmits the generated assembly drawing to the terminal, which displays it to the user.

[0703] Input: Assembly drawing data

[0704] Output: Assembly diagram displayed on an information terminal

[0705] Specific operation: The server compresses and organizes the assembly drawing data and sends it to the terminal via the Internet. The information terminal decompresses the received data and displays it on the user interface.

[0706] Step 8:

[0707] The user assembles the object using the component parts according to the displayed assembly drawing.

[0708] Input: Assembly drawing, components

[0709] Output: Finished object

[0710] Specific operation: The user manually assembles the components according to the specific assembly instructions displayed on the terminal. After completing the assembly, the user proceeds to the confirmation step via the terminal.

[0711] Prompt Sentence Examples

[0712] Generate a 3D model of the object from 360° images, select the optimal components based on that, and generate assembly drawings.

[0713] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0714] This invention is a system that aims to capture 360° images of an object, analyze the images, and recreate the object using block parts, and by combining it with an emotion engine, it aims to improve the enjoyment of the toy, its educational effects, and the user experience. Specific embodiments of this system are described in detail below.

[0715] System configuration and operation procedures

[0716] This system consists of four main components: a terminal, a server, an emotion engine, and a user.

[0717] 1. Terminal

[0718] The device is an electronic device such as a smartphone or tablet, equipped with a camera capable of 360° photography. The user first starts up the device and launches the application. Next, they operate the camera according to the application's instructions to photograph the object from multiple angles. The device saves the captured images in real time and sends the image data to a server once the capture is complete.

[0719] 2. Server

[0720] The server receives multiple images sent from the device and analyzes them to generate a 3D model of the object. Specifically, the server's AI analysis engine processes the images and analyzes the object's shape and structure. It then selects the optimal block parts from a past database and generates a detailed assembly drawing. Once the assembly drawing is complete, the server sends the data back to the device.

[0721] 3. Emotion Engine

[0722] The emotion engine has the ability to analyze the user's facial expressions in real time and recognize their emotions. The device's camera captures images of the user during assembly and sends the captured images to the emotion engine. Based on the emotion recognition results, the emotion engine dynamically changes the display content of the assembly diagram and the interactions. It also compares the user's emotion data with a past database to learn the user's preferences and tendencies.

[0723] 4. Users

[0724] The user operates the device to display the assembly diagram sent to them and assemble the object using the block parts. The assembly diagram includes animations and three-dimensional views that show each step in detail, making it easy for the user to understand. The user completes the steps according to the assembly progress and finally recreates the object. The system is equipped with an emotion engine, so the system provides appropriate feedback and assistance depending on the user's facial expressions and emotions.

[0725] Program processing and specific examples

[0726] 1. Initialization and configuration

[0727] The user launches the app on their smartphone and grants permission to access the camera and the emotion engine. The device then performs the necessary initial settings and displays the main screen.

[0728] 2. Photographing the subject

[0729] The user takes photos of an object, such as a coffee cup, from multiple angles, and the device checks the quality of the images in real time, moving on to the next step once the required number of photos have been taken.

[0730] 3. Image analysis and data transmission

[0731] The device sends the captured images to a server, which receives them and analyzes them to generate a 3D model.

[0732] 4. Block part selection and assembly drawing generation

[0733] Based on the analysis results, the server selects the optimal block parts from the database and generates detailed assembly drawings, which are then sent to the terminal.

[0734] 5. Displaying assembly diagrams and assembly

[0735] The terminal displays the received assembly diagram to the user, showing each step in detail. The user assembles the block parts according to the assembly diagram and recreates the object.

[0736] 6. Emotion Recognition and Feedback

[0737] The device's camera captures the user's facial expressions during assembly and sends the image data to the emotion engine, which analyzes the user's emotions and provides appropriate feedback and assistance in real time. For example, if the user is confused, the system will display detailed animations to provide additional explanations.

[0738] Specific examples

[0739] For example, if a user wants to recreate a coffee cup at home, he or she would follow the steps below.

[0740] 1. The user launches the app and takes photos of a coffee cup from multiple angles.

[0741] 2. The device sends the captured image to the server.

[0742] 3. The server analyzes the image and generates a 3D model of the coffee cup.

[0743] 4. The server references the past database, selects the optimal block parts, and generates an assembly drawing.

[0744] 5. The assembly drawing is sent to the terminal, and the user recreates the coffee cup using the block parts according to the assembly drawing.

[0745] 6. During the assembly process, the device's camera captures the user's facial expressions, and the emotion engine analyzes the user's emotions and provides appropriate feedback. For example, if the user is happy, the system will add praise or animation effects.

[0746] In this way, the present invention not only provides a system that allows users to easily recreate objects at home using blocks, but also has the effect of improving the user experience through the emotion engine.

[0747] The processing flow will be explained below.

[0748] Step 1:

[0749] The user launches the application. The device performs an initial setup check and asks the user for permission to use the camera and emotion engine.

[0750] Step 2:

[0751] The device checks the camera and emotion engine permissions and displays the main screen, where the user selects the option to take a photo of the object.

[0752] Step 3:

[0753] The user takes photos of an object from multiple angles, and the device displays guidelines during the photo shoot to help ensure each angle is captured accurately.

[0754] Step 4:

[0755] The device saves the captured images in real time and checks the image quality (resolution, brightness, focus, etc.). Once the minimum number of images has been collected, a notification will be displayed indicating that the capture is complete.

[0756] Step 5:

[0757] The device compresses the captured image and sends it to the server. The progress of the sending process is displayed to the user in real time and a notification is sent when it is complete.

[0758] Step 6:

[0759] The server receives the image data and checks its consistency. If there are no problems, it starts image analysis.

[0760] Step 7:

[0761] The server's AI analysis engine analyzes the images and recognizes the shape and structure of the object, then builds a 3D model based on the generated data.

[0762] Step 8:

[0763] The server runs an algorithm to select the optimal block parts from a database based on the 3D model, and generates an assembly drawing using the selected parts.

[0764] Step 9:

[0765] The server sends the assembly drawing to the terminal. The terminal checks the progress of the sending process and notifies the user when it is complete.

[0766] Step 10:

[0767] The terminal unpacks the received assembly drawing and checks its consistency. If there are no problems, the assembly drawing is displayed to the user.

[0768] Step 11:

[0769] The device's camera captures the user during assembly and sends the captured video data to the emotion engine, which analyzes the user's facial expressions in real time and recognizes their emotions.

[0770] Step 12:

[0771] Based on the analysis results of the emotion engine, the device dynamically changes the content displayed on the assembly diagram and the interaction. For example, if the user is confused, it will display a detailed animation to provide additional explanation.

[0772] Step 13:

[0773] As the user completes each step, they press a progress check button and the terminal displays the next assembly step.

[0774] Step 14:

[0775] After the device is finished assembling, it will display a pop-up asking the user for feedback, which will help improve the algorithm for the next time.

[0776] Specific examples

[0777] For example, if a user wants to recreate a coffee cup at home, he or she would follow the steps below.

[0778] 1. The user launches the app and grants permission to use the camera and emotion engine.

[0779] 2. The user takes photos of a coffee cup from multiple angles.

[0780] 3. The device sends the captured image to the server.

[0781] 4. The server analyzes the image and generates a 3D model of the coffee cup.

[0782] 5. The server selects the optimal block parts and generates an assembly drawing.

[0783] 6. The assembly diagram is sent to the terminal, and the user recreates the coffee cup using the block parts according to the assembly diagram.

[0784] 7. The device's camera captures the user's facial expressions, and the emotion engine provides appropriate feedback based on the analysis results.

[0785] This system not only allows users to easily recreate objects with blocks, but also enables a richer, more interactive experience with dynamic feedback provided by the emotion engine.

[0786] Example 2

[0787] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0788] Conventional block play and educational systems lack support for users to fully utilize their creativity in recreating objects. Furthermore, they lack appropriate feedback and support based on the user's emotions and progress, resulting in insufficient educational benefits and user experience. Furthermore, it can take time for users to select the specific steps and components needed to recreate an object, which can discourage users from learning.

[0789] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0790] In this invention, the server includes means for receiving and analyzing multiple images captured by the imaging device to generate a 3D model of the object, means for selecting optimal components from past information storage based on the 3D model, and means for transmitting the generated assembly procedure to a mobile information terminal. This allows the user to easily recreate the object, and the emotion analysis device can provide feedback and support in real time, improving the educational effect and user experience.

[0791] A "personal digital assistant" is an electronic device that can be carried by a user, and includes devices such as smartphones and tablet terminals equipped with communication and camera functions.

[0792] "Photography equipment" refers to cameras and sensors that can capture 360-degree images of an object, and are capable of capturing high-resolution images from multiple angles.

[0793] The "information processing device" refers to a device such as a server or computer that analyzes received image data and generates a three-dimensional model of the target object.

[0794] "Components" refer to the block parts and components required to reproduce an object, and are the basic units for assembling each part.

[0795] A "three-dimensional model" is digital data that represents an object in three dimensions, and is a model with a three-dimensional structure generated from multiple images.

[0796] An "assembly procedure" is an instruction or guide for recreating an object using components, including detailed instructions and animations for each step.

[0797] The "operator" refers to a user who operates a mobile information terminal, follows an assembly procedure, and assembles block parts to recreate an object.

[0798] An "emotion analysis device" is a device that captures the operator's facial expressions and analyzes their emotional state, and has the function of providing feedback and support based on emotions.

[0799] System configuration

[0800] This invention is a system that captures 360° images of an object, analyzes the images, and recreates the object using block parts. This system is composed of four main components: a mobile information terminal, an information processing device (server), an emotion analysis device, and an operator. Specific embodiments of the system are described in detail below.

[0801] Mobile information terminal

[0802] A mobile information terminal is an electronic device such as a smartphone or tablet, equipped with a camera that supports 360-degree photography. The operator first starts up the mobile information terminal and launches the application. Next, to photograph an object from multiple angles, the operator operates the camera according to the application's instructions. The captured images are saved in real time, and when the photographing is complete, the image data is sent to an information processing device.

[0803] Information processing device (server)

[0804] The information processing device receives multiple images sent from the mobile information terminal and analyzes them to generate a three-dimensional model of the object. Specifically, an AI analysis engine in the server processes the images and analyzes the shape and structure of the object. It then selects the optimal components (block parts) from a database of past information and generates detailed assembly instructions. Once the assembly instructions are complete, the information processing device sends the data back to the mobile information terminal.

[0805] Emotion analysis device

[0806] The emotion analysis device has the ability to analyze the operator's facial expressions in real time and recognize their emotions. The camera in the mobile information terminal captures images of the operator during assembly and sends the captured images to the emotion analysis device. The emotion analysis device dynamically changes the assembly procedure display and interactions based on the emotion recognition results. It also compares the operator's emotional data with a past database to learn the operator's preferences and tendencies.

[0807] operator

[0808] The operator operates the mobile information terminal to display the assembly instructions sent to them and assemble the object using the block parts. The assembly instructions include animations and 3D views that show each step in detail, making them easy for the operator to understand. The operator completes the steps according to the assembly progress and finally recreates the object. The system is equipped with an emotion analysis device, so the system provides appropriate feedback and assistance depending on the operator's facial expressions and emotions.

[0809] Specific examples

[0810] For example, if an operator wants to recreate a coffee cup at home, the operator performs the following steps.

[0811] 1. The operator launches the app and takes photos of a coffee cup from multiple angles.

[0812] 2. The mobile information terminal transmits the captured image to the information processing device.

[0813] 3. The information processing device analyzes the image and generates a three-dimensional model of the coffee cup.

[0814] 4. The information processing device refers to the past information database, selects the optimal components, and generates an assembly procedure.

[0815] 5. The assembly instructions are sent to the mobile information terminal, and the operator follows the assembly instructions to recreate the coffee cup using the block parts.

[0816] 6. During the assembly work, the mobile information terminal's camera captures the operator's facial expressions, and the emotion analyzer analyzes the operator's emotions and provides appropriate feedback. For example, if the operator is happy, the system will add praise or animation effects.

[0817] Prompt Sentence Examples

[0818] "This program should create a system in which a 360-degree image of an object (such as a coffee cup) is taken, the server generates a 3D model, selects the optimal block parts, and displays assembly instructions to the operator. In addition, please include a process in which an emotion analysis device is used to analyze the operator's facial expressions in real time and provide appropriate feedback."

[0819] In this way, the present invention not only provides a system that allows an operator to easily recreate objects at home using blocks, but also has the effect of improving the operator's experience by using an emotion analysis device.

[0820] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0821] Step 1: Initialize and configure the app

[0822] Subject: Terminal

[0823] When the user launches the app, the device displays a dialog requesting permission to access the camera and the emotion analyzer. Once the user grants these permissions, the device performs initial setup and displays the main screen, which displays a "Start Recording" button and a "Settings" button.

[0824] Input: User action (launching an app, granting permission)

[0825] Output: Display of the app's main screen (preparation before shooting begins)

[0826] Step 2: Capture a 360° shot of the object

[0827] Subject: User, Device

[0828] The user presses the "Start Capture" button and takes photos of an object (such as a coffee cup) from multiple angles. The device application then instructs the device to rotate the object 360 degrees, taking photos at each angle. The device checks the image quality in real time and proceeds to the next step once the required number of photos have been taken.

[0829] Input: User operation (taking a photo of an object)

[0830] Output: High-quality multi-angle images taken

[0831] Step 3: Sending image data

[0832] Subject: Terminal

[0833] The device sends the captured images to an information processing device (server). At this time, the device uses a compression algorithm (e.g., JPEG compression) to optimize the data size and shorten the upload time.

[0834] Input: Multiple captured image data

[0835] Output: Compressed and transmitted image data

[0836] Step 4: Image analysis and 3D model generation

[0837] Subject: Server

[0838] The server takes in the received image data and begins image processing using its AI analysis engine. First, it analyzes each image to extract the object's shape information, then integrates them to generate a 3D model. The server then integrates information from different angles of the object to build an accurate 3D model. The generated 3D model is temporarily stored in an internal database.

[0839] Input: Compressed image data

[0840] Output: Generated 3D model of the object

[0841] Step 5: Select components and generate assembly procedures

[0842] Subject: Server

[0843] The server selects the optimal building blocks (block parts) from a database based on the 3D model. It searches for parts that suit the shape and size of the object and selects the optimal set. It then generates detailed assembly instructions using the selected parts. The generated assembly instructions are sent to the terminal.

[0844] Input: 3D model

[0845] Output: Selected components and detailed assembly instructions

[0846] Step 6: Display assembly instructions and assemble

[0847] Subject: Terminal, User

[0848] The terminal receives the assembly instructions sent from the server and displays them to the user. The assembly instructions include detailed animations and 3D views, making them easy for the user to understand. The user follows the assembly instructions displayed and recreates the object using the block parts in the correct order.

[0849] Input: Assembly instructions received from the server

[0850] Output: Assembly steps displayed to the user and the actual assembly work

[0851] Step 7: Emotion Recognition and Feedback

[0852] Subject: Terminal, emotion analysis device

[0853] During assembly, the device's camera captures the user's facial expressions in real time and sends the image data to an emotion analyzer, which analyzes the images and determines the user's emotional state. If the user is confused, the emotion analyzer prompts the display of detailed animations or additional explanations. If the user is happy, compliments or animation effects are added.

[0854] Input: User's facial expression data

[0855] Output: Sentiment analysis results and feedback based on them

[0856] (Application example 2)

[0857] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0858] Conventional building systems lack feedback that takes into account the user's emotions and experiences during the process of building an object. This can lead to confusion and loss of interest during the building process, resulting in a poor experience. This is particularly problematic in situations where educational benefits and maintaining user motivation are required.

[0859] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0860] In this invention, the server includes a terminal equipped with a camera for capturing 360-degree images of an object, means for receiving and analyzing multiple images captured by the camera to generate a 3D model of the object, means for selecting block parts for reproducing the object based on the 3D model, generating an assembly diagram and transmitting the diagram to the terminal, an emotion engine for analyzing a user's emotions and providing feedback, and means for dynamically changing interactions and display content based on the emotion analysis results. This allows the user to receive appropriate feedback according to their emotions during the process of assembling the blocks, resulting in a richer experience and improved educational effectiveness.

[0861] "Object" refers to an object that is captured as an image through 360° photography.

[0862] "Camera" means the photographic equipment required to capture 360° images of an object.

[0863] "Terminal" refers to an electronic device that is equipped with a camera and performs processes such as acquiring and transmitting images.

[0864] "Image" means visual data of an object captured by a camera.

[0865] A "server" is a computer system that receives images sent from a terminal and processes and analyzes them.

[0866] A "three-dimensional model" is a three-dimensional digital model of an object generated based on image data analyzed by the server.

[0867] "Block parts" refer to parts that physically reproduce a three-dimensional model.

[0868] An "assembly drawing" is a design drawing for recreating an object using block parts.

[0869] "User" refers to the person who operates the terminal or system to photograph and assemble the object.

[0870] An "emotion engine" is a system that has the ability to analyze a user's emotions and provide feedback based on the results.

[0871] "Interaction" refers to two-way communication and action between a system and a user.

[0872] "Feedback" refers to the response or assistance provided by the system in response to the user's actions or emotions.

[0873] "Algorithm" refers to a computational procedure for solving a particular problem.

[0874] A "database" is a collection of information in which past data is systematically accumulated.

[0875] "Animation" refers to moving images that express movement or change.

[0876] "View" refers to the visual representation of a three-dimensional model.

[0877] This invention provides a system that allows users to take 360° photos of an object and use the data to recreate it using block parts. Furthermore, it aims to improve the user experience by incorporating an emotion engine that analyzes the user's emotions.

[0878] System configuration

[0879] 1. Terminal

[0880] The device is a smartphone or tablet equipped with a camera that can capture 360-degree images of the object. This device captures the object from multiple angles and sends the image data to a server. It also displays assembly diagrams and emotional feedback.

[0881] 2. Server

[0882] The server is a computer system that receives and analyzes multiple images sent from the device. Specifically, it performs the following processes:

[0883] 3D model generation: Multiple images are analyzed to generate a 3D model of the object. This process utilizes image analysis software and AI technology.

[0884] Block part selection: Based on the 3D model, an algorithm is run to select the optimal block parts from a past database.

[0885] Assembly Drawing Generation: Generate detailed assembly drawings based on selected block parts, including animations and 3D views detailing each step.

[0886] 3. Emotion Engine

[0887] The emotion engine is a system that analyzes the user's facial expressions and provides feedback based on their emotions. It captures the user's facial expressions with the device's camera and performs emotional analysis in real time.

[0888] Program processing

[0889] The process of a program in this system is as follows:

[0890] 1. Activating the device and photographing the object: The user activates their smartphone or tablet and takes a 360° photograph of the object. The device then transmits the captured image data to the server in real time.

[0891] 2. 3D model generation: The server receives the image data, analyzes it, and processes it to generate a 3D model of the object. This process uses an image processing library (e.g., OpenCV) and an AI analysis model.

[0892] 3. Block part selection and assembly drawing generation: The server uses the generated 3D model to select the optimal block parts from the database. Based on the selected block parts, it generates a detailed assembly drawing and sends it to the terminal.

[0893] 4. Displaying and assembling assembly diagrams: Users assemble objects using the building blocks at hand while referring to the assembly diagrams displayed on their devices. The assembly diagrams include animations and 3D views, and instructions are displayed in a way that is easy for users to understand.

[0894] 5. Emotion analysis and feedback: The device's camera captures the user's facial expressions during assembly and sends the footage to the emotion engine. The emotion engine analyzes the user's emotions in real time and provides appropriate feedback based on the results. For example, if the user shows a confused expression, the system will display a detailed animation to assist them.

[0895] Specific examples

[0896] For example, if a user wants to recreate a coffee cup they have at home, they might perform the following steps:

[0897] 1. Photographing an object: The user launches the smartphone app and takes photos of a coffee cup from multiple angles in 360°.

[0898] 2. Sending image data: The device sends the captured image data to the server.

[0899] 3. Model generation: The server analyzes the image and generates a 3D model of the coffee cup.

[0900] 4. Block part selection: The server selects the optimal block parts from the database and generates an assembly drawing.

[0901] 5. Displaying the assembly diagram: The assembly diagram is sent to the terminal, and the user follows the instructions to recreate the coffee cup using blocks.

[0902] 6. Providing feedback: If the user is confused, the system will display detailed animations to assist with assembly.

[0903] This application allows users to have fun recreating objects while receiving emotional feedback.

[0904] Example prompts to input to a generative AI model:

[0905] "Design an application that analyzes images of an object scanned by a user in 360° using a smartphone, generates a 3D model, displays the model in a virtual space using a head-mounted display, and uses an emotion analysis engine to provide real-time feedback based on the user's emotions."

[0906] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0907] Step 1:

[0908] The user turns on their smartphone or tablet and starts the application. Following the application's instructions, they take photos of the object from multiple angles across 360°. The device's camera takes images from each angle and temporarily stores the image data. The input is the image data of the object, and the output is the multiple temporarily stored images.

[0909] Step 2:

[0910] The device checks the quality of the multiple image data taken, and if it is found to be of sufficient quality, it sends it to the server.Specifically, it automatically detects the image resolution, brightness, etc., and if it does not meet the standards, it displays a message urging the user to take another photo.The input is temporarily saved image data, and the output is image data sent to the server.

[0911] Step 3:

[0912] The server receives image data sent from the device and generates a 3D model using an image processing library (e.g., OpenCV) and an AI analysis model. Specific operations include aligning images, extracting feature points, and generating point cloud data. The input is multiple image data, and the output is a 3D model.

[0913] Step 4:

[0914] The server runs an algorithm to select the optimal block parts from a database based on the generated 3D model. Specifically, it uses a shape recognition algorithm to analyze the characteristics of the 3D model and searches the database for corresponding block parts. The input is the 3D model, and the output is information about the selected block parts.

[0915] Step 5:

[0916] The server generates an assembly drawing based on the selected block parts. The assembly drawing includes animations and 3D views that show each step in detail. Specific operations involve using CAD software or 3D modeling tools to visually represent the block part placement and assembly steps. The input is block part information, and the output is a detailed assembly drawing.

[0917] Step 6:

[0918] The server sends the generated assembly drawing to the terminal. The terminal displays the received assembly drawing to the user, who then follows the instructions to assemble the object using the block parts. Specifically, the application that displays the assembly drawing outputs a 3D view and animation to the user's screen. The input is detailed assembly drawing data, and the output is the user's assembly work.

[0919] Step 7:

[0920] The device's camera captures the user's facial expressions in real time during assembly and sends the image data to the emotion engine. The emotion engine analyzes the user's facial expressions and returns the results to the server. The input is the user's facial expression data, and the output is the emotion analysis results.

[0921] Step 8:

[0922] The server provides appropriate feedback and assistance based on the emotion analysis results. For example, if the user is confused, the system displays detailed animations or additional explanations. Specific operations include emotion recognition using an AI model and generating interactions based on that. The input is the emotion analysis results, and the output is dynamically changing user interactions.

[0923] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0924] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0925] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0926] [Third embodiment]

[0927] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0928] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0929] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0930] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0931] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0932] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0933] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0934] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0935] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0936] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0937] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0938] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0939] This invention is a system that aims to capture 360° images of an object, analyze the images, and recreate the object using block parts. Specific embodiments of this system are described in detail below.

[0940] System configuration and operation procedures

[0941] The system consists of three main components: terminals, servers and users.

[0942] 1. Terminal

[0943] The device is an electronic device such as a smartphone or tablet, equipped with a camera capable of 360° photography. The user first starts up the device and launches the application. Next, they operate the camera according to the application's instructions to photograph the object from multiple angles. The device saves the captured images in real time and sends the image data to a server once the capture is complete.

[0944] 2. Server

[0945] The server receives multiple images sent from the device and analyzes them to generate a 3D model of the object. Specifically, the server's AI analysis engine processes the images and analyzes the object's shape and structure. It then selects the optimal block parts from a past database and generates a detailed assembly drawing. Once the assembly drawing is complete, the server sends the data back to the device.

[0946] 3. Users

[0947] The user operates the terminal to display the assembly diagram sent and assemble the object using the block parts. The assembly diagram includes animations and three-dimensional views that show each step in detail, making it easy for the user to understand. The user completes the steps according to the assembly progress and finally recreates the object.

[0948] Program processing and specific examples

[0949] 1. Initialization and configuration

[0950] The user launches the app on their smartphone and grants permission to use the camera. The device then performs the necessary initial setup and displays the main screen.

[0951] 2. Photographing the subject

[0952] The user takes photos of an object, such as a coffee cup, from multiple angles, and the device checks the quality of the images in real time, moving on to the next step once the required number of photos have been taken.

[0953] 3. Image analysis and data transmission

[0954] The device sends the captured images to a server, which receives them and analyzes them to generate a 3D model.

[0955] 4. Block part selection and assembly drawing generation

[0956] Based on the analysis results, the server selects the optimal block parts from the database and generates detailed assembly drawings, which are then sent to the terminal.

[0957] 5. Displaying assembly diagrams and assembly

[0958] The terminal displays the received assembly diagram to the user, showing each step in detail. The user assembles the block parts according to the assembly diagram and recreates the object.

[0959] Specific examples

[0960] For example, if a user wants to recreate a coffee cup at home, he or she would follow the steps below.

[0961] 1. The user launches the app and takes photos of a coffee cup from multiple angles.

[0962] 2. The device sends the captured image to the server.

[0963] 3. The server analyzes the image and generates a 3D model of the coffee cup.

[0964] 4. The server references the past database, selects the optimal block parts, and generates an assembly drawing.

[0965] 5. The assembly drawing is sent to the terminal, and the user recreates the coffee cup using the block parts according to the assembly drawing.

[0966] Thus, the present invention provides a system that allows users to easily recreate objects in their home using blocks.

[0967] The processing flow will be explained below.

[0968] Step 1:

[0969] The user launches the application. The device performs an initial setup check and asks the user for permission to access the camera.

[0970] Step 2:

[0971] The device confirms camera access permission and displays the main screen, where the user selects the option to take a photo of the object.

[0972] Step 3:

[0973] The user takes photos of an object from multiple angles, and the device displays guidelines during the photo shoot to help ensure each angle is captured accurately.

[0974] Step 4:

[0975] The device saves the images in real time, checks the image quality (resolution, brightness, focus, etc.), and displays a notification that the image has been taken once the minimum number of images have been taken.

[0976] Step 5:

[0977] The device compresses the captured image and sends it to the server. The progress of the sending process is displayed to the user in real time and a notification is sent when it is complete.

[0978] Step 6:

[0979] The server receives the image data and checks its consistency. If there are no problems, it starts image analysis.

[0980] Step 7:

[0981] The server's AI analysis engine analyzes the images and recognizes the shape and structure of the object, then builds a 3D model based on the generated data.

[0982] Step 8:

[0983] The server runs an algorithm to select the optimal block parts from a database based on the 3D model, and generates an assembly drawing using the selected parts.

[0984] Step 9:

[0985] The server sends the assembly drawing to the terminal. The terminal checks the progress of the sending process and notifies the user when it is complete.

[0986] Step 10:

[0987] The terminal unpacks the received assembly drawing and checks its consistency. If there are no problems, the assembly drawing is displayed to the user.

[0988] Step 11:

[0989] The user assembles the block parts according to the assembly diagram. A progress check button is provided on the terminal to check the progress of each step.

[0990] Step 12:

[0991] As the user completes each step, they press a progress check button and the terminal displays the next assembly step.

[0992] Step 13:

[0993] After the device is finished assembling, it will display a pop-up asking the user for feedback, which will help improve the algorithm for the next time.

[0994] In this way, each step works in conjunction with the entire system, allowing the user to easily recreate an object using blocks.

[0995] Example 1

[0996] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0997] Conventional methods require specialized knowledge and skills to recreate an object using block parts, making them difficult for average users. Furthermore, generating a 3D model from images taken from multiple angles, selecting the block parts, and creating an assembly diagram all take time. The objective of this invention is to solve these problems and provide a system that allows users to easily recreate an object using block parts.

[0998] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0999] In this invention, the server includes a photographing means for allowing a user to photograph an object in 360°, an analysis means for receiving and analyzing multiple images photographed by the photographing means to generate a 3D model of the object, and a generation means for selecting block parts for reproducing the object based on the 3D model, generating an assembly drawing, and presenting the drawing to the user. This enables a user to easily reproduce an object using block parts without requiring specialized knowledge.

[1000] "Photographing means" refers to a device such as a camera or smartphone that allows a user to take 360-degree photographs of an object.

[1001] The "analysis means" refers to a device or system that receives multiple captured images, analyzes them, and generates a three-dimensional model of the object.

[1002] "Generation means" refers to a device or system that selects block parts to reproduce an object based on a three-dimensional model, generates an assembly drawing, and presents it to the user.

[1003] The "assembly means" is a system having operations for a user to assemble block parts according to an assembly drawing and a function to support this.

[1004] A "block part" is an individual part used to recreate an object based on a three-dimensional model.

[1005] An "assembly diagram" is a detailed drawing or animation that shows the user how to assemble building blocks in the correct order.

[1006] A "three-dimensional model" is data that digitizes the shape and structure of an object in three dimensions.

[1007] An "algorithm" is a set of computational steps or processes used by an analyzing means or generating means in its operation.

[1008] "Animation" refers to moving images or videos that detail each step in the user's assembly of building blocks.

[1009] A "3D view" is a visual representation of an object displayed in three dimensions, allowing a user to see the object from all directions.

[1010] This invention is a system that allows users to take 360° photographs of an object, analyze the photographed images, and recreate the object using block parts. It is designed to automatically perform everything from photographing the object to analyzing it, selecting block parts, and generating assembly drawings, allowing users to easily recreate the object.

[1011] System configuration

[1012] The system consists of three main components: terminals, servers, and users.

[1013] Terminal

[1014] The device is a smartphone, tablet, or other device equipped with a camera capable of 360° photography. The user operates the device to photograph an object from multiple angles. The application is installed, and after the necessary initial settings are made, the main screen is displayed.

[1015] server

[1016] The server is equipped with an AI analysis engine that receives multiple images sent from the device, analyzes them, and generates a 3D model of the object. Based on the generated 3D model, it selects the optimal block parts from a database and generates a detailed assembly drawing.

[1017] User

[1018] The user operates the device and assembles the blocks while viewing the assembly diagram sent to them. The assembly diagram is shown in detail, step by step, and provides animations and three-dimensional views, making it easy for the user to understand.

[1019] Example

[1020] Initialization and Configuration

[1021] The user launches the app on their smartphone and grants permission to use the camera. The device then performs the necessary initial setup and displays the main screen.

[1022] Photographing the subject

[1023] Users take photos of an object, such as a coffee cup, from multiple angles. The application displays a shooting guide and instructs the user on the optimal angle. The captured images are saved to the device in real time.

[1024] Image analysis and data transmission

[1025] The device sends the stored images to a server, which receives them and uses an AI analysis engine to generate a 3D model.

[1026] Block part selection and assembly drawing generation

[1027] The server selects the optimal block parts from a database based on the 3D model and generates a detailed assembly drawing, which is then sent to the terminal.

[1028] Displaying assembly drawings and assembly

[1029] Users assemble the blocks while viewing the assembly diagram displayed on their device. The assembly diagram includes animations and 3D views, and each step is shown in detail, making it easy for users to understand and proceed with the assembly.

[1030] Specific examples

[1031] For example, if a user wants to recreate a coffee cup they have at home, they can follow these steps:

[1032] 1. The user launches the app and takes photos of a coffee cup from multiple angles.

[1033] 2. The device sends the captured image to the server.

[1034] 3. The server analyzes the image and generates a 3D model of the coffee cup.

[1035] 4. The server references the database, selects the optimal block parts, and generates an assembly drawing.

[1036] 5. The assembly drawing is sent to the terminal, and the user recreates the coffee cup using the block parts according to the assembly drawing.

[1037] Prompt Sentence Examples

[1038] "This system takes 360-degree photographs of an object, sends the images to a server, generates a 3D model, and recreates it using block components. Please explain in detail the steps, including how the user operates the camera, how the server processes the images, and how the assembly drawing is generated."

[1039] In this way, this system is designed to enable users to easily recreate objects using block parts. By using an AI analysis engine, it is possible to quickly and accurately generate 3D models and select block parts, significantly reducing the burden on users.

[1040] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1041] Step 1: Initialization and Configuration

[1042] Terminal

[1043] Input: A user launches an app on their smartphone.

[1044] What it does: The application displays a popup requesting camera permission.

[1045] Output: If the user grants permission, the application performs some initial setup and displays the main screen.

[1046] Step 2: Photograph the object

[1047] User

[1048] Input: A user takes photos of an object, such as a coffee cup, from multiple angles.

[1049] What it does: The application displays a shooting guide and instructs the user on the best shooting angle.

[1050] Output: The captured image is saved on the device.

[1051] Terminal

[1052] Input: Multiple captured images.

[1053] How it works: The device's camera checks the quality of the images in real time, and once the required number of images are collected, it moves on to the next step.

[1054] Output: High quality image data in the required quantity.

[1055] Step 3: Image analysis and data transmission

[1056] Terminal

[1057] Input: Multiple high-quality stored image data.

[1058] Operation: Image data is compressed and encrypted before being sent to the server.

[1059] Output: The transmitted image data.

[1060] server

[1061] Input: Image data sent from the device.

[1062] Operation: The server passes the image data to the AI ​​analysis engine, which then starts the process of generating a 3D model. An image recognition algorithm using deep learning technology is then put into operation.

[1063] Output: The generated 3D model.

[1064] Step 4: Selecting block parts and generating assembly drawings

[1065] server

[1066] Input: The generated 3D model.

[1067] Operation: The server selects the optimal block parts from the database based on the analysis results of the 3D model. A detailed assembly drawing is generated based on the selected block parts.

[1068] Output: Assembly drawing data.

[1069] Step 5: Send and view assembly drawings

[1070] server

[1071] Input: Assembly drawing data.

[1072] Operation: The server sends assembly drawing data to the terminal.

[1073] Output: The assembly drawing data sent.

[1074] Terminal

[1075] Input: Submitted assembly drawing data.

[1076] How it works: The device displays an assembly diagram to the user, including animations and 3D views to help the user understand each step in detail.

[1077] Output: Visually detailed assembly drawings.

[1078] Step 6: Assembling the Block Components

[1079] User

[1080] Input: A visually detailed assembly diagram displayed on a terminal.

[1081] Operation: The user assembles the blocks according to the assembly diagram. The user manipulates the blocks at hand and follows the instructions to assemble the blocks.

[1082] Output: The completed assembly.

[1083] In this way, the system works by having the user take a 360-degree photograph of the object, the server analyzes the image and generates a three-dimensional model, then selects the most suitable block parts and generates an assembly drawing, and finally the user recreates the object using the block parts.

[1084] (Application example 1)

[1085] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1086] In modern brick-and-mortar stores, users often have difficulty purchasing or using products because they are not provided with sufficient information to understand how to assemble or recreate the product. Furthermore, when users recreate products using block parts, the assembly process is complex and time-consuming, resulting in a poor user experience. Therefore, there is a need for a system that allows users to easily understand and recreate products.

[1087] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1088] In this invention, the server includes an information terminal equipped with a camera for capturing 360-degree images of an object, an information processing device that receives and analyzes multiple images captured by the camera to generate a 3D shape model of the object, an information processing device that selects components for reproducing the object based on the 3D shape model, generates an assembly drawing and transmits it to the information terminal, and a user who assembles the components according to the assembly drawing displayed on the information terminal. This enables support for users to easily reproduce and use products in physical stores.

[1089] An "information terminal" is an electronic device operated by a user, and is a terminal equipped with a camera capable of 360° photography.

[1090] A "photography device" is a device that is built into or connected to an information terminal and is used to take 360-degree photographs of an object.

[1091] An "information processing device" is a device that receives image data sent from an information terminal, analyzes it, generates a three-dimensional shape model, and creates an assembly drawing.

[1092] A "three-dimensional shape model" is a model that reproduces the three-dimensional structure of an object based on multiple images taken by a photographing device.

[1093] A "component" is a part used to reproduce an object based on a three-dimensional shape model.

[1094] An "assembly drawing" is information that includes detailed drawings and instructions for a user to recreate an object using component parts.

[1095] A "user" is a person who operates an information terminal and assembles components according to the displayed assembly diagram.

[1096] "Brick and Mortar Store" means a physical store that a User can visit in person, where Products are displayed and sold.

[1097] The "database" is an information system that stores and manages three-dimensional shape models and the information necessary for selecting optimal component parts.

[1098] An "algorithm" refers to a calculation method or processing procedure for selecting optimal components from a database based on a three-dimensional shape model.

[1099] "Animated display" is a display method that includes animations and visual effects to visually explain assembly drawings to users.

[1100] "Three-dimensional view display" is a display method that displays assembly drawings from a three-dimensional perspective, helping users to understand them in three dimensions.

[1101] This invention is a system that takes 360° images of an object, analyzes the images, and recreates the object using block parts. Specific embodiments of this system are described in detail below.

[1102] System configuration and operation procedures

[1103] 1. Information terminal

[1104] An information terminal is an electronic device operated by a user and equipped with a camera capable of 360° photography. The user activates the device and takes photos of an object from multiple angles. Examples of such devices include smartphones and tablets.

[1105] 2. Imaging equipment

[1106] The image capturing device is a device that is built into or connected to an information terminal and captures 360-degree images of an object, and is used when a user takes a picture of an object (such as furniture) in a store.

[1107] 3. Information Processing Device

[1108] The information processing device receives image data sent from the information terminal and generates a 3D shape model using an AI analysis engine. Specifically, it uses image analysis software such as TensorFlow. Based on the generated model, it also selects the optimal components from a database and creates an assembly drawing. MySQL is used as the database software.

[1109] 4. Database

[1110] The database is an information system that stores and manages the information necessary for selecting 3D geometric models and optimal component parts, including past analysis results and part information.

[1111] 5. Algorithms

[1112] An algorithm is a calculation method or procedure for selecting optimal components from a database based on a 3D geometric model. It is optimized using a generative AI model.

[1113] 6. Generating and displaying assembly drawings

[1114] The server generates a detailed assembly drawing based on the analysis results. This drawing includes video and 3D views to instruct the user on each step in detail. For use in brick-and-mortar stores, the system takes 360-degree photos of the product on-site, analyzes the images, and provides instructions for recreating the product using the component parts.

[1115] A natural language description of what the program does

[1116] The server first receives multiple images sent from the information terminal. Next, it analyzes the image data using an AI analysis engine (e.g., TensorFlow) and generates a 3D shape model of the object. Based on this 3D shape model, it selects the optimal components from a database (MySQL) and generates a detailed assembly drawing using an algorithm. The generated assembly drawing is then sent to the information terminal and displayed to the user. The user recreates the object by assembling the components according to the displayed assembly drawing.

[1117] Specific examples

[1118] For example, if a user wants to recreate a chair sold in an interior design shop, the following steps are performed: First, the user starts up their information terminal and takes photos of the chair from multiple angles. Next, the captured images are sent to the server. The server analyzes the images and generates a 3D shape model. After that, it references a database, selects the optimal components, and generates an assembly drawing. Finally, the assembly drawing is sent to the information terminal, and the user recreates the chair according to its contents.

[1119] Example prompts for generative AI models

[1120] Generate a 3D model of the object from 360° images, select the optimal components based on that, and generate assembly drawings.

[1121] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1122] Step 1:

[1123] The user starts up the information terminal and grants permission to access the camera. The terminal performs initial settings and displays the main screen.

[1124] Input: User actions, camera permissions

[1125] Output: Main screen of the information terminal

[1126] Specific operation: The information terminal prepares the camera and sensors and launches the application as instructed by the user.

[1127] Step 2:

[1128] A user uses an information terminal to take a 360-degree image of an object (such as a chair) in a physical store. The terminal checks the quality of the images in real time and proceeds to the next step once the required number of images have been collected.

[1129] Input: User-taken image

[1130] Output: High-quality 360° images

[1131] Specific operation: The information terminal uses an image processing algorithm to analyze the quality of captured images in real time and instructs the user to take a new image if any are insufficient.

[1132] Step 3:

[1133] The terminal transmits the captured image data to the server.

[1134] Input: Captured image data

[1135] Output: Image data received on the server side

[1136] Specific operation: The information terminal compresses and organizes the image data and sends it to a server via the Internet.

[1137] Step 4:

[1138] The server analyzes the received image data and generates a three-dimensional shape model using an AI analysis engine (TensorFlow).

[1139] Input: Received image data

[1140] Output: 3D shape model

[1141] How it works: The server uses an AI analysis engine to analyze the relationships between images and build a 3D shape model. Parallel processing technology is used for the calculations.

[1142] Step 5:

[1143] The server runs an algorithm to select the optimal components from a database (MySQL) based on the generated 3D shape model.

[1144] Input: 3D shape model

[1145] Output: List of components

[1146] Specific operation: The server uses a database search algorithm to quickly search and select the optimal components for the 3D shape model.

[1147] Step 6:

[1148] The server generates detailed assembly drawings based on the selected components.

[1149] Input: List of components

[1150] Output: Assembly drawing

[1151] What it does: The server uses CAD software to generate assembly drawings of how components fit together, and creates animations and 3D views.

[1152] Step 7:

[1153] The server transmits the generated assembly drawing to the terminal, which displays it to the user.

[1154] Input: Assembly drawing data

[1155] Output: Assembly diagram displayed on an information terminal

[1156] Specific operation: The server compresses and organizes the assembly drawing data and sends it to the terminal via the Internet. The information terminal decompresses the received data and displays it on the user interface.

[1157] Step 8:

[1158] The user assembles the object using the component parts according to the displayed assembly drawing.

[1159] Input: Assembly drawing, components

[1160] Output: Finished object

[1161] Specific operation: The user manually assembles the components according to the specific assembly instructions displayed on the terminal. After completing the assembly, the user proceeds to the confirmation step via the terminal.

[1162] Prompt Sentence Examples

[1163] Generate a 3D model of the object from 360° images, select the optimal components based on that, and generate assembly drawings.

[1164] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1165] This invention is a system that aims to capture 360° images of an object, analyze the images, and recreate the object using block parts, and by combining it with an emotion engine, it aims to improve the enjoyment of the toy, its educational effects, and the user experience. Specific embodiments of this system are described in detail below.

[1166] System configuration and operation procedures

[1167] This system consists of four main components: a terminal, a server, an emotion engine, and a user.

[1168] 1. Terminal

[1169] The device is an electronic device such as a smartphone or tablet, equipped with a camera capable of 360° photography. The user first starts up the device and launches the application. Next, they operate the camera according to the application's instructions to photograph the object from multiple angles. The device saves the captured images in real time and sends the image data to a server once the capture is complete.

[1170] 2. Server

[1171] The server receives multiple images sent from the device and analyzes them to generate a 3D model of the object. Specifically, the server's AI analysis engine processes the images and analyzes the object's shape and structure. It then selects the optimal block parts from a past database and generates a detailed assembly drawing. Once the assembly drawing is complete, the server sends the data back to the device.

[1172] 3. Emotion Engine

[1173] The emotion engine has the ability to analyze the user's facial expressions in real time and recognize their emotions. The device's camera captures images of the user during assembly and sends the captured images to the emotion engine. Based on the emotion recognition results, the emotion engine dynamically changes the display content of the assembly diagram and the interactions. It also compares the user's emotion data with a past database to learn the user's preferences and tendencies.

[1174] 4. Users

[1175] The user operates the device to display the assembly diagram sent to them and assemble the object using the block parts. The assembly diagram includes animations and three-dimensional views that show each step in detail, making it easy for the user to understand. The user completes the steps according to the assembly progress and finally recreates the object. The system is equipped with an emotion engine, so the system provides appropriate feedback and assistance depending on the user's facial expressions and emotions.

[1176] Program processing and specific examples

[1177] 1. Initialization and configuration

[1178] The user launches the app on their smartphone and grants permission to access the camera and the emotion engine. The device then performs the necessary initial settings and displays the main screen.

[1179] 2. Photographing the subject

[1180] The user takes photos of an object, such as a coffee cup, from multiple angles, and the device checks the quality of the images in real time, moving on to the next step once the required number of photos have been taken.

[1181] 3. Image analysis and data transmission

[1182] The device sends the captured images to a server, which receives them and analyzes them to generate a 3D model.

[1183] 4. Block part selection and assembly drawing generation

[1184] Based on the analysis results, the server selects the optimal block parts from the database and generates detailed assembly drawings, which are then sent to the terminal.

[1185] 5. Displaying assembly diagrams and assembly

[1186] The terminal displays the received assembly diagram to the user, showing each step in detail. The user assembles the block parts according to the assembly diagram and recreates the object.

[1187] 6. Emotion Recognition and Feedback

[1188] The device's camera captures the user's facial expressions during assembly and sends the image data to the emotion engine, which analyzes the user's emotions and provides appropriate feedback and assistance in real time. For example, if the user is confused, the system will display detailed animations to provide additional explanations.

[1189] Specific examples

[1190] For example, if a user wants to recreate a coffee cup at home, he or she would follow the steps below.

[1191] 1. The user launches the app and takes photos of a coffee cup from multiple angles.

[1192] 2. The device sends the captured image to the server.

[1193] 3. The server analyzes the image and generates a 3D model of the coffee cup.

[1194] 4. The server references the past database, selects the optimal block parts, and generates an assembly drawing.

[1195] 5. The assembly drawing is sent to the terminal, and the user recreates the coffee cup using the block parts according to the assembly drawing.

[1196] 6. During the assembly process, the device's camera captures the user's facial expressions, and the emotion engine analyzes the user's emotions and provides appropriate feedback. For example, if the user is happy, the system will add praise or animation effects.

[1197] In this way, the present invention not only provides a system that allows users to easily recreate objects at home using blocks, but also has the effect of improving the user experience through the emotion engine.

[1198] The processing flow will be explained below.

[1199] Step 1:

[1200] The user launches the application. The device performs an initial setup check and asks the user for permission to use the camera and emotion engine.

[1201] Step 2:

[1202] The device checks the camera and emotion engine permissions and displays the main screen, where the user selects the option to take a photo of the object.

[1203] Step 3:

[1204] The user takes photos of an object from multiple angles, and the device displays guidelines during the photo shoot to help ensure each angle is captured accurately.

[1205] Step 4:

[1206] The device saves the captured images in real time and checks the image quality (resolution, brightness, focus, etc.). Once the minimum number of images has been collected, a notification will be displayed indicating that the capture is complete.

[1207] Step 5:

[1208] The device compresses the captured image and sends it to the server. The progress of the sending process is displayed to the user in real time and a notification is sent when it is complete.

[1209] Step 6:

[1210] The server receives the image data and checks its consistency. If there are no problems, it starts image analysis.

[1211] Step 7:

[1212] The server's AI analysis engine analyzes the images and recognizes the shape and structure of the object, then builds a 3D model based on the generated data.

[1213] Step 8:

[1214] The server runs an algorithm to select the optimal block parts from a database based on the 3D model, and generates an assembly drawing using the selected parts.

[1215] Step 9:

[1216] The server sends the assembly drawing to the terminal. The terminal checks the progress of the sending process and notifies the user when it is complete.

[1217] Step 10:

[1218] The terminal unpacks the received assembly drawing and checks its consistency. If there are no problems, the assembly drawing is displayed to the user.

[1219] Step 11:

[1220] The device's camera captures the user during assembly and sends the captured video data to the emotion engine, which analyzes the user's facial expressions in real time and recognizes their emotions.

[1221] Step 12:

[1222] Based on the analysis results of the emotion engine, the device dynamically changes the content displayed on the assembly diagram and the interaction. For example, if the user is confused, it will display a detailed animation to provide additional explanation.

[1223] Step 13:

[1224] As the user completes each step, they press a progress check button and the terminal displays the next assembly step.

[1225] Step 14:

[1226] After the device is finished assembling, it will display a pop-up asking the user for feedback, which will help improve the algorithm for the next time.

[1227] Specific examples

[1228] For example, if a user wants to recreate a coffee cup at home, he or she would follow the steps below.

[1229] 1. The user launches the app and grants permission to use the camera and emotion engine.

[1230] 2. The user takes photos of a coffee cup from multiple angles.

[1231] 3. The device sends the captured image to the server.

[1232] 4. The server analyzes the image and generates a 3D model of the coffee cup.

[1233] 5. The server selects the optimal block parts and generates an assembly drawing.

[1234] 6. The assembly diagram is sent to the terminal, and the user recreates the coffee cup using the block parts according to the assembly diagram.

[1235] 7. The device's camera captures the user's facial expressions, and the emotion engine provides appropriate feedback based on the analysis results.

[1236] This system not only allows users to easily recreate objects with blocks, but also enables a richer, more interactive experience with dynamic feedback provided by the emotion engine.

[1237] Example 2

[1238] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1239] Conventional block play and educational systems lack support for users to fully utilize their creativity in recreating objects. Furthermore, they lack appropriate feedback and support based on the user's emotions and progress, resulting in insufficient educational benefits and user experience. Furthermore, it can take time for users to select the specific steps and components needed to recreate an object, which can discourage users from learning.

[1240] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1241] In this invention, the server includes means for receiving and analyzing multiple images captured by the imaging device to generate a 3D model of the object, means for selecting optimal components from past information storage based on the 3D model, and means for transmitting the generated assembly procedure to a mobile information terminal. This allows the user to easily recreate the object, and the emotion analysis device can provide feedback and support in real time, improving the educational effect and user experience.

[1242] A "personal digital assistant" is an electronic device that can be carried by a user, and includes devices such as smartphones and tablet terminals equipped with communication and camera functions.

[1243] "Photography equipment" refers to cameras and sensors that can capture 360-degree images of an object, and are capable of capturing high-resolution images from multiple angles.

[1244] The "information processing device" refers to a device such as a server or computer that analyzes received image data and generates a three-dimensional model of the target object.

[1245] "Components" refer to the block parts and components required to reproduce an object, and are the basic units for assembling each part.

[1246] A "three-dimensional model" is digital data that represents an object in three dimensions, and is a model with a three-dimensional structure generated from multiple images.

[1247] An "assembly procedure" is an instruction or guide for recreating an object using components, including detailed instructions and animations for each step.

[1248] The "operator" refers to a user who operates a mobile information terminal, follows an assembly procedure, and assembles block parts to recreate an object.

[1249] An "emotion analysis device" is a device that captures the operator's facial expressions and analyzes their emotional state, and has the function of providing feedback and support based on emotions.

[1250] System configuration

[1251] This invention is a system that captures 360° images of an object, analyzes the images, and recreates the object using block parts. This system is composed of four main components: a mobile information terminal, an information processing device (server), an emotion analysis device, and an operator. Specific embodiments of the system are described in detail below.

[1252] Mobile information terminal

[1253] A mobile information terminal is an electronic device such as a smartphone or tablet, equipped with a camera that supports 360-degree photography. The operator first starts up the mobile information terminal and launches the application. Next, to photograph an object from multiple angles, the operator operates the camera according to the application's instructions. The captured images are saved in real time, and when the photographing is complete, the image data is sent to an information processing device.

[1254] Information processing device (server)

[1255] The information processing device receives multiple images sent from the mobile information terminal and analyzes them to generate a three-dimensional model of the object. Specifically, an AI analysis engine in the server processes the images and analyzes the shape and structure of the object. It then selects the optimal components (block parts) from a database of past information and generates detailed assembly instructions. Once the assembly instructions are complete, the information processing device sends the data back to the mobile information terminal.

[1256] Emotion analysis device

[1257] The emotion analysis device has the ability to analyze the operator's facial expressions in real time and recognize their emotions. The camera in the mobile information terminal captures images of the operator during assembly and sends the captured images to the emotion analysis device. The emotion analysis device dynamically changes the assembly procedure display and interactions based on the emotion recognition results. It also compares the operator's emotional data with a past database to learn the operator's preferences and tendencies.

[1258] operator

[1259] The operator operates the mobile information terminal to display the assembly instructions sent to them and assemble the object using the block parts. The assembly instructions include animations and 3D views that show each step in detail, making them easy for the operator to understand. The operator completes the steps according to the assembly progress and finally recreates the object. The system is equipped with an emotion analysis device, so the system provides appropriate feedback and assistance depending on the operator's facial expressions and emotions.

[1260] Specific examples

[1261] For example, if an operator wants to recreate a coffee cup at home, the operator performs the following steps.

[1262] 1. The operator launches the app and takes photos of a coffee cup from multiple angles.

[1263] 2. The mobile information terminal transmits the captured image to the information processing device.

[1264] 3. The information processing device analyzes the image and generates a three-dimensional model of the coffee cup.

[1265] 4. The information processing device refers to the past information database, selects the optimal components, and generates an assembly procedure.

[1266] 5. The assembly instructions are sent to the mobile information terminal, and the operator follows the assembly instructions to recreate the coffee cup using the block parts.

[1267] 6. During the assembly work, the mobile information terminal's camera captures the operator's facial expressions, and the emotion analyzer analyzes the operator's emotions and provides appropriate feedback. For example, if the operator is happy, the system will add praise or animation effects.

[1268] Prompt Sentence Examples

[1269] "This program should create a system in which a 360-degree image of an object (such as a coffee cup) is taken, the server generates a 3D model, selects the optimal block parts, and displays assembly instructions to the operator. In addition, please include a process in which an emotion analysis device is used to analyze the operator's facial expressions in real time and provide appropriate feedback."

[1270] In this way, the present invention not only provides a system that allows an operator to easily recreate objects at home using blocks, but also has the effect of improving the operator's experience by using an emotion analysis device.

[1271] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1272] Step 1: Initialize and configure the app

[1273] Subject: Terminal

[1274] When the user launches the app, the device displays a dialog requesting permission to access the camera and the emotion analyzer. Once the user grants these permissions, the device performs initial setup and displays the main screen, which displays a "Start Recording" button and a "Settings" button.

[1275] Input: User action (launching an app, granting permission)

[1276] Output: Display of the app's main screen (preparation before shooting begins)

[1277] Step 2: Capture a 360° shot of the object

[1278] Subject: User, Device

[1279] The user presses the "Start Capture" button and takes photos of an object (such as a coffee cup) from multiple angles. The device application then instructs the device to rotate the object 360 degrees, taking photos at each angle. The device checks the image quality in real time and proceeds to the next step once the required number of photos have been taken.

[1280] Input: User operation (taking a photo of an object)

[1281] Output: High-quality multi-angle images taken

[1282] Step 3: Sending image data

[1283] Subject: Terminal

[1284] The device sends the captured images to an information processing device (server). At this time, the device uses a compression algorithm (e.g., JPEG compression) to optimize the data size and shorten the upload time.

[1285] Input: Multiple captured image data

[1286] Output: Compressed and transmitted image data

[1287] Step 4: Image analysis and 3D model generation

[1288] Subject: Server

[1289] The server takes in the received image data and begins image processing using its AI analysis engine. First, it analyzes each image to extract the object's shape information, then integrates them to generate a 3D model. The server then integrates information from different angles of the object to build an accurate 3D model. The generated 3D model is temporarily stored in an internal database.

[1290] Input: Compressed image data

[1291] Output: Generated 3D model of the object

[1292] Step 5: Select components and generate assembly procedures

[1293] Subject: Server

[1294] The server selects the optimal building blocks (block parts) from a database based on the 3D model. It searches for parts that suit the shape and size of the object and selects the optimal set. It then generates detailed assembly instructions using the selected parts. The generated assembly instructions are sent to the terminal.

[1295] Input: 3D model

[1296] Output: Selected components and detailed assembly instructions

[1297] Step 6: Display assembly instructions and assemble

[1298] Subject: Terminal, User

[1299] The terminal receives the assembly instructions sent from the server and displays them to the user. The assembly instructions include detailed animations and 3D views, making them easy for the user to understand. The user follows the assembly instructions displayed and recreates the object using the block parts in the correct order.

[1300] Input: Assembly instructions received from the server

[1301] Output: Assembly steps displayed to the user and the actual assembly work

[1302] Step 7: Emotion Recognition and Feedback

[1303] Subject: Terminal, emotion analysis device

[1304] During assembly, the device's camera captures the user's facial expressions in real time and sends the image data to an emotion analyzer, which analyzes the images and determines the user's emotional state. If the user is confused, the emotion analyzer prompts the display of detailed animations or additional explanations. If the user is happy, compliments or animation effects are added.

[1305] Input: User's facial expression data

[1306] Output: Sentiment analysis results and feedback based on them

[1307] (Application example 2)

[1308] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1309] Conventional building systems lack feedback that takes into account the user's emotions and experiences during the process of building an object. This can lead to confusion and loss of interest during the building process, resulting in a poor experience. This is particularly problematic in situations where educational benefits and maintaining user motivation are required.

[1310] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1311] In this invention, the server includes a terminal equipped with a camera for capturing 360-degree images of an object, means for receiving and analyzing multiple images captured by the camera to generate a 3D model of the object, means for selecting block parts for reproducing the object based on the 3D model, generating an assembly diagram and transmitting the diagram to the terminal, an emotion engine for analyzing a user's emotions and providing feedback, and means for dynamically changing interactions and display content based on the emotion analysis results. This allows the user to receive appropriate feedback according to their emotions during the process of assembling the blocks, resulting in a richer experience and improved educational effectiveness.

[1312] "Object" refers to an object that is captured as an image through 360° photography.

[1313] "Camera" means the photographic equipment required to capture 360° images of an object.

[1314] "Terminal" refers to an electronic device that is equipped with a camera and performs processes such as acquiring and transmitting images.

[1315] "Image" means visual data of an object captured by a camera.

[1316] A "server" is a computer system that receives images sent from a terminal and processes and analyzes them.

[1317] A "three-dimensional model" is a three-dimensional digital model of an object generated based on image data analyzed by the server.

[1318] "Block parts" refer to parts that physically reproduce a three-dimensional model.

[1319] An "assembly drawing" is a design drawing for recreating an object using block parts.

[1320] "User" refers to the person who operates the terminal or system to photograph and assemble the object.

[1321] An "emotion engine" is a system that has the ability to analyze a user's emotions and provide feedback based on the results.

[1322] "Interaction" refers to two-way communication and action between a system and a user.

[1323] "Feedback" refers to the response or assistance provided by the system in response to the user's actions or emotions.

[1324] "Algorithm" refers to a computational procedure for solving a particular problem.

[1325] A "database" is a collection of information in which past data is systematically accumulated.

[1326] "Animation" refers to moving images that express movement or change.

[1327] "View" refers to the visual representation of a three-dimensional model.

[1328] This invention provides a system that allows users to take 360° photos of an object and use the data to recreate it using block parts. Furthermore, it aims to improve the user experience by incorporating an emotion engine that analyzes the user's emotions.

[1329] System configuration

[1330] 1. Terminal

[1331] The device is a smartphone or tablet equipped with a camera that can capture 360-degree images of the object. This device captures the object from multiple angles and sends the image data to a server. It also displays assembly diagrams and emotional feedback.

[1332] 2. Server

[1333] The server is a computer system that receives and analyzes multiple images sent from the device. Specifically, it performs the following processes:

[1334] 3D model generation: Multiple images are analyzed to generate a 3D model of the object. This process utilizes image analysis software and AI technology.

[1335] Block part selection: Based on the 3D model, an algorithm is run to select the optimal block parts from a past database.

[1336] Assembly Drawing Generation: Generate detailed assembly drawings based on selected block parts, including animations and 3D views detailing each step.

[1337] 3. Emotion Engine

[1338] The emotion engine is a system that analyzes the user's facial expressions and provides feedback based on their emotions. It captures the user's facial expressions with the device's camera and performs emotional analysis in real time.

[1339] Program processing

[1340] The process of a program in this system is as follows:

[1341] 1. Activating the device and photographing the object: The user activates their smartphone or tablet and takes a 360° photograph of the object. The device then transmits the captured image data to the server in real time.

[1342] 2. 3D model generation: The server receives the image data, analyzes it, and processes it to generate a 3D model of the object. This process uses an image processing library (e.g., OpenCV) and an AI analysis model.

[1343] 3. Block part selection and assembly drawing generation: The server uses the generated 3D model to select the optimal block parts from the database. Based on the selected block parts, it generates a detailed assembly drawing and sends it to the terminal.

[1344] 4. Displaying and assembling assembly diagrams: Users assemble objects using the building blocks at hand while referring to the assembly diagrams displayed on their devices. The assembly diagrams include animations and 3D views, and instructions are displayed in a way that is easy for users to understand.

[1345] 5. Emotion analysis and feedback: The device's camera captures the user's facial expressions during assembly and sends the footage to the emotion engine. The emotion engine analyzes the user's emotions in real time and provides appropriate feedback based on the results. For example, if the user shows a confused expression, the system will display a detailed animation to assist them.

[1346] Specific examples

[1347] For example, if a user wants to recreate a coffee cup they have at home, they might perform the following steps:

[1348] 1. Photographing an object: The user launches the smartphone app and takes photos of a coffee cup from multiple angles in 360°.

[1349] 2. Sending image data: The device sends the captured image data to the server.

[1350] 3. Model generation: The server analyzes the image and generates a 3D model of the coffee cup.

[1351] 4. Block part selection: The server selects the optimal block parts from the database and generates an assembly drawing.

[1352] 5. Displaying the assembly diagram: The assembly diagram is sent to the terminal, and the user follows the instructions to recreate the coffee cup using blocks.

[1353] 6. Providing feedback: If the user is confused, the system will display detailed animations to assist with assembly.

[1354] This application allows users to have fun recreating objects while receiving emotional feedback.

[1355] Example prompts to input to a generative AI model:

[1356] "Design an application that analyzes images of an object scanned by a user in 360° using a smartphone, generates a 3D model, displays the model in a virtual space using a head-mounted display, and uses an emotion analysis engine to provide real-time feedback based on the user's emotions."

[1357] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1358] Step 1:

[1359] The user turns on their smartphone or tablet and starts the application. Following the application's instructions, they take photos of the object from multiple angles across 360°. The device's camera takes images from each angle and temporarily stores the image data. The input is the image data of the object, and the output is the multiple temporarily stored images.

[1360] Step 2:

[1361] The device checks the quality of the multiple image data taken, and if it is found to be of sufficient quality, it sends it to the server.Specifically, it automatically detects the image resolution, brightness, etc., and if it does not meet the standards, it displays a message urging the user to take another photo.The input is temporarily saved image data, and the output is image data sent to the server.

[1362] Step 3:

[1363] The server receives image data sent from the device and generates a 3D model using an image processing library (e.g., OpenCV) and an AI analysis model. Specific operations include aligning images, extracting feature points, and generating point cloud data. The input is multiple image data, and the output is a 3D model.

[1364] Step 4:

[1365] The server runs an algorithm to select the optimal block parts from a database based on the generated 3D model. Specifically, it uses a shape recognition algorithm to analyze the characteristics of the 3D model and searches the database for corresponding block parts. The input is the 3D model, and the output is information about the selected block parts.

[1366] Step 5:

[1367] The server generates an assembly drawing based on the selected block parts. The assembly drawing includes animations and 3D views that show each step in detail. Specific operations involve using CAD software or 3D modeling tools to visually represent the block part placement and assembly steps. The input is block part information, and the output is a detailed assembly drawing.

[1368] Step 6:

[1369] The server sends the generated assembly drawing to the terminal. The terminal displays the received assembly drawing to the user, who then follows the instructions to assemble the object using the block parts. Specifically, the application that displays the assembly drawing outputs a 3D view and animation to the user's screen. The input is detailed assembly drawing data, and the output is the user's assembly work.

[1370] Step 7:

[1371] The device's camera captures the user's facial expressions in real time during assembly and sends the image data to the emotion engine. The emotion engine analyzes the user's facial expressions and returns the results to the server. The input is the user's facial expression data, and the output is the emotion analysis results.

[1372] Step 8:

[1373] The server provides appropriate feedback and assistance based on the emotion analysis results. For example, if the user is confused, the system displays detailed animations or additional explanations. Specific operations include emotion recognition using an AI model and generating interactions based on that. The input is the emotion analysis results, and the output is dynamically changing user interactions.

[1374] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1375] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1376] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1377] [Fourth embodiment]

[1378] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1379] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1380] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1381] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1382] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1383] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1384] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1385] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1386] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1387] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1388] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1389] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1390] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1391] This invention is a system that aims to capture 360° images of an object, analyze the images, and recreate the object using block parts. Specific embodiments of this system are described in detail below.

[1392] System configuration and operation procedures

[1393] The system consists of three main components: terminals, servers and users.

[1394] 1. Terminal

[1395] The device is an electronic device such as a smartphone or tablet, equipped with a camera capable of 360° photography. The user first starts up the device and launches the application. Next, they operate the camera according to the application's instructions to photograph the object from multiple angles. The device saves the captured images in real time and sends the image data to a server once the capture is complete.

[1396] 2. Server

[1397] The server receives multiple images sent from the device and analyzes them to generate a 3D model of the object. Specifically, the server's AI analysis engine processes the images and analyzes the object's shape and structure. It then selects the optimal block parts from a past database and generates a detailed assembly drawing. Once the assembly drawing is complete, the server sends the data back to the device.

[1398] 3. Users

[1399] The user operates the terminal to display the assembly diagram sent and assemble the object using the block parts. The assembly diagram includes animations and three-dimensional views that show each step in detail, making it easy for the user to understand. The user completes the steps according to the assembly progress and finally recreates the object.

[1400] Program processing and specific examples

[1401] 1. Initialization and configuration

[1402] The user launches the app on their smartphone and grants permission to use the camera. The device then performs the necessary initial setup and displays the main screen.

[1403] 2. Photographing the subject

[1404] The user takes photos of an object, such as a coffee cup, from multiple angles, and the device checks the quality of the images in real time, moving on to the next step once the required number of photos have been taken.

[1405] 3. Image analysis and data transmission

[1406] The device sends the captured images to a server, which receives them and analyzes them to generate a 3D model.

[1407] 4. Block part selection and assembly drawing generation

[1408] Based on the analysis results, the server selects the optimal block parts from the database and generates detailed assembly drawings, which are then sent to the terminal.

[1409] 5. Displaying assembly diagrams and assembly

[1410] The terminal displays the received assembly diagram to the user, showing each step in detail. The user assembles the block parts according to the assembly diagram and recreates the object.

[1411] Specific examples

[1412] For example, if a user wants to recreate a coffee cup at home, he or she would follow the steps below.

[1413] 1. The user launches the app and takes photos of a coffee cup from multiple angles.

[1414] 2. The device sends the captured image to the server.

[1415] 3. The server analyzes the image and generates a 3D model of the coffee cup.

[1416] 4. The server references the past database, selects the optimal block parts, and generates an assembly drawing.

[1417] 5. The assembly drawing is sent to the terminal, and the user recreates the coffee cup using the block parts according to the assembly drawing.

[1418] Thus, the present invention provides a system that allows users to easily recreate objects in their home using blocks.

[1419] The processing flow will be explained below.

[1420] Step 1:

[1421] The user launches the application. The device performs an initial setup check and asks the user for permission to access the camera.

[1422] Step 2:

[1423] The device confirms camera access permission and displays the main screen, where the user selects the option to take a photo of the object.

[1424] Step 3:

[1425] The user takes photos of an object from multiple angles, and the device displays guidelines during the photo shoot to help ensure each angle is captured accurately.

[1426] Step 4:

[1427] The device saves the images in real time, checks the image quality (resolution, brightness, focus, etc.), and displays a notification that the image has been taken once the minimum number of images have been taken.

[1428] Step 5:

[1429] The device compresses the captured image and sends it to the server. The progress of the sending process is displayed to the user in real time and a notification is sent when it is complete.

[1430] Step 6:

[1431] The server receives the image data and checks its consistency. If there are no problems, it starts image analysis.

[1432] Step 7:

[1433] The server's AI analysis engine analyzes the images and recognizes the shape and structure of the object, then builds a 3D model based on the generated data.

[1434] Step 8:

[1435] The server runs an algorithm to select the optimal block parts from a database based on the 3D model, and generates an assembly drawing using the selected parts.

[1436] Step 9:

[1437] The server sends the assembly drawing to the terminal. The terminal checks the progress of the sending process and notifies the user when it is complete.

[1438] Step 10:

[1439] The terminal unpacks the received assembly drawing and checks its consistency. If there are no problems, the assembly drawing is displayed to the user.

[1440] Step 11:

[1441] The user assembles the block parts according to the assembly diagram. A progress check button is provided on the terminal to check the progress of each step.

[1442] Step 12:

[1443] As the user completes each step, they press a progress check button and the terminal displays the next assembly step.

[1444] Step 13:

[1445] After the device is finished assembling, it will display a pop-up asking the user for feedback, which will help improve the algorithm for the next time.

[1446] In this way, each step works in conjunction with the entire system, allowing the user to easily recreate an object using blocks.

[1447] Example 1

[1448] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1449] Conventional methods require specialized knowledge and skills to recreate an object using block parts, making them difficult for average users. Furthermore, generating a 3D model from images taken from multiple angles, selecting the block parts, and creating an assembly diagram all take time. The objective of this invention is to solve these problems and provide a system that allows users to easily recreate an object using block parts.

[1450] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1451] In this invention, the server includes a photographing means for allowing a user to photograph an object in 360°, an analysis means for receiving and analyzing multiple images photographed by the photographing means to generate a 3D model of the object, and a generation means for selecting block parts for reproducing the object based on the 3D model, generating an assembly drawing, and presenting the drawing to the user. This enables a user to easily reproduce an object using block parts without requiring specialized knowledge.

[1452] "Photographing means" refers to a device such as a camera or smartphone that allows a user to take 360-degree photographs of an object.

[1453] The "analysis means" refers to a device or system that receives multiple captured images, analyzes them, and generates a three-dimensional model of the object.

[1454] "Generation means" refers to a device or system that selects block parts to reproduce an object based on a three-dimensional model, generates an assembly drawing, and presents it to the user.

[1455] The "assembly means" is a system having operations for a user to assemble block parts according to an assembly drawing and a function to support this.

[1456] A "block part" is an individual part used to recreate an object based on a three-dimensional model.

[1457] An "assembly diagram" is a detailed drawing or animation that shows the user how to assemble building blocks in the correct order.

[1458] A "three-dimensional model" is data that digitizes the shape and structure of an object in three dimensions.

[1459] An "algorithm" is a set of computational steps or processes used by an analyzing means or generating means in its operation.

[1460] "Animation" refers to moving images or videos that detail each step in the user's assembly of building blocks.

[1461] A "3D view" is a visual representation of an object displayed in three dimensions, allowing a user to see the object from all directions.

[1462] This invention is a system that allows users to take 360° photographs of an object, analyze the photographed images, and recreate the object using block parts. It is designed to automatically perform everything from photographing the object to analyzing it, selecting block parts, and generating assembly drawings, allowing users to easily recreate the object.

[1463] System configuration

[1464] The system consists of three main components: terminals, servers, and users.

[1465] Terminal

[1466] The device is a smartphone, tablet, or other device equipped with a camera capable of 360° photography. The user operates the device to photograph an object from multiple angles. The application is installed, and after the necessary initial settings are made, the main screen is displayed.

[1467] server

[1468] The server is equipped with an AI analysis engine that receives multiple images sent from the device, analyzes them, and generates a 3D model of the object. Based on the generated 3D model, it selects the optimal block parts from a database and generates a detailed assembly drawing.

[1469] User

[1470] The user operates the device and assembles the blocks while viewing the assembly diagram sent to them. The assembly diagram is shown in detail, step by step, and provides animations and three-dimensional views, making it easy for the user to understand.

[1471] Example

[1472] Initialization and Configuration

[1473] The user launches the app on their smartphone and grants permission to use the camera. The device then performs the necessary initial setup and displays the main screen.

[1474] Photographing the subject

[1475] Users take photos of an object, such as a coffee cup, from multiple angles. The application displays a shooting guide and instructs the user on the optimal angle. The captured images are saved to the device in real time.

[1476] Image analysis and data transmission

[1477] The device sends the stored images to a server, which receives them and uses an AI analysis engine to generate a 3D model.

[1478] Block part selection and assembly drawing generation

[1479] The server selects the optimal block parts from a database based on the 3D model and generates a detailed assembly drawing, which is then sent to the terminal.

[1480] Displaying assembly drawings and assembly

[1481] Users assemble the blocks while viewing the assembly diagram displayed on their device. The assembly diagram includes animations and 3D views, and each step is shown in detail, making it easy for users to understand and proceed with the assembly.

[1482] Specific examples

[1483] For example, if a user wants to recreate a coffee cup they have at home, they can follow these steps:

[1484] 1. The user launches the app and takes photos of a coffee cup from multiple angles.

[1485] 2. The device sends the captured image to the server.

[1486] 3. The server analyzes the image and generates a 3D model of the coffee cup.

[1487] 4. The server references the database, selects the optimal block parts, and generates an assembly drawing.

[1488] 5. The assembly drawing is sent to the terminal, and the user recreates the coffee cup using the block parts according to the assembly drawing.

[1489] Prompt Sentence Examples

[1490] "This system takes 360-degree photographs of an object, sends the images to a server, generates a 3D model, and recreates it using block components. Please explain in detail the steps, including how the user operates the camera, how the server processes the images, and how the assembly drawing is generated."

[1491] In this way, this system is designed to enable users to easily recreate objects using block parts. By using an AI analysis engine, it is possible to quickly and accurately generate 3D models and select block parts, significantly reducing the burden on users.

[1492] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1493] Step 1: Initialization and Configuration

[1494] Terminal

[1495] Input: A user launches an app on their smartphone.

[1496] What it does: The application displays a popup requesting camera permission.

[1497] Output: If the user grants permission, the application performs some initial setup and displays the main screen.

[1498] Step 2: Photograph the object

[1499] User

[1500] Input: A user takes photos of an object, such as a coffee cup, from multiple angles.

[1501] What it does: The application displays a shooting guide and instructs the user on the best shooting angle.

[1502] Output: The captured image is saved on the device.

[1503] Terminal

[1504] Input: Multiple captured images.

[1505] How it works: The device's camera checks the quality of the images in real time, and once the required number of images are collected, it moves on to the next step.

[1506] Output: High quality image data in the required quantity.

[1507] Step 3: Image analysis and data transmission

[1508] Terminal

[1509] Input: Multiple high-quality stored image data.

[1510] Operation: Image data is compressed and encrypted before being sent to the server.

[1511] Output: The transmitted image data.

[1512] server

[1513] Input: Image data sent from the device.

[1514] Operation: The server passes the image data to the AI ​​analysis engine, which then starts the process of generating a 3D model. An image recognition algorithm using deep learning technology is then put into operation.

[1515] Output: The generated 3D model.

[1516] Step 4: Selecting block parts and generating assembly drawings

[1517] server

[1518] Input: The generated 3D model.

[1519] Operation: The server selects the optimal block parts from the database based on the analysis results of the 3D model. A detailed assembly drawing is generated based on the selected block parts.

[1520] Output: Assembly drawing data.

[1521] Step 5: Send and view assembly drawings

[1522] server

[1523] Input: Assembly drawing data.

[1524] Operation: The server sends assembly drawing data to the terminal.

[1525] Output: The assembly drawing data sent.

[1526] Terminal

[1527] Input: Submitted assembly drawing data.

[1528] How it works: The device displays an assembly diagram to the user, including animations and 3D views to help the user understand each step in detail.

[1529] Output: Visually detailed assembly drawings.

[1530] Step 6: Assembling the Block Components

[1531] User

[1532] Input: A visually detailed assembly diagram displayed on a terminal.

[1533] Operation: The user assembles the blocks according to the assembly diagram. The user manipulates the blocks at hand and follows the instructions to assemble the blocks.

[1534] Output: The completed assembly.

[1535] In this way, the system works by having the user take a 360-degree photograph of the object, the server analyzes the image and generates a three-dimensional model, then selects the most suitable block parts and generates an assembly drawing, and finally the user recreates the object using the block parts.

[1536] (Application example 1)

[1537] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1538] In modern brick-and-mortar stores, users often have difficulty purchasing or using products because they are not provided with sufficient information to understand how to assemble or recreate the product. Furthermore, when users recreate products using block parts, the assembly process is complex and time-consuming, resulting in a poor user experience. Therefore, there is a need for a system that allows users to easily understand and recreate products.

[1539] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1540] In this invention, the server includes an information terminal equipped with a camera for capturing 360-degree images of an object, an information processing device that receives and analyzes multiple images captured by the camera to generate a 3D shape model of the object, an information processing device that selects components for reproducing the object based on the 3D shape model, generates an assembly drawing and transmits it to the information terminal, and a user who assembles the components according to the assembly drawing displayed on the information terminal. This enables support for users to easily reproduce and use products in physical stores.

[1541] An "information terminal" is an electronic device operated by a user, and is a terminal equipped with a camera capable of 360° photography.

[1542] A "photography device" is a device that is built into or connected to an information terminal and is used to take 360-degree photographs of an object.

[1543] An "information processing device" is a device that receives image data sent from an information terminal, analyzes it, generates a three-dimensional shape model, and creates an assembly drawing.

[1544] A "three-dimensional shape model" is a model that reproduces the three-dimensional structure of an object based on multiple images taken by a photographing device.

[1545] A "component" is a part used to reproduce an object based on a three-dimensional shape model.

[1546] An "assembly drawing" is information that includes detailed drawings and instructions for a user to recreate an object using component parts.

[1547] A "user" is a person who operates an information terminal and assembles components according to the displayed assembly diagram.

[1548] "Brick and Mortar Store" means a physical store that a User can visit in person, where Products are displayed and sold.

[1549] The "database" is an information system that stores and manages three-dimensional shape models and the information necessary for selecting optimal component parts.

[1550] An "algorithm" refers to a calculation method or processing procedure for selecting optimal components from a database based on a three-dimensional shape model.

[1551] "Animated display" is a display method that includes animations and visual effects to visually explain assembly drawings to users.

[1552] "Three-dimensional view display" is a display method that displays assembly drawings from a three-dimensional perspective, helping users to understand them in three dimensions.

[1553] This invention is a system that takes 360° images of an object, analyzes the images, and recreates the object using block parts. Specific embodiments of this system are described in detail below.

[1554] System configuration and operation procedures

[1555] 1. Information terminal

[1556] An information terminal is an electronic device operated by a user and equipped with a camera capable of 360° photography. The user activates the device and takes photos of an object from multiple angles. Examples of such devices include smartphones and tablets.

[1557] 2. Imaging equipment

[1558] The image capturing device is a device that is built into or connected to an information terminal and captures 360-degree images of an object, and is used when a user takes a picture of an object (such as furniture) in a store.

[1559] 3. Information Processing Device

[1560] The information processing device receives image data sent from the information terminal and generates a 3D shape model using an AI analysis engine. Specifically, it uses image analysis software such as TensorFlow. Based on the generated model, it also selects the optimal components from a database and creates an assembly drawing. MySQL is used as the database software.

[1561] 4. Database

[1562] The database is an information system that stores and manages the information necessary for selecting 3D geometric models and optimal component parts, including past analysis results and part information.

[1563] 5. Algorithms

[1564] An algorithm is a calculation method or procedure for selecting optimal components from a database based on a 3D geometric model. It is optimized using a generative AI model.

[1565] 6. Generating and displaying assembly drawings

[1566] The server generates a detailed assembly drawing based on the analysis results. This drawing includes video and 3D views to instruct the user on each step in detail. For use in brick-and-mortar stores, the system takes 360-degree photos of the product on-site, analyzes the images, and provides instructions for recreating the product using the component parts.

[1567] A natural language description of what the program does

[1568] The server first receives multiple images sent from the information terminal. Next, it analyzes the image data using an AI analysis engine (e.g., TensorFlow) and generates a 3D shape model of the object. Based on this 3D shape model, it selects the optimal components from a database (MySQL) and generates a detailed assembly drawing using an algorithm. The generated assembly drawing is then sent to the information terminal and displayed to the user. The user recreates the object by assembling the components according to the displayed assembly drawing.

[1569] Specific examples

[1570] For example, if a user wants to recreate a chair sold in an interior design shop, the following steps are performed: First, the user starts up their information terminal and takes photos of the chair from multiple angles. Next, the captured images are sent to the server. The server analyzes the images and generates a 3D shape model. After that, it references a database, selects the optimal components, and generates an assembly drawing. Finally, the assembly drawing is sent to the information terminal, and the user recreates the chair according to its contents.

[1571] Example prompts for generative AI models

[1572] Generate a 3D model of the object from 360° images, select the optimal components based on that, and generate assembly drawings.

[1573] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1574] Step 1:

[1575] The user starts up the information terminal and grants permission to access the camera. The terminal performs initial settings and displays the main screen.

[1576] Input: User actions, camera permissions

[1577] Output: Main screen of the information terminal

[1578] Specific operation: The information terminal prepares the camera and sensors and launches the application as instructed by the user.

[1579] Step 2:

[1580] A user uses an information terminal to take a 360-degree image of an object (such as a chair) in a physical store. The terminal checks the quality of the images in real time and proceeds to the next step once the required number of images have been collected.

[1581] Input: User-taken image

[1582] Output: High-quality 360° images

[1583] Specific operation: The information terminal uses an image processing algorithm to analyze the quality of captured images in real time and instructs the user to take a new image if any are insufficient.

[1584] Step 3:

[1585] The terminal transmits the captured image data to the server.

[1586] Input: Captured image data

[1587] Output: Image data received on the server side

[1588] Specific operation: The information terminal compresses and organizes the image data and sends it to a server via the Internet.

[1589] Step 4:

[1590] The server analyzes the received image data and generates a three-dimensional shape model using an AI analysis engine (TensorFlow).

[1591] Input: Received image data

[1592] Output: 3D shape model

[1593] How it works: The server uses an AI analysis engine to analyze the relationships between images and build a 3D shape model. Parallel processing technology is used for the calculations.

[1594] Step 5:

[1595] The server runs an algorithm to select the optimal components from a database (MySQL) based on the generated 3D shape model.

[1596] Input: 3D shape model

[1597] Output: List of components

[1598] Specific operation: The server uses a database search algorithm to quickly search and select the optimal components for the 3D shape model.

[1599] Step 6:

[1600] The server generates detailed assembly drawings based on the selected components.

[1601] Input: List of components

[1602] Output: Assembly drawing

[1603] What it does: The server uses CAD software to generate assembly drawings of how components fit together, and creates animations and 3D views.

[1604] Step 7:

[1605] The server transmits the generated assembly drawing to the terminal, which displays it to the user.

[1606] Input: Assembly drawing data

[1607] Output: Assembly diagram displayed on an information terminal

[1608] Specific operation: The server compresses and organizes the assembly drawing data and sends it to the terminal via the Internet. The information terminal decompresses the received data and displays it on the user interface.

[1609] Step 8:

[1610] The user assembles the object using the component parts according to the displayed assembly drawing.

[1611] Input: Assembly drawing, components

[1612] Output: Finished object

[1613] Specific operation: The user manually assembles the components according to the specific assembly instructions displayed on the terminal. After completing the assembly, the user proceeds to the confirmation step via the terminal.

[1614] Prompt Sentence Examples

[1615] Generate a 3D model of the object from 360° images, select the optimal components based on that, and generate assembly drawings.

[1616] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1617] This invention is a system that aims to capture 360° images of an object, analyze the images, and recreate the object using block parts, and by combining it with an emotion engine, it aims to improve the enjoyment of the toy, its educational effects, and the user experience. Specific embodiments of this system are described in detail below.

[1618] System configuration and operation procedures

[1619] This system consists of four main components: a terminal, a server, an emotion engine, and a user.

[1620] 1. Terminal

[1621] The device is an electronic device such as a smartphone or tablet, equipped with a camera capable of 360° photography. The user first starts up the device and launches the application. Next, they operate the camera according to the application's instructions to photograph the object from multiple angles. The device saves the captured images in real time and sends the image data to a server once the capture is complete.

[1622] 2. Server

[1623] The server receives multiple images sent from the device and analyzes them to generate a 3D model of the object. Specifically, the server's AI analysis engine processes the images and analyzes the object's shape and structure. It then selects the optimal block parts from a past database and generates a detailed assembly drawing. Once the assembly drawing is complete, the server sends the data back to the device.

[1624] 3. Emotion Engine

[1625] The emotion engine has the ability to analyze the user's facial expressions in real time and recognize their emotions. The device's camera captures images of the user during assembly and sends the captured images to the emotion engine. Based on the emotion recognition results, the emotion engine dynamically changes the display content of the assembly diagram and the interactions. It also compares the user's emotion data with a past database to learn the user's preferences and tendencies.

[1626] 4. Users

[1627] The user operates the device to display the assembly diagram sent to them and assemble the object using the block parts. The assembly diagram includes animations and three-dimensional views that show each step in detail, making it easy for the user to understand. The user completes the steps according to the assembly progress and finally recreates the object. The system is equipped with an emotion engine, so the system provides appropriate feedback and assistance depending on the user's facial expressions and emotions.

[1628] Program processing and specific examples

[1629] 1. Initialization and configuration

[1630] The user launches the app on their smartphone and grants permission to access the camera and the emotion engine. The device then performs the necessary initial settings and displays the main screen.

[1631] 2. Photographing the subject

[1632] The user takes photos of an object, such as a coffee cup, from multiple angles, and the device checks the quality of the images in real time, moving on to the next step once the required number of photos have been taken.

[1633] 3. Image analysis and data transmission

[1634] The device sends the captured images to a server, which receives them and analyzes them to generate a 3D model.

[1635] 4. Block part selection and assembly drawing generation

[1636] Based on the analysis results, the server selects the optimal block parts from the database and generates detailed assembly drawings, which are then sent to the terminal.

[1637] 5. Displaying assembly diagrams and assembly

[1638] The terminal displays the received assembly diagram to the user, showing each step in detail. The user assembles the block parts according to the assembly diagram and recreates the object.

[1639] 6. Emotion Recognition and Feedback

[1640] The device's camera captures the user's facial expressions during assembly and sends the image data to the emotion engine, which analyzes the user's emotions and provides appropriate feedback and assistance in real time. For example, if the user is confused, the system will display detailed animations to provide additional explanations.

[1641] Specific examples

[1642] For example, if a user wants to recreate a coffee cup at home, he or she would follow the steps below.

[1643] 1. The user launches the app and takes photos of a coffee cup from multiple angles.

[1644] 2. The device sends the captured image to the server.

[1645] 3. The server analyzes the image and generates a 3D model of the coffee cup.

[1646] 4. The server references the past database, selects the optimal block parts, and generates an assembly drawing.

[1647] 5. The assembly drawing is sent to the terminal, and the user recreates the coffee cup using the block parts according to the assembly drawing.

[1648] 6. During the assembly process, the device's camera captures the user's facial expressions, and the emotion engine analyzes the user's emotions and provides appropriate feedback. For example, if the user is happy, the system will add praise or animation effects.

[1649] In this way, the present invention not only provides a system that allows users to easily recreate objects at home using blocks, but also has the effect of improving the user experience through the emotion engine.

[1650] The processing flow will be explained below.

[1651] Step 1:

[1652] The user launches the application. The device performs an initial setup check and asks the user for permission to use the camera and emotion engine.

[1653] Step 2:

[1654] The device checks the camera and emotion engine permissions and displays the main screen, where the user selects the option to take a photo of the object.

[1655] Step 3:

[1656] The user takes photos of an object from multiple angles, and the device displays guidelines during the photo shoot to help ensure each angle is captured accurately.

[1657] Step 4:

[1658] The device saves the captured images in real time and checks the image quality (resolution, brightness, focus, etc.). Once the minimum number of images has been collected, a notification will be displayed indicating that the capture is complete.

[1659] Step 5:

[1660] The device compresses the captured image and sends it to the server. The progress of the sending process is displayed to the user in real time and a notification is sent when it is complete.

[1661] Step 6:

[1662] The server receives the image data and checks its consistency. If there are no problems, it starts image analysis.

[1663] Step 7:

[1664] The server's AI analysis engine analyzes the images and recognizes the shape and structure of the object, then builds a 3D model based on the generated data.

[1665] Step 8:

[1666] The server runs an algorithm to select the optimal block parts from a database based on the 3D model, and generates an assembly drawing using the selected parts.

[1667] Step 9:

[1668] The server sends the assembly drawing to the terminal. The terminal checks the progress of the sending process and notifies the user when it is complete.

[1669] Step 10:

[1670] The terminal unpacks the received assembly drawing and checks its consistency. If there are no problems, the assembly drawing is displayed to the user.

[1671] Step 11:

[1672] The device's camera captures the user during assembly and sends the captured video data to the emotion engine, which analyzes the user's facial expressions in real time and recognizes their emotions.

[1673] Step 12:

[1674] Based on the analysis results of the emotion engine, the device dynamically changes the content displayed on the assembly diagram and the interaction. For example, if the user is confused, it will display a detailed animation to provide additional explanation.

[1675] Step 13:

[1676] As the user completes each step, they press a progress check button and the terminal displays the next assembly step.

[1677] Step 14:

[1678] After the device is finished assembling, it will display a pop-up asking the user for feedback, which will help improve the algorithm for the next time.

[1679] Specific examples

[1680] For example, if a user wants to recreate a coffee cup at home, he or she would follow the steps below.

[1681] 1. The user launches the app and grants permission to use the camera and emotion engine.

[1682] 2. The user takes photos of a coffee cup from multiple angles.

[1683] 3. The device sends the captured image to the server.

[1684] 4. The server analyzes the image and generates a 3D model of the coffee cup.

[1685] 5. The server selects the optimal block parts and generates an assembly drawing.

[1686] 6. The assembly diagram is sent to the terminal, and the user recreates the coffee cup using the block parts according to the assembly diagram.

[1687] 7. The device's camera captures the user's facial expressions, and the emotion engine provides appropriate feedback based on the analysis results.

[1688] This system not only allows users to easily recreate objects with blocks, but also enables a richer, more interactive experience with dynamic feedback provided by the emotion engine.

[1689] Example 2

[1690] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1691] Conventional block play and educational systems lack support for users to fully utilize their creativity in recreating objects. Furthermore, they lack appropriate feedback and support based on the user's emotions and progress, resulting in insufficient educational benefits and user experience. Furthermore, it can take time for users to select the specific steps and components needed to recreate an object, which can discourage users from learning.

[1692] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1693] In this invention, the server includes means for receiving and analyzing multiple images captured by the imaging device to generate a 3D model of the object, means for selecting optimal components from past information storage based on the 3D model, and means for transmitting the generated assembly procedure to a mobile information terminal. This allows the user to easily recreate the object, and the emotion analysis device can provide feedback and support in real time, improving the educational effect and user experience.

[1694] A "personal digital assistant" is an electronic device that can be carried by a user, and includes devices such as smartphones and tablet terminals equipped with communication and camera functions.

[1695] "Photography equipment" refers to cameras and sensors that can capture 360-degree images of an object, and are capable of capturing high-resolution images from multiple angles.

[1696] The "information processing device" refers to a device such as a server or computer that analyzes received image data and generates a three-dimensional model of the target object.

[1697] "Components" refer to the block parts and components required to reproduce an object, and are the basic units for assembling each part.

[1698] A "three-dimensional model" is digital data that represents an object in three dimensions, and is a model with a three-dimensional structure generated from multiple images.

[1699] An "assembly procedure" is an instruction or guide for recreating an object using components, including detailed instructions and animations for each step.

[1700] The "operator" refers to a user who operates a mobile information terminal, follows an assembly procedure, and assembles block parts to recreate an object.

[1701] An "emotion analysis device" is a device that captures the operator's facial expressions and analyzes their emotional state, and has the function of providing feedback and support based on emotions.

[1702] System configuration

[1703] This invention is a system that captures 360° images of an object, analyzes the images, and recreates the object using block parts. This system is composed of four main components: a mobile information terminal, an information processing device (server), an emotion analysis device, and an operator. Specific embodiments of the system are described in detail below.

[1704] Mobile information terminal

[1705] A mobile information terminal is an electronic device such as a smartphone or tablet, equipped with a camera that supports 360-degree photography. The operator first starts up the mobile information terminal and launches the application. Next, to photograph an object from multiple angles, the operator operates the camera according to the application's instructions. The captured images are saved in real time, and when the photographing is complete, the image data is sent to an information processing device.

[1706] Information processing device (server)

[1707] The information processing device receives multiple images sent from the mobile information terminal and analyzes them to generate a three-dimensional model of the object. Specifically, an AI analysis engine in the server processes the images and analyzes the shape and structure of the object. It then selects the optimal components (block parts) from a database of past information and generates detailed assembly instructions. Once the assembly instructions are complete, the information processing device sends the data back to the mobile information terminal.

[1708] Emotion analysis device

[1709] The emotion analysis device has the ability to analyze the operator's facial expressions in real time and recognize their emotions. The camera in the mobile information terminal captures images of the operator during assembly and sends the captured images to the emotion analysis device. The emotion analysis device dynamically changes the assembly procedure display and interactions based on the emotion recognition results. It also compares the operator's emotional data with a past database to learn the operator's preferences and tendencies.

[1710] operator

[1711] The operator operates the mobile information terminal to display the assembly instructions sent to them and assemble the object using the block parts. The assembly instructions include animations and 3D views that show each step in detail, making them easy for the operator to understand. The operator completes the steps according to the assembly progress and finally recreates the object. The system is equipped with an emotion analysis device, so the system provides appropriate feedback and assistance depending on the operator's facial expressions and emotions.

[1712] Specific examples

[1713] For example, if an operator wants to recreate a coffee cup at home, the operator performs the following steps.

[1714] 1. The operator launches the app and takes photos of a coffee cup from multiple angles.

[1715] 2. The mobile information terminal transmits the captured image to the information processing device.

[1716] 3. The information processing device analyzes the image and generates a three-dimensional model of the coffee cup.

[1717] 4. The information processing device refers to the past information database, selects the optimal components, and generates an assembly procedure.

[1718] 5. The assembly instructions are sent to the mobile information terminal, and the operator follows the assembly instructions to recreate the coffee cup using the block parts.

[1719] 6. During the assembly work, the mobile information terminal's camera captures the operator's facial expressions, and the emotion analyzer analyzes the operator's emotions and provides appropriate feedback. For example, if the operator is happy, the system will add praise or animation effects.

[1720] Prompt Sentence Examples

[1721] "This program should create a system in which a 360-degree image of an object (such as a coffee cup) is taken, the server generates a 3D model, selects the optimal block parts, and displays assembly instructions to the operator. In addition, please include a process in which an emotion analysis device is used to analyze the operator's facial expressions in real time and provide appropriate feedback."

[1722] In this way, the present invention not only provides a system that allows an operator to easily recreate objects at home using blocks, but also has the effect of improving the operator's experience by using an emotion analysis device.

[1723] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1724] Step 1: Initialize and configure the app

[1725] Subject: Terminal

[1726] When the user launches the app, the device displays a dialog requesting permission to access the camera and the emotion analyzer. Once the user grants these permissions, the device performs initial setup and displays the main screen, which displays a "Start Recording" button and a "Settings" button.

[1727] Input: User action (launching an app, granting permission)

[1728] Output: Display of the app's main screen (preparation before shooting begins)

[1729] Step 2: Capture a 360° shot of the object

[1730] Subject: User, Device

[1731] The user presses the "Start Capture" button and takes photos of an object (such as a coffee cup) from multiple angles. The device application then instructs the device to rotate the object 360 degrees, taking photos at each angle. The device checks the image quality in real time and proceeds to the next step once the required number of photos have been taken.

[1732] Input: User operation (taking a photo of an object)

[1733] Output: High-quality multi-angle images taken

[1734] Step 3: Sending image data

[1735] Subject: Terminal

[1736] The device sends the captured images to an information processing device (server). At this time, the device uses a compression algorithm (e.g., JPEG compression) to optimize the data size and shorten the upload time.

[1737] Input: Multiple captured image data

[1738] Output: Compressed and transmitted image data

[1739] Step 4: Image analysis and 3D model generation

[1740] Subject: Server

[1741] The server takes in the received image data and begins image processing using its AI analysis engine. First, it analyzes each image to extract the object's shape information, then integrates them to generate a 3D model. The server then integrates information from different angles of the object to build an accurate 3D model. The generated 3D model is temporarily stored in an internal database.

[1742] Input: Compressed image data

[1743] Output: Generated 3D model of the object

[1744] Step 5: Select components and generate assembly procedures

[1745] Subject: Server

[1746] The server selects the optimal building blocks (block parts) from a database based on the 3D model. It searches for parts that suit the shape and size of the object and selects the optimal set. It then generates detailed assembly instructions using the selected parts. The generated assembly instructions are sent to the terminal.

[1747] Input: 3D model

[1748] Output: Selected components and detailed assembly instructions

[1749] Step 6: Display assembly instructions and assemble

[1750] Subject: Terminal, User

[1751] The terminal receives the assembly instructions sent from the server and displays them to the user. The assembly instructions include detailed animations and 3D views, making them easy for the user to understand. The user follows the assembly instructions displayed and recreates the object using the block parts in the correct order.

[1752] Input: Assembly instructions received from the server

[1753] Output: Assembly steps displayed to the user and the actual assembly work

[1754] Step 7: Emotion Recognition and Feedback

[1755] Subject: Terminal, emotion analysis device

[1756] During assembly, the device's camera captures the user's facial expressions in real time and sends the image data to an emotion analyzer, which analyzes the images and determines the user's emotional state. If the user is confused, the emotion analyzer prompts the display of detailed animations or additional explanations. If the user is happy, compliments or animation effects are added.

[1757] Input: User's facial expression data

[1758] Output: Sentiment analysis results and feedback based on them

[1759] (Application example 2)

[1760] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1761] Conventional building systems lack feedback that takes into account the user's emotions and experiences during the process of building an object. This can lead to confusion and loss of interest during the building process, resulting in a poor experience. This is particularly problematic in situations where educational benefits and maintaining user motivation are required.

[1762] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1763] In this invention, the server includes a terminal equipped with a camera for capturing 360-degree images of an object, means for receiving and analyzing multiple images captured by the camera to generate a 3D model of the object, means for selecting block parts for reproducing the object based on the 3D model, generating an assembly diagram and transmitting the diagram to the terminal, an emotion engine for analyzing a user's emotions and providing feedback, and means for dynamically changing interactions and display content based on the emotion analysis results. This allows the user to receive appropriate feedback according to their emotions during the process of assembling the blocks, resulting in a richer experience and improved educational effectiveness.

[1764] "Object" refers to an object that is captured as an image through 360° photography.

[1765] "Camera" means the photographic equipment required to capture 360° images of an object.

[1766] "Terminal" refers to an electronic device that is equipped with a camera and performs processes such as acquiring and transmitting images.

[1767] "Image" means visual data of an object captured by a camera.

[1768] A "server" is a computer system that receives images sent from a terminal and processes and analyzes them.

[1769] A "three-dimensional model" is a three-dimensional digital model of an object generated based on image data analyzed by the server.

[1770] "Block parts" refer to parts that physically reproduce a three-dimensional model.

[1771] An "assembly drawing" is a design drawing for recreating an object using block parts.

[1772] "User" refers to the person who operates the terminal or system to photograph and assemble the object.

[1773] An "emotion engine" is a system that has the ability to analyze a user's emotions and provide feedback based on the results.

[1774] "Interaction" refers to two-way communication and action between a system and a user.

[1775] "Feedback" refers to the response or assistance provided by the system in response to the user's actions or emotions.

[1776] "Algorithm" refers to a computational procedure for solving a particular problem.

[1777] A "database" is a collection of information in which past data is systematically accumulated.

[1778] "Animation" refers to moving images that express movement or change.

[1779] "View" refers to the visual representation of a three-dimensional model.

[1780] This invention provides a system that allows users to take 360° photos of an object and use the data to recreate it using block parts. Furthermore, it aims to improve the user experience by incorporating an emotion engine that analyzes the user's emotions.

[1781] System configuration

[1782] 1. Terminal

[1783] The device is a smartphone or tablet equipped with a camera that can capture 360-degree images of the object. This device captures the object from multiple angles and sends the image data to a server. It also displays assembly diagrams and emotional feedback.

[1784] 2. Server

[1785] The server is a computer system that receives and analyzes multiple images sent from the device. Specifically, it performs the following processes:

[1786] 3D model generation: Multiple images are analyzed to generate a 3D model of the object. This process utilizes image analysis software and AI technology.

[1787] Block part selection: Based on the 3D model, an algorithm is run to select the optimal block parts from a past database.

[1788] Assembly Drawing Generation: Generate detailed assembly drawings based on selected block parts, including animations and 3D views detailing each step.

[1789] 3. Emotion Engine

[1790] The emotion engine is a system that analyzes the user's facial expressions and provides feedback based on their emotions. It captures the user's facial expressions with the device's camera and performs emotional analysis in real time.

[1791] Program processing

[1792] The process of a program in this system is as follows:

[1793] 1. Activating the device and photographing the object: The user activates their smartphone or tablet and takes a 360° photograph of the object. The device then transmits the captured image data to the server in real time.

[1794] 2. 3D model generation: The server receives the image data, analyzes it, and processes it to generate a 3D model of the object. This process uses an image processing library (e.g., OpenCV) and an AI analysis model.

[1795] 3. Block part selection and assembly drawing generation: The server uses the generated 3D model to select the optimal block parts from the database. Based on the selected block parts, it generates a detailed assembly drawing and sends it to the terminal.

[1796] 4. Displaying and assembling assembly diagrams: Users assemble objects using the building blocks at hand while referring to the assembly diagrams displayed on their devices. The assembly diagrams include animations and 3D views, and instructions are displayed in a way that is easy for users to understand.

[1797] 5. Emotion analysis and feedback: The device's camera captures the user's facial expressions during assembly and sends the footage to the emotion engine. The emotion engine analyzes the user's emotions in real time and provides appropriate feedback based on the results. For example, if the user shows a confused expression, the system will display a detailed animation to assist them.

[1798] Specific examples

[1799] For example, if a user wants to recreate a coffee cup they have at home, they might perform the following steps:

[1800] 1. Photographing an object: The user launches the smartphone app and takes photos of a coffee cup from multiple angles in 360°.

[1801] 2. Sending image data: The device sends the captured image data to the server.

[1802] 3. Model generation: The server analyzes the image and generates a 3D model of the coffee cup.

[1803] 4. Block part selection: The server selects the optimal block parts from the database and generates an assembly drawing.

[1804] 5. Displaying the assembly diagram: The assembly diagram is sent to the terminal, and the user follows the instructions to recreate the coffee cup using blocks.

[1805] 6. Providing feedback: If the user is confused, the system will display detailed animations to assist with assembly.

[1806] This application allows users to have fun recreating objects while receiving emotional feedback.

[1807] Example prompts to input to a generative AI model:

[1808] "Design an application that analyzes images of an object scanned by a user in 360° using a smartphone, generates a 3D model, displays the model in a virtual space using a head-mounted display, and uses an emotion analysis engine to provide real-time feedback based on the user's emotions."

[1809] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1810] Step 1:

[1811] The user turns on their smartphone or tablet and starts the application. Following the application's instructions, they take photos of the object from multiple angles across 360°. The device's camera takes images from each angle and temporarily stores the image data. The input is the image data of the object, and the output is the multiple temporarily stored images.

[1812] Step 2:

[1813] The device checks the quality of the multiple image data taken, and if it is found to be of sufficient quality, it sends it to the server.Specifically, it automatically detects the image resolution, brightness, etc., and if it does not meet the standards, it displays a message urging the user to take another photo.The input is temporarily saved image data, and the output is image data sent to the server.

[1814] Step 3:

[1815] The server receives image data sent from the device and generates a 3D model using an image processing library (e.g., OpenCV) and an AI analysis model. Specific operations include aligning images, extracting feature points, and generating point cloud data. The input is multiple image data, and the output is a 3D model.

[1816] Step 4:

[1817] The server runs an algorithm to select the optimal block parts from a database based on the generated 3D model. Specifically, it uses a shape recognition algorithm to analyze the characteristics of the 3D model and searches the database for corresponding block parts. The input is the 3D model, and the output is information about the selected block parts.

[1818] Step 5:

[1819] The server generates an assembly drawing based on the selected block parts. The assembly drawing includes animations and 3D views that show each step in detail. Specific operations involve using CAD software or 3D modeling tools to visually represent the block part placement and assembly steps. The input is block part information, and the output is a detailed assembly drawing.

[1820] Step 6:

[1821] The server sends the generated assembly drawing to the terminal. The terminal displays the received assembly drawing to the user, who then follows the instructions to assemble the object using the block parts. Specifically, the application that displays the assembly drawing outputs a 3D view and animation to the user's screen. The input is detailed assembly drawing data, and the output is the user's assembly work.

[1822] Step 7:

[1823] The device's camera captures the user's facial expressions in real time during assembly and sends the image data to the emotion engine. The emotion engine analyzes the user's facial expressions and returns the results to the server. The input is the user's facial expression data, and the output is the emotion analysis results.

[1824] Step 8:

[1825] The server provides appropriate feedback and assistance based on the emotion analysis results. For example, if the user is confused, the system displays detailed animations or additional explanations. Specific operations include emotion recognition using an AI model and generating interactions based on that. The input is the emotion analysis results, and the output is dynamically changing user interactions.

[1826] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1827] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1828] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1829] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1830] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1831] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1832] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1833] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1834] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1835] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1836] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1837] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1838] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1839] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1840] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1841] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1842] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1843] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1844] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1845] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1846] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1847] The following is further disclosed regarding the above embodiment.

[1848] (Claim 1)

[1849] A device equipped with a camera for capturing 360° images of an object;

[1850] a server that receives and analyzes a plurality of images taken by the camera to generate a three-dimensional model of the object;

[1851] a server that selects block parts for reproducing an object based on the three-dimensional model, generates an assembly drawing, and transmits the assembly drawing to the terminal;

[1852] a user who assembles the blocks according to an assembly drawing displayed on the terminal;

[1853] A system including:

[1854] (Claim 2)

[1855] 2. The system according to claim 1, wherein the server includes an algorithm for selecting optimal block parts from a past database based on the three-dimensional model.

[1856] (Claim 3)

[1857] 10. The system of claim 1, wherein the assembly drawings generated by the server include animations and three-dimensional views to instruct the user on each step in detail.

[1858] "Example 1"

[1859] (Claim 1)

[1860] An imaging means for a user to capture a 360° image of an object;

[1861] an analysis means for receiving and analyzing the multiple images captured by the imaging means to generate a three-dimensional model of the object;

[1862] a generating means for selecting block parts for reproducing an object based on the three-dimensional model, generating an assembly drawing, and presenting the assembly drawing to a user;

[1863] assembly means for allowing the user to assemble block parts according to an assembly drawing;

[1864] A system including:

[1865] (Claim 2)

[1866] 2. The system according to claim 1, wherein the analysis means includes an algorithm for selecting an optimum block part from a past database based on the three-dimensional model.

[1867] (Claim 3)

[1868] 2. The system of claim 1, wherein the assembly drawing generated by the generating means includes animations and three-dimensional views to instruct the user on each step in detail.

[1869] "Application Example 1"

[1870] (Claim 1)

[1871] an information terminal equipped with a photographing device for photographing an object in 360 degrees;

[1872] an information processing device that receives and analyzes a plurality of images captured by the imaging device and generates a three-dimensional shape model of the object;

[1873] an information processing device that selects components for reproducing an object based on the three-dimensional shape model, generates an assembly drawing, and transmits the assembly drawing to the information terminal;

[1874] a user who assembles components according to an assembly drawing displayed on the information terminal;

[1875] A system including:

[1876] (Claim 2)

[1877] 2. The system according to claim 1, wherein the information processing device includes an algorithm for selecting optimal components from a past database based on the three-dimensional shape model.

[1878] (Claim 3)

[1879] 2. The system according to claim 1, wherein the assembly drawing generated by the information processing device includes an animation display or a three-dimensional view display for instructing the user on each step in detail.

[1880] (Claim 4)

[1881] The system of claim 1, wherein the information terminal is adapted for use in a physical store, photographs products on-site in 360°, analyzes the images, and provides instructions for recreating the products using component parts.

[1882] (Claim 5)

[1883] 5. The system according to claim 4, wherein the information terminal includes a function for checking the quality of images on-site in real time and supporting the user to take high-quality images.

[1884] "Example 2: Combining Emotion Engines"

[1885] (Claim 1)

[1886] A mobile information terminal equipped with a photographing device for photographing a target in 360 degrees;

[1887] an information processing device that receives and analyzes a plurality of images captured by the imaging device to generate a three-dimensional model of the target;

[1888] an information processing device that selects components for reproducing an object based on the three-dimensional model, generates an assembly procedure, and transmits the procedure to the mobile information terminal;

[1889] an operator who assembles components according to the assembly procedure displayed on the mobile information terminal;

[1890] an emotion analysis device that captures the facial expression of the operator during photography and assembly and analyzes the emotions;

[1891] A system including:

[1892] (Claim 2)

[1893] 2. The system according to claim 1, wherein the information processing device includes an algorithm for selecting optimal components from past information accumulation based on the three-dimensional model.

[1894] (Claim 3)

[1895] 2. The system according to claim 1, wherein the assembly procedure generated by the information processing device includes animations and three-dimensional views for instructing the operator on each step in detail.

[1896] (Claim 4)

[1897] 2. The system according to claim 1, wherein the emotion analysis device includes a function for providing feedback and assistance in real time based on the emotions of the operator.

[1898] "Application example 2 when combining emotion engines"

[1899] (Claim 1)

[1900] A device equipped with a camera for capturing 360° images of an object;

[1901] a server that receives and analyzes a plurality of images taken by the camera to generate a three-dimensional model of the object;

[1902] a server that selects block parts for reproducing an object based on the three-dimensional model, generates an assembly drawing, and transmits the assembly drawing to the terminal;

[1903] a user who assembles the blocks according to an assembly drawing displayed on the terminal;

[1904] an emotion engine that analyzes the user's emotions and provides feedback;

[1905] A means for dynamically changing interactions and display content based on the results of emotion analysis;

[1906] A system including:

[1907] (Claim 2)

[1908] 2. The system according to claim 1, wherein the server includes an algorithm for selecting optimal block parts from a past database based on the three-dimensional model.

[1909] (Claim 3)

[1910] 10. The system of claim 1, wherein the assembly drawings generated by the server include animations and three-dimensional views to instruct the user on each step in detail. [Explanation of symbols]

[1911] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. A device equipped with a camera for capturing 360° images of an object; a server that receives and analyzes a plurality of images taken by the camera to generate a three-dimensional model of the object; a server that selects block parts for reproducing an object based on the three-dimensional model, generates an assembly drawing, and transmits the assembly drawing to the terminal; a user who assembles the blocks according to an assembly drawing displayed on the terminal; A system including:

2. 2. The system according to claim 1, wherein the server includes an algorithm for selecting optimal block parts from a past database based on the three-dimensional model.

3. 10. The system of claim 1, wherein the assembly drawings generated by the server include animations and three-dimensional views to instruct the user on each step in detail.

Images