System

Abstract

A system is provided.SOLUTION: A system including means for a user to take a picture of a room and input a request, means for transmitting the picture data and the request to a server, means for analyzing the picture data and recognizing the size of the room and an existing furniture layout, means for generating candidates for optimal furniture and household appliances based on the recognition result and the request, means for acquiring product information from a plurality of distributors, means for generating a layout based on the acquired product information and displaying a 3D simulation to a user, and means for allowing the user who has checked the 3D simulation to perform a purchase procedure for the product.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] When choosing furniture or home appliances, many users are often overwhelmed by the sheer number of options, wondering about size and whether they fit the image of the room. Furthermore, existing applications using AR technology have the problem that users must make their own choices and are limited to products from specific stores. Furthermore, there is a lack of furniture and layout suggestions that meet the user's specific needs. These issues need to be resolved. [Means for solving the problem]

[0005] The present invention provides a system that includes a means for a user to take a photo of a room and input their requirements; a means for transmitting the photo data and the requirements to a server; a means for analyzing the photo data to recognize the room's size and existing furniture layout; a means for generating optimal furniture and home appliance candidates based on the recognition results and the user's requirements; a means for acquiring product information from multiple retailers; a means for generating a layout based on the acquired product information and displaying a 3D simulation to the user; and a means for the user to check the 3D simulation and then proceed with the purchase of the products. This allows a user to automatically receive suggestions for optimal furniture and home appliances simply by taking a few photos of the room and inputting their requirements. The suggested products are not limited to a specific store and include product information from multiple manufacturers and retailers, allowing the user to select the optimal product from a wide range of options. The proposed layout is also displayed as a 3D simulation, allowing the user to visually see how the layout will fit into their actual room.

[0006] "User" refers to a person who uses the system to receive suggestions for furniture and home appliances suitable for their room.

[0007] "Room photo" refers to an image containing visual information of one's room that a user provides to the system.

[0008] "Requests" refers to the wishes and demands regarding the layout and design of the room input by the user.

[0009] "Device" means the device (e.g., smartphone, tablet, computer) used by a User to upload photos and enter requests.

[0010] "Server" refers to the central system that receives photo data and requests sent by users, and performs analysis, processing, and data management.

[0011] "Photo data" refers to digital data of a photo of a room that a user sends to the server via a terminal.

[0012] "Analysis" refers to the computational process by which the server processes the photo data received and determines the size of the room and information about the objects placed in it.

[0013] A "generative AI model" refers to a form of artificial intelligence used to suggest optimal furniture and appliances based on user requests and photo data.

[0014] "Furniture" refers to items such as tables, chairs, sofas, etc. that are suggested as part of a room layout.

[0015] "Appliances" refers to electrical appliances proposed for installation in the room (e.g., television, refrigerator, lighting).

[0016] "Retailer" refers to the manufacturer or online store offering the proposed furniture or appliances.

[0017] "Product information" refers to product name, price, size, inventory information, etc. obtained from the retailer.

[0018] "Layout" refers to a plan showing the proposed placement of furniture and appliances.

[0019] "3D simulation" refers to a three-dimensional display that places furniture and appliances within a room space based on two-dimensional photographic data.

[0020] The "purchase procedure" refers to a series of operations that a user performs to actually purchase a suggested product.

[0021] The term "system" refers to the overall mechanism for executing a series of processes provided by the present invention. [Brief explanation of the drawings]

[0022] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2]1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0023] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0024] First, the terms used in the following description will be explained.

[0025] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0026] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0027] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0028] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0029] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0030] [First embodiment]

[0031] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0032] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0033] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0034] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0035] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0036] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0037] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0038] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0039] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0040] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0041] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0042] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0043] MODE FOR CARRYING OUT THE INVENTION

[0044] System Configuration

[0045] The "AI Interior Coordinator" system of this invention is primarily composed of a terminal operated by the user, a server that processes data, and a product information database that utilizes the APIs of multiple retailers. The system starts when the user takes a photo of the room and inputs their requirements, and then makes interior design suggestions based on those requirements.

[0046] Program processing

[0047] The program processing of this system will be explained in natural language below.

[0048] 1. User Input and Data Submission

[0049] The user first takes a photo of the room. It is recommended to take several photos from different angles to capture the entire room. The user also inputs their requests for the room layout in text format. For example, they might say, "I want the room to look spacious" or "I want it to have a modern feel." When the user enters this information into the app and presses the "Send" button, the data is sent to the server via their device.

[0050] 2. Data analysis and proposal generation

[0051] The server receives the photo data and requests sent from the device. The received photo data is first analyzed using an image analysis algorithm. This analysis identifies the size of the room, the layout of existing furniture, the color of the walls, and other factors. For example, the OpenCV library or the YOLO object detection algorithm can be used. Next, based on the user's requests, the data is input into a generative AI model (e.g., GPT-4) to generate optimal furniture and appliance candidates. For example, a glass table or a small sofa could be suggested to make the room appear larger.

[0052] 3. Obtaining product information and generating layouts

[0053] The server accesses the APIs of multiple retailers to obtain product information for the generated list of furniture and home appliance candidates. This product information includes product name, price, size, and stock information. Based on the obtained product information, the server generates the optimal layout for the user's room. This layout is generated as a 3D simulation model and provided to the user. For example, Unity or Three.js can be used to create the 3D model and virtually arrange the furniture in the room.

[0054] 4. View and confirm the proposal

[0055] The terminal displays the 3D simulation model and furniture list received from the server to the user. The user can view this 3D simulation via a smartphone or tablet. In addition, it is possible to virtually place furniture in a real room using AR technology and check how it will look. If the user is satisfied with the suggestions, they can add the desired items to their cart and complete the purchase process.

[0056] 5. Purchase Procedures and Transaction Notifications

[0057] If the user likes the suggested items, they press the "Purchase" button to add them to their cart. As part of the purchasing process, they enter their payment information and confirm the order. The terminal then sends the user's purchase request and payment information to the server. The server accesses the seller's API to check inventory and process delivery, and once the order is confirmed, it notifies the user's terminal of the transaction status. This notification allows the user to check the progress of the transaction and delivery information in real time.

[0058] Specific examples

[0059] Example 1: Make the room look larger

[0060] Suppose a user takes three photos of a 30-square-meter apartment for a single person and inputs the request to make it appear larger. The device sends this data to a server, which analyzes the photo data and identifies the dimensions of the room and the location of existing furniture. Based on the request, the generative AI model suggests a glass table and a small sofa, and obtains information on these products from multiple retailers.

[0061] The server generates a 3D simulation based on the acquired product information and sends it to the terminal. The user can check the 3D simulation and, if they like it, purchase the suggested product. The server then sends an order to the seller and notifies the terminal of the transaction status and delivery information.

[0062] The processing flow will be explained below.

[0063] Step 1:

[0064] The user takes three photos of the room. The photos are taken from different angles to capture information about the entire room. In addition, the user inputs preferences through the interface, such as "I want it to look spacious" or "I want a modern design."

[0065] Step 2:

[0066] The device receives the user's photo data and requests. This data is converted into the appropriate format and prepared for transmission. The device also encrypts the data to ensure security.

[0067] Step 3:

[0068] The device sends the encrypted photo data and request to the server using a secure communication protocol (e.g., HTTPS). It is recommended to use a checksum during data transmission to prevent errors.

[0069] Step 4:

[0070] The server stores the received photo data and requests in a database. To begin analyzing the photo data, it first sends the data to an image analysis algorithm, which uses libraries such as OpenCV and YOLO to identify the size of the room, the existing furniture arrangement, and the color of the walls.

[0071] Step 5:

[0072] The server inputs the analysis results and the user's requests into a generative AI model (e.g., GPT-4) to generate a list of suitable furniture and appliances. For example, furniture that makes a room appear larger might include a glass table, a small sofa, and a mirror with reflective materials.

[0073] Step 6:

[0074] Based on the generated candidate list, the server accesses the APIs of multiple retailers to obtain product information, including product name, size, price, stock status, etc. This information is then stored in a database.

[0075] Step 7:

[0076] The server generates a 3D simulation model based on the acquired product information. This simulation model virtually arranges furniture in the room layout and is created using a 3D graphics library (e.g., Unity or Three.js).

[0077] Step 8:

[0078] The server then sends the generated 3D simulation model and product information to the terminal, where the data is packetized in an appropriate format and transmitted again using a secure communication protocol.

[0079] Step 9:

[0080] The device analyzes the received data and displays a 3D simulation to the user, who can view the simulation through the interface of their smartphone or tablet. Additionally, by selecting the AR mode, it is possible to place virtual furniture in a real room.

[0081] Step 10:

[0082] The user checks the proposed furniture and appliance arrangement, and if they like it, they press the "Purchase" button, which adds the desired items to their cart and proceeds to the purchase process.

[0083] Step 11:

[0084] The terminal receives the purchase request and payment information from the user, encrypts it, and sends it to the server. The payment information is handled using a security protocol (e.g., TLS).

[0085] Step 12:

[0086] The server accesses the seller's API to check inventory and initiate the shipping process. Once the order is confirmed, the server calculates the commission and secures the revenue.

[0087] Step 13:

[0088] After the order is confirmed and the delivery procedure is completed, the server notifies the terminal of the transaction status, allowing the user to check the transaction progress and delivery information in real time.

[0089] Step 14:

[0090] The terminal displays transaction status and delivery information to the user and notifies them that the purchase has been completed.

[0091] Example 1

[0092] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0093] Conventional interior coordination systems have issues such as difficulty for users in visualizing the room layout, lack of information, and the time and effort required to make a purchasing decision. Furthermore, when acquiring product information from multiple retailers, data acquisition and management can be cumbersome, making it necessary to develop an efficient system. In addition, users lack a visual experience to confirm the proposals, making it difficult to visualize the actual layout.

[0094] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0095] In this invention, the server includes: a means for a user to acquire an image of a room and input their requirements; a means for transmitting the image data and the requirements to an information processing device; a means for analyzing the image data and identifying the dimensions and existing layout of the room; a means for generating optimal furniture and electronic device candidates based on the identification results and the requirements using a generative AI model; a means for acquiring product information from multiple vendors; a means for generating a layout based on the acquired product information and displaying a 3D simulation to the user; and a means for the user to check the 3D simulation and then proceed with purchasing the products. This allows the user to visually confirm the room layout, making it easier to make a purchasing decision through efficient information acquisition and a visual experience.

[0096] "User" refers to an individual or organization that uses the system.

[0097] "Image" refers to a digital photograph or graphic visual record of the exterior or interior of a room.

[0098] "Information processing device" refers to a computer system that receives, analyzes, generates, and transmits data.

[0099] "Analysis" refers to the process of determining room dimensions and furniture layout based on image data.

[0100] "Dimensions" refers to physical measurements related to the size of a room or furniture.

[0101] "Layout" refers to the relative positions of furniture and decorations within a room.

[0102] "Specific results" refers to data regarding the dimensions and layout of the room obtained through analysis.

[0103] "Candidates" refers to a list of furniture and electronic devices that are optimally selected based on the user's requirements.

[0104] A "generative AI model" refers to an algorithm that uses machine learning to suggest furniture and electronic devices that meet user needs.

[0105] "Product information" refers to data such as the name, price, size, and inventory of furniture and electronic devices.

[0106] "Layout" refers to a virtual layout of furniture and decorations in a room.

[0107] "3D simulation" refers to a model that visually reproduces the furniture arrangement in a room using 3D computer graphics.

[0108] "Augmented reality technology" refers to technology that overlays digital information onto real space.

[0109] "Retailer" refers to a business or store that sells furniture or electronics.

[0110] "Checkout" refers to the process by which a user selects the desired product, enters payment information, and confirms the order.

[0111] "Order" refers to a transaction request for an item that a user wishes to purchase.

[0112] MODE FOR CARRYING OUT THE INVENTION

[0113] The "AI Interior Coordinator" system of this invention begins when a user acquires a photo of a room and inputs their requirements. The photo data taken by the user and the input requirements are sent to an information processing device (server), which then performs a series of data analyses and proposes interior designs. Specific embodiments are described below.

[0114] User Actions

[0115] First, the user launches a dedicated application and takes photos of the room from multiple angles. For example, it is recommended to take four photos: one from the front, one from the left, one from the right, and one from the back. The user also enters their room-related requirements in text format (e.g., "I want the room to look spacious" or "I want a modern feel"). Once these are entered, the user presses the "Send" button in the application. This sends the photo data and requirements from the device to the information processing device (server).

[0116] Server Operation

[0117] The server receives the submitted photo data and request. First, it analyzes the photo data using the OpenCV library and the YOLO object detection algorithm to determine the room dimensions and the layout of existing furniture. Specifically, it uses the YOLO algorithm to identify furniture and obtain the position and size of each piece of furniture. Next, it uses a generative AI model (e.g., GPT-4) based on the user's request to generate optimal furniture and electronic device candidates. At this time, it inputs data into the generative AI model using prompt statements.

[0118] Prompt Sentence Examples

[0119] "Please suggest the best interior coordination to make a 30 square meter room look larger. Using glass furniture and a small sofa."

[0120] "Can you give me some ideas for modernizing this room? The current furniture arrangement is included in the images."

[0121] Obtaining product information

[0122] Based on the generated candidate list of furniture and electronic devices, the server accesses the APIs of multiple vendors to obtain corresponding product information. The obtained product information includes product name, price, size, stock information, etc. For example, the product information is obtained using the API of an internet shopping mall.

[0123] Layout generation and display

[0124] The server uses Unity or Three.js to generate a 3D simulation model based on the acquired product information and create an optimal layout. These 3D simulations include furniture virtually placed in the room. This model is sent to the device so that the user can visually check it.

[0125] Proposal confirmation and purchase process

[0126] The device displays the 3D simulation model and product list received from the server to the user. The user can view this 3D simulation through the app. Augmented reality technology using the smartphone camera also allows the user to place virtual furniture in a real room and simulate how it will look. If the user is satisfied with the suggestions, they can add the desired products to their cart and proceed with the purchase process.

[0127] Purchase Process and Transaction Notifications

[0128] The user presses the "Purchase" button, adds the product to the cart, enters payment information, and confirms the order. The terminal sends the user's purchase request and payment information to the server. The server accesses the vendor's API to check inventory and process delivery, and once the order is confirmed, it notifies the user's terminal of the transaction status and delivery information. This allows the user to check the progress of the transaction and product delivery information in real time.

[0129] In this way, the "AI interior coordinator" of the present invention provides users with an efficient and intuitive interior coordination proposal and purchasing process.

[0130] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0131] Step 1: User Input and Data Submission

[0132] The user uses a dedicated application to take photos of the room from multiple angles. As input data, they take photos of the front, left, right, and back of the room, and enter their requests in text format, such as "I want the room to look spacious" or "I want a modern feel." When they press the "Send" button, this data is sent from the device to the server. As output, the photo data and the requests in text format are sent to the server.

[0133] Step 2: Receiving and initial processing of data

[0134] The server receives the photo data and requests sent from the terminal. As input data, it receives the photo data and requests sent from the user. Based on this, it performs initial processing of the data and prepares it for data analysis. As output, it generates the photo data and requests converted into the format required for analysis.

[0135] Step 3: Data analysis

[0136] The server analyzes the photo data using the OpenCV library and the YOLO object detection algorithm. It uses the photo data formatted for processing as input data. Based on this, it identifies furniture and room dimensions in the image and extracts attribute information (position, size, color, etc.) of the identified objects. The output is the room dimensions and the existing furniture layout.

[0137] Step 4: Proposal generation using generative AI models

[0138] The server inputs the identification results and the user's requests into a generative AI model (e.g., GPT-4) and generates optimal furniture and electronic device candidates. The input data includes the room dimensions, existing furniture layout, and the user's requests (in text format). Based on the generated prompt, the AI ​​model analyzes and outputs specific furniture and interior design suggestions. The output is a list of suggested furniture and electronic devices.

[0139] Step 5: Obtaining product information using the API

[0140] The server accesses the APIs of multiple vendors to obtain product information based on the generated list of furniture and electronic device candidates. The server uses the list of proposed furniture and electronic devices as input data. Based on this, it makes API requests to obtain product names, prices, sizes, stock information, etc. The server generates a list of the obtained product information as output.

[0141] Step 6: Generate a 3D simulation model

[0142] The server uses Unity or Three.js to generate a 3D simulation model based on the acquired product information. The input data is a list of product information and the dimensions of the existing room. Based on this, it creates an optimal layout and generates a 3D model of the furniture virtually placed in the room. The generated 3D simulation model is obtained as the output.

[0143] Step 7: View and Review the Simulation

[0144] The device displays the 3D simulation model and product list received from the server to the user. The 3D simulation model sent from the server is used as input data. The user can visually check this 3D simulation through the app. It is also possible to use augmented reality technology to place virtual furniture in a real room and simulate how it will look. The output is a visual 3D simulation that the user can check.

[0145] Step 8: Checkout

[0146] Based on the suggestions they like, the user presses the "Purchase" button, adds the product to their cart, enters payment information, and confirms the order. The input data is product information and payment information. Based on this, the purchase procedure is carried out. As an output, information indicating that the user has completed the purchase procedure is sent to the server.

[0147] Step 9: Transaction Notification

[0148] The server accesses the seller's API based on the user's purchase request and payment information to check inventory and process delivery. Detailed purchase process information is used as input data. Once the order is confirmed, transaction status and delivery information is sent to the user's device. As an output, the user receives notifications that allow them to check the transaction progress and delivery information in real time.

[0149] (Application example 1)

[0150] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0151] In today's brick-and-mortar stores, customers often have difficulty visualizing the product in its actual placement when selecting the right interior product for their home. Traditionally, purchasing decisions are often made solely based on catalogs and displays, increasing the risk of making the wrong choice. Similarly, online shopping makes it difficult to confirm the product's suitability in the actual room environment. This leads to an increase in returns and exchanges after purchase, lowering customer satisfaction and increasing operational costs.

[0152] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0153] In this invention, the server includes: a means for a user to take an image of a room and input their preferences; a means for transmitting the image data and preferences to the server; a means for analyzing the image data to recognize the size of the room and the existing item layout; a means for generating optimal device and tool candidates based on the recognition results and the user's preferences; a means for acquiring product information from multiple sellers; a means for generating a layout based on the acquired product information and displaying a 3D simulation to the user; a means for the user to check the 3D simulation and then complete a product purchase procedure; and a means for displaying virtual items arranged in a real space and using augmented reality technology to confirm the display. This allows users to visually confirm the actual layout of virtual interior items in a physical store or at home. Furthermore, customers can accurately confirm the suitability of products before purchase, preventing incorrect selections and reducing returns and exchanges.

[0154] A "user" refers to an individual who uses the system to take pictures and input their preferences to coordinate the interior of a room.

[0155] "Image data" refers to photographic data of a room taken by a user, and includes information such as the size of the room and the layout of existing items.

[0156] "Wishes" are requests that the user inputs regarding the interior of the room, and refer to specific requests such as "I want the room to look spacious."

[0157] "Server" refers to a computer system that receives image data and requests sent by users and performs analysis and data processing.

[0158] "Analysis" refers to the process of identifying the size of a room and the layout of existing items based on image data, and is done using computer vision techniques and object recognition algorithms.

[0159] "Device and tool candidates" refers to a list of optimal interior products suggested by the system based on the user's preferences and analysis results.

[0160] "Sellers" refer to companies and stores that sell interior products, and the system obtains product information from them.

[0161] "Product Information" refers to detailed data about interior products provided by sellers, such as product name, price, size, and inventory information.

[0162] "Layout" refers to the layout of interior products virtually arranged in a room based on the acquired product information.

[0163] "Three-dimensional simulation" refers to three-dimensional simulation data for visually displaying the generated layout to the user.

[0164] "Augmented reality technology" refers to a technology that displays virtual objects superimposed on a real-world environment, allowing users to visually check virtual interior products in real space.

[0165] "Purchase procedure" refers to the series of operations that a user performs to actually purchase interior products suggested through the system.

[0166] This invention provides an "interior AR shopping assistant" system that allows users to try out interior products in real time in a "real store." To realize this system, the following hardware and software are required:

[0167] Hardware and software used

[0168] Hardware: Requires a smartphone (camera, display, internet connection).

[0169] Software: OpenCV, YOLO (object detection), Unity (3D simulation), generative AI model (GPT-4API), etc. are required.

[0170] System configuration and operation

[0171] 1. User Action:

[0172] Users use their smartphones to take pictures of their room from multiple angles and input their desired interior style and requirements in text format, such as "I want the room to look spacious" or "I want a modern feel."

[0173] 2. Data transmission:

[0174] The user's terminal transmits the captured image data and the desired text data to the server.

[0175] 3. Analysis of imaging data:

[0176] The server analyzes the received image data using OpenCV and YOLO to determine the size of the room and the layout of existing items. This analysis determines the floor area, the location of walls, and how existing furniture is arranged.

[0177] 4. Generate optimal furniture candidates:

[0178] The server uses a generative AI model (GPT-4) to generate optimal interior product candidates based on the analyzed data and the user's preferences. For example, if the user desires a "modern taste" for a living room measuring approximately 20 square meters, the generative AI model will suggest a simple white sofa and a glass coffee table.

[0179] Example prompt sentence:

[0180] Room features: 20 square meters, white walls, existing furniture: dark wooden table and bookshelf.

[0181] User's request: I want a modern and simple interior.

[0182] 5. Obtaining product information:

[0183] The server obtains information about the interior products listed as candidates through APIs of multiple sellers, including product names, prices, sizes, and stock information.

[0184] 6. Generation of 3D simulations:

[0185] The server uses Unity to create a 3D simulation based on the acquired product information, and the simulation is generated in a virtual room with the proposed furniture arranged.

[0186] 7. View the simulation:

[0187] The user's device displays the 3D simulation sent from the server and uses augmented reality (AR) technology to overlay virtual interior products onto the real space, allowing the user to visually confirm how the virtual interior products will actually look when placed in a physical store or at home.

[0188] Specific examples

[0189] For example, if a user wants to make a 30-square-meter living room look modern and spacious, the system analyzes images taken from various angles of the room and generates suggestions such as a white sofa and a glass table. Product information is then acquired and a 3D simulation is generated based on the acquired information. The user can then virtually place these suggested furniture items in their real room using their smartphone, see how they will look, and then purchase the items they like.

[0190] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0191] Step 1:

[0192] The user takes a picture of the room and inputs their preferences.

[0193] The user uses a smartphone to take multiple images of the room from different angles. They also input their preferences for the room's interior in text format. For example, they might input a preference such as "I want the room to look spacious." The input data is saved as image data and text data.

[0194] Step 2:

[0195] The device sends the photo data and your request to the server.

[0196] The device sends the image data and text data the user has entered to the server, which then securely transfers the data over the Internet and stores it on the server.

[0197] Step 3:

[0198] The server analyzes the photo data and recognizes the size of the room and the layout of existing items.

[0199] The server analyzes the received image data using OpenCV and YOLO object detection algorithms. This analysis identifies the size of the room, the location of walls, and the layout of existing furniture. For example, the floor area and furniture locations are automatically calculated.

[0200] Step 4:

[0201] The server generates optimal device and tool candidates based on the recognition results and user preferences.

[0202] The server uses a generative AI model (GPT-4) based on the analysis results and the user's preferences to generate optimal device and tool candidates. The analysis results and the user's preferences are input into the prompt text.

[0203] Example prompt sentence:

[0204] Room features: 20 square meters, white walls, existing furniture: dark wooden table and bookshelf.

[0205] User's wish: A modern and simple interior.

[0206] Based on this prompt, the generative AI model suggests specific interior product options, such as a simple white sofa or a glass table.

[0207] Step 5:

[0208] The server retrieves product information from multiple sellers

[0209] Based on the generated list of furniture and tool candidates, the server accesses the APIs of multiple sellers to obtain product information. The obtained product information includes product name, price, size, stock information, etc. This information is imported into the server as structured data in JSON format, etc.

[0210] Step 6:

[0211] The server generates a layout based on the product information acquired and creates a 3D simulation.

[0212] The server uses Unity to create a 3D simulation based on the acquired product information. The placement data is constructed as a 3D model, and a simulation of the virtual furniture placement in the room is generated. This simulation data is visualized for the user.

[0213] Step 7:

[0214] The server sends the 3D simulation to the terminal, which displays it.

[0215] The server sends the generated 3D simulation to the user's device, which then displays the received simulation data, allowing the user to use augmented reality (AR) technology to overlay virtual furniture onto the real space, creating a visual experience that makes it seem as if the new furniture is actually placed in the room.

[0216] Step 8:

[0217] The user completes the purchase process

[0218] The user checks the 3D simulation and selects the product they wish to purchase. As part of the purchase process, they enter payment information and confirm the order. The terminal sends the user's purchase request and payment information to the server, which then accesses the seller's API to check inventory and process delivery. Once the order is confirmed, the user's terminal is notified of the transaction status and delivery information.

[0219] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0220] MODE FOR CARRYING OUT THE INVENTION

[0221] System Configuration

[0222] The "AI Interior Coordinator" system of this invention not only proposes optimal furniture and home appliances based on user input data, but also recognizes the user's emotions and adjusts the proposals accordingly. This system is primarily composed of a terminal operated by the user, a server that performs data analysis and emotion recognition, and a product information database that utilizes the APIs of multiple retailers.

[0223] Program processing

[0224] The program processing of this system will be explained in natural language below.

[0225] 1. User Input and Data Submission

[0226] The user takes three photos of the room and inputs their requirements. It is recommended that the photos be taken from different angles to capture the entire room. The user inputs their requirements, such as "I want the room to look spacious" or "I want a modern design." In addition, the user collects their facial expressions and voice data through the interface. This data is sent to the emotion engine.

[0227] 2. Data analysis and proposal generation

[0228] The server receives the photo data, requests, facial expressions, and voice data sent from the device. The photo data is analyzed using an image analysis algorithm to determine the size of the room, the layout of existing furniture, the color of the walls, etc. The analysis results and the user's requests are input into a generative AI model to generate a list of optimal furniture and home appliances.

[0229] 3. Emotion recognition and suggestion adjustment

[0230] The server uses an emotion engine to analyze the user's facial and voice data and recognize emotions such as joy, surprise, and dissatisfaction. For example, a machine learning library is used for facial expression analysis, and a voice recognition algorithm is used for voice analysis. Based on the emotion recognition results, the generated candidate list of furniture and home appliances is re-adjusted according to the user's emotions.

[0231] 4. Obtaining product information and generating layouts

[0232] The server accesses the APIs of multiple retailers based on the adjusted candidate list to obtain product information. The obtained product information includes product name, size, price, stock status, etc., and generates a 3D simulation model based on this information. This 3D simulation virtually arranges furniture in a room and is created using a 3D graphics library (e.g., Unity or Three.js).

[0233] 5. View and confirm the proposal

[0234] The terminal displays the 3D simulation model and product information received from the server to the user. The user can check the 3D simulation through the interface of their smartphone or tablet. In addition, it is possible to use AR technology to place virtual furniture in a real room and check how it will look. If the user is satisfied with the suggestions, they can add the desired products to their cart and complete the purchase process.

[0235] 6. Purchase Procedures and Transaction Notifications

[0236] The user checks the proposed furniture and appliance arrangement, and if they like it, presses the "Purchase" button to add it to their cart. The device receives the user's purchase request and payment information, encrypts it, and sends it to the server. The server accesses the seller's API to check inventory and initiate delivery procedures, and once the order is confirmed, notifies the user's device of the transaction status. This allows the user to check the transaction progress and delivery information in real time.

[0237] Specific examples

[0238] Example 1: To make the room appear larger

[0239] A user takes three photos of a 30-square-meter apartment for a single person and inputs the request to "make it look larger." The emotion engine then collects the user's facial and voice data to recognize emotions such as joy and dissatisfaction. The server analyzes the photo data and the request to identify the room's dimensions and the location of existing furniture. Based on the request, the generative AI model suggests a glass table and a small sofa, and the suggestions are adjusted based on the emotion recognition results.

[0240] The server obtains the coordinated product information from multiple sellers, generates a 3D simulation, and sends it to the device. The user reviews the 3D simulation and, if they like it, purchases the suggested product. The server then sends the order to the seller and notifies the device of the transaction status and delivery information.

[0241] These specific processes enable optimal interior design proposals that reflect the user's emotions.

[0242] The processing flow will be explained below.

[0243] Step 1:

[0244] The user takes three photos of the room. The photos are taken from different angles to gather information about the entire room. The user also inputs requests, such as "I want the room to look larger." In addition, an interface is used to send the user's facial expressions and voice to the emotion engine.

[0245] Step 2:

[0246] The device collects the user's photos, input requests, and facial and voice data, which are then converted into a format for recognition and all data is prepared for transmission.

[0247] Step 3:

[0248] The device sends the photo data, desires, and emotion data in encrypted form to the server using a secure communication protocol (e.g., HTTPS).

[0249] Step 4:

[0250] The server stores the received photo data, requests, and emotion data in a database. The photo data is input into an image analysis algorithm (e.g., OpenCV, YOLO) to extract information such as the size of the room, the layout of existing furniture, and the color of the walls.

[0251] Step 5:

[0252] The server inputs the image analysis results and the customer's requests into a generative AI model to generate an initial list of furniture and appliance candidates. For example, a glass table or a small sofa might be selected to make the room appear larger.

[0253] Step 6:

[0254] The server inputs the user's facial expression and voice data into the emotion engine to recognize the user's emotions. Emotions such as joy, dissatisfaction, and surprise are recognized, and emotion recognition results are generated.

[0255] Step 7:

[0256] The server then adjusts the list of furniture and appliance candidates generated by the generative AI model based on the emotion recognition results. For example, if the user expresses dissatisfaction, furniture with a different design or functionality will be suggested.

[0257] Step 8:

[0258] Based on the adjusted candidate list, the server accesses the APIs of multiple sellers to retrieve product information, including product name, price, size, and availability, and stores it in a database.

[0259] Step 9:

[0260] The server generates a 3D simulation model based on the acquired product information. This simulation model virtually arranges furniture in a room and is created using a 3D graphics library (e.g., Unity, Three.js).

[0261] Step 10:

[0262] The server then transmits the generated 3D simulation model and product information to the terminal using a communication protocol. The data is then packetized and transmitted again using a secure communication protocol.

[0263] Step 11:

[0264] The device analyzes the received 3D simulation model and product information and displays it to the user. The user can view the 3D simulation via a smartphone or tablet. It is also possible to use the AR function to place virtual furniture in a real room.

[0265] Step 12:

[0266] The user checks the proposed furniture and appliance arrangement, and if satisfied, presses the "Purchase" button, which adds the desired items to the cart and proceeds with the purchase.

[0267] Step 13:

[0268] The terminal receives the purchase request and payment information from the user, encrypts it, and transmits it to the server using a security protocol.

[0269] Step 14:

[0270] The server accesses the seller's API to check inventory and initiate the shipping process. Once the order is confirmed, the server calculates the commission and secures the revenue.

[0271] Step 15:

[0272] The server completes the order confirmation and delivery procedures and notifies the terminal of the transaction status, allowing the user to check the transaction progress and delivery information in real time.

[0273] Step 16:

[0274] The terminal displays transaction status and delivery information to the user and notifies them that the purchase has been completed.

[0275] Example 2

[0276] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0277] Interior design has become an important element in modern lifestyles. However, it is not easy for users to select and appropriately arrange furniture and appliances themselves. In particular, it is difficult to obtain proposals that reflect one's own feelings and desires. This results in users spending a great deal of time and effort to create a satisfying space. A system that can solve this problem and provide optimal interior proposals more quickly and efficiently is needed.

[0278] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0279] In this invention, the server includes: means for a user to acquire an image of a room and input their requirements; means for transmitting the image data and the requirements to the server; means for analyzing the image data and recognizing the size of the space and the existing furniture layout; means for generating optimal furniture and home appliance candidates based on the recognition results and the requirements using a generative AI model; means for analyzing the user's facial expressions and voice data and recognizing emotions; means for adjusting the generated furniture and home appliance candidates based on the recognized emotions; means for acquiring product information from multiple retailers; means for generating a layout based on the acquired product information and displaying a 3D simulation to the user; and means for the user to check the 3D simulation and then proceed with purchasing the products. This allows the user to quickly and effectively obtain interior design suggestions that reflect their emotions and requirements.

[0280] "User" refers to the entity that uses the system to receive interior design proposals.

[0281] "Image" refers to visual data captured of the room situation.

[0282] "Requests" refer to the requirements for interior design desired by the user.

[0283] A "terminal" is an electronic device that allows a user to take pictures and input data.

[0284] "Server" means a central processing unit for analyzing received data, generating and managing offers.

[0285] "Image data" refers to digital information obtained from images of a room taken by a user.

[0286] "Analysis" refers to the process of extracting and understanding specific information from image data, audio data, etc.

[0287] "Space" refers to the physical size or area of ​​a room.

[0288] "Existing furniture arrangement" refers to the position and layout of furniture already placed in the room.

[0289] A "generative AI model" is an algorithm that uses artificial intelligence to generate suggestions based on user requests.

[0290] "Facial expression data" refers to digital information that captures a user's facial movements and expressions.

[0291] "Voice data" refers to digital information that records a user's speech or voice.

[0292] "Emotion" refers to information that indicates the user's psychological state.

[0293] "Adjustment" is the process of modifying the proposal to better suit the user's feelings and desires.

[0294] "Retailer" refers to a company or store that offers furniture or home appliances.

[0295] "Product information" refers to detailed product information provided by a retailer.

[0296] "Layout" refers to the arrangement plan of furniture and appliances.

[0297] "3D simulation" is a technology that visualizes the placement of furniture and home appliances in a virtual space.

[0298] "Augmented reality technology" is a technology that displays virtual elements superimposed on real scenery.

[0299] The "purchase procedure" refers to the procedure for actually purchasing the proposed product.

[0300] MODE FOR CARRYING OUT THE INVENTION

[0301] The "AI Interior Coordinator" system of this invention not only proposes optimal furniture and home appliances based on user input data, but also recognizes the user's emotions and adjusts the proposals accordingly. This system is primarily composed of a terminal operated by the user, a server that performs data analysis and emotion recognition, and a product information database that utilizes the APIs of multiple retailers.

[0302] User Input and Data Submission

[0303] The user takes three photos of the room using the device's camera and inputs their preferences, such as "I want the room to look spacious" or "I want a modern design." The photos should be taken from different angles to capture the entire room. The user also collects their facial expressions and voice data through the interface. This data is encrypted and sent to the server.

[0304] Data analysis and proposal generation

[0305] The server receives the photo data, requests, facial expressions, and voice data sent from the device. The photo data is analyzed using image analysis algorithms (e.g., OpenCV, TensorFlow). As a result of the analysis, the size of the room, the layout of existing furniture, the color of the walls, etc. are identified.

[0306] Next, the server inputs the analysis results and the user's request into a generative AI model (e.g., GPT-4, DALL-E). The prompt is in the form of "Please suggest furniture arrangements that will make a 30-square-meter room appear larger," and the AI ​​model generates a list of optimal furniture and appliance candidates.

[0307] Emotion recognition and suggestion adjustment

[0308] The server analyzes the user's facial expressions and voice data using an emotion engine (e.g., Microsoft Azure Emotion API, Google Cloud Vision API). For example, it uses face-api.js to analyze facial movements and Google Speech-to-Text API to analyze voice data. The emotion engine identifies emotions such as joy, surprise, and dissatisfaction and adjusts the generated list of furniture and appliance candidates.

[0309] Obtaining product information and generating layouts

[0310] Based on the adjusted candidate list, the server accesses APIs of multiple sellers (e.g., Amazon Product Advertising API, Rakuten API) to obtain product information. The obtained product information includes product name, size, price, and stock status. A 3D simulation model is generated based on this information. This simulation is created using a 3D graphics library (e.g., Unity, Three.js).

[0311] View and review suggestions

[0312] The device displays the 3D simulation model and product information received from the server to the user. Users can check the 3D simulation through the interface of their smartphone or tablet. Furthermore, it is possible to place virtual furniture in a real room using AR technology (e.g., ARKit, ARCore) and check how it will look.

[0313] Purchase Process and Transaction Notifications

[0314] The user checks the proposed furniture and appliance arrangement, and if they like it, they press the "Purchase" button to add it to their cart. The device receives the user's purchase request and payment information, encrypts it, and sends it to the server. The server checks inventory and initiates delivery procedures via the seller's API, and once the order is confirmed, notifies the user's device of the transaction status. This allows the user to check the transaction progress and delivery information in real time.

[0315] Examples of concrete examples and prompts

[0316] Specific examples

[0317] The user takes three photos of a 30-square-meter apartment for a single person and inputs the request to "make it look larger." The emotion engine then collects the user's facial expressions and voice data to recognize emotions. The server analyzes the photo data and the request to identify the room's dimensions and the location of existing furniture. Based on the request, the generative AI model suggests a glass table or a small sofa, and the suggestions are adjusted based on the emotion recognition results. The adjusted product information is obtained from multiple sellers, and a 3D simulation is generated and sent to the device. The user reviews the 3D simulation and, if they like it, purchases the suggested product. The server then sends the order to the seller and notifies the device of the transaction status and delivery information.

[0318] Prompt Sentence Examples

[0319] "Please suggest the best furniture layout for a 30 square meter room for a single person."

[0320] "Please tell me some interior design ideas to make a room look larger."

[0321] "Please suggest some furniture with a modern design."

[0322] This system allows users to quickly and effectively obtain interior design suggestions that reflect their own feelings and desires.

[0323] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0324] Step 1:

[0325] The user takes three photos of the room using the device's camera and enters their requirements.

[0326] Input: 3 photos of the room taken by the user, request text

[0327] How it works: The user inputs their preferences, such as "I want the room to look larger" or "I want a modern design," into the device's interface. Once the photo is taken, this data is collected and encrypted by the device.

[0328] Output: Encrypted photo data, request text

[0329] Step 2:

[0330] The terminal transmits the collected photo data and the request text to the server.

[0331] Input: Encrypted photo data, request text

[0332] How it works: The device sends this data to a server over the Internet, where it is encrypted and sent using a secure protocol.

[0333] Output: Photo data received by the server, request text

[0334] Step 3:

[0335] The server then uses image analysis algorithms to analyze the received photo data and determine the size of the room and the existing furniture layout.

[0336] Input: Photo data

[0337] How it works: The server uses image analysis software such as OpenCV and TensorFlow to process the data and determine the size of the room, the color of the walls, and the layout of the existing furniture.

[0338] Output: Data about the analyzed room size, existing furniture placement, and wall color

[0339] Step 4:

[0340] The server inputs the analyzed data and the desired text into a generative AI model to generate a list of suitable furniture and home appliances.

[0341] Input: Parsed data, requested text

[0342] Specific operation: The server uses a generative AI model (e.g., GPT-4, DALL-E) to create a prompt and input it into the model. Specifically, a prompt such as "Please suggest a furniture arrangement that will make a 30-square-meter room appear larger" is used.

[0343] Output: A list of the best furniture and appliances

[0344] Step 5:

[0345] The server uses an emotion engine to analyze the user's facial expressions and voice data and recognize the user's emotions.

[0346] Input: User's facial expression data, voice data

[0347] Specific operation: The server uses an emotion engine (e.g., Microsoft Azure Emotion API, Google Cloud Vision API) to analyze the user's facial expressions and voice to identify emotions such as joy, surprise, dissatisfaction, etc. For example, facial expression analysis is performed using face-api.js, and voice analysis is performed using Google Speech-to-Text API.

[0348] Output: Recognized emotion data

[0349] Step 6:

[0350] The server adjusts the list of furniture and home appliance candidates generated based on the emotion data.

[0351] Input: List of suitable furniture and appliances, recognized emotion data

[0352] Specific behavior: When a user shows a facial expression of joy, the candidate list is adjusted to prioritize suggestions that increase that joy.

[0353] Output: A tailored list of furniture and appliances

[0354] Step 7:

[0355] The server accesses the APIs of multiple sellers to obtain product information based on the adjusted candidate list.

[0356] Input: A tailored list of furniture and appliances

[0357] Specific operation: The server uses the API to obtain the necessary product information from multiple sellers, such as product name, size, price, and availability.

[0358] Output: Retrieved product information

[0359] Step 8:

[0360] The server generates a 3D simulation model based on the acquired product information.

[0361] Input: Retrieved product information

[0362] What it does: The server uses a 3D graphics library (e.g. Unity, Three.js) to generate a simulation of placing furniture in a virtual model of the room.

[0363] Output: 3D simulation model

[0364] Step 9:

[0365] The terminal displays the 3D simulation model and product information received from the server to the user.

[0366] Input: 3D simulation model, product information

[0367] How it works: The device displays this 3D simulation to the user through the interface of a smartphone or tablet. It is also possible to place virtual furniture in a real room using AR technology (e.g., ARKit, ARCore) and see how it will look.

[0368] Output: 3D simulation and product information displayed to the user

[0369] Step 10:

[0370] The user checks the proposed furniture and appliance arrangement, and if they like it, they press the "Purchase" button to add it to their cart. The terminal encrypts the purchase request and payment information and sends it to the server.

[0371] Input: User purchase request, payment information

[0372] Specific operation: When the user presses the purchase button, the terminal encrypts the entered payment information and sends it to the server.

[0373] Output: Purchase request sent to server, payment information

[0374] Step 11:

[0375] The server will check inventory and initiate delivery procedures via the seller's API, and once the order is confirmed, will notify the user's device of the transaction status.

[0376] Input: Purchase request, payment information

[0377] Specific operation: The server checks inventory and processes delivery, and once the order is confirmed, it notifies the user's terminal of the transaction status in real time.

[0378] Output: Transaction status reported to the user's terminal

[0379] This allows users to quickly and effectively receive interior design suggestions that reflect their feelings and desires, ensuring a consistent and smooth process leading up to the purchase.

[0380] (Application example 2)

[0381] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0382] Conventional interior design suggestion systems have had the problem of making it difficult for users to find the perfect interior design that matches their own emotions and preferences. In particular, when receiving suggestions in a physical store, it is not possible to adjust the suggestions in real time, making it difficult to provide suggestions that will satisfy the user.

[0383] The specification processing by the specification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes: means for a user to take a photo of a room and input their requirements; means for transmitting the photo data and the requirements to the server; means for analyzing the photo data and recognizing the size of the room and the layout of existing furniture; means for generating optimal furniture and home appliance candidates based on the recognition results and the requirements; means for analyzing the user's facial expressions and voice data using an emotion recognition engine and recognizing emotions; means for adjusting the proposal content based on the recognized emotions; means for acquiring product information from multiple retailers; means for generating a layout based on the acquired product information and displaying a 3D simulation to the user; and means for the user to check the 3D simulation and then proceed with the purchase of the product. This enables optimal interior design proposals to be made in real time according to the user's emotions and requirements.

[0384] "Means for users to take photos of a room and input their requirements" refers to equipment and an interface that allows users to take multiple photos of a room from different angles and input their desired interior style, furniture requirements, etc.

[0385] The "means for transmitting the photo data and request to the server" refers to a function and mechanism for transmitting the photograph taken and the request entered from the user terminal to the server via a communication line.

[0386] "Means for analyzing the photographic data and recognizing the size of the room and the existing furniture layout" refers to a device and program that uses an image analysis algorithm to identify the dimensions of a room and the current furniture layout from the photographic data taken.

[0387] The "means for generating optimal furniture and home appliance candidates based on the recognition results and user requests" refers to a generation algorithm and engine for selecting and listing appropriate furniture and home appliances based on the analyzed data and the user's requests.

[0388] "Means for recognizing emotions by analyzing a user's facial expressions and voice data using an emotion recognition engine" refers to a technology that uses emotion recognition software and machine learning models to analyze a user's facial expressions and voice to identify emotions such as joy, dissatisfaction, and surprise.

[0389] The "means for adjusting the suggestions based on the recognized emotion" refers to algorithms and programs for modifying and optimizing the furniture and appliance suggestions list depending on the user's emotional state.

[0390] "Means for obtaining product information from multiple retailers" refers to systems and programs that use APIs from various retailers to collect data such as product name, price, inventory, and size.

[0391] The "means for generating a layout based on the acquired product information and displaying a 3D simulation to the user" refers to a 3D graphics engine and user interface for creating a 3D layout using the acquired product information and visually displaying it to the user.

[0392] The "means for a user who has reviewed the 3D simulation to complete the purchase of a product" refers to an interface and process that allows a user to review the 3D simulation, select a product they like, and make a payment to purchase the product.

[0393] MODE FOR CARRYING OUT THE INVENTION

[0394] System Configuration

[0395] The "interior design proposal support system" of this invention has the function of proposing optimal furniture and home appliances based on user input data, as well as the function of recognizing the user's emotions and adjusting the proposal content based on those emotions. This system is mainly composed of a terminal operated by the user, a server that performs data analysis and emotion recognition, and a product information database that utilizes the APIs of multiple retailers.

[0396] Program action

[0397] 1. User Input and Data Submission

[0398] The user takes three photos of the room and inputs their preferences. The photos should be taken from different angles to capture the entire room. The user may input preferences such as "I want the room to look spacious" or "I want a modern design." In addition, the smart glasses collect facial and voice data through the built-in camera and microphone. This data is sent to the server.

[0399] 2. Data analysis and proposal generation

[0400] The server receives the photo data, requests, facial expressions, and voice data sent from the device. The photo data is analyzed using an image analysis algorithm to determine the size of the room, the layout of existing furniture, the color of the walls, and other information. The analysis results and the user's requests are input into a generative AI model to generate a list of optimal furniture and home appliances.

[0401] 3. Emotion recognition and suggestion adjustment

[0402] The server uses an emotion recognition engine to analyze the user's facial and voice data and recognize emotions such as joy, surprise, and dissatisfaction. For example, a machine learning library is used for facial expression analysis, and a voice recognition algorithm is used for voice analysis. Based on the emotion recognition results, the generated candidate list of furniture and home appliances is adjusted according to the user's emotions.

[0403] 4. Obtaining product information and generating layouts

[0404] The server accesses the APIs of multiple retailers based on the adjusted candidate list to obtain product information. The obtained product information includes product name, size, price, stock status, etc., and generates a 3D simulation model based on this information. This 3D simulation virtually arranges furniture in a room and is created using a 3D graphics library (e.g., Unity or Three.js).

[0405] 5. View and confirm the proposal

[0406] The terminal displays the 3D simulation model and product information received from the server to the user. The user can check the 3D simulation through an interface using smart glasses. In addition, it is possible to use AR technology to place virtual furniture in a real room and check how it will look. If the user is satisfied with the suggestions, they can add the desired products to their cart and complete the purchase process.

[0407] 6. Purchase Procedures and Transaction Notifications

[0408] The user checks the proposed furniture and appliance arrangement, and if they like it, presses the "Purchase" button to add it to their cart. The device receives the user's purchase request and payment information, encrypts it, and sends it to the server. The server accesses the seller's API to check inventory and initiate the delivery process, and once the order is confirmed, notifies the user's device of the transaction status. This allows the user to check the transaction progress and delivery information in real time.

[0409] Specific examples

[0410] Example 1: To make the room appear larger

[0411] The user takes three photos of a 30-square-meter apartment for a single person and inputs the request to "make it look larger." The emotion engine then collects the user's facial and voice data to recognize emotions such as joy and dissatisfaction. The server analyzes the photo data and the request to identify the room's dimensions and the location of existing furniture. Based on the request, the generative AI model suggests a glass table or a small sofa, and the suggestions are adjusted based on the emotion recognition results. The server then obtains the adjusted product information from multiple sellers, generates a 3D simulation, and sends it to the device. The user reviews the 3D simulation and, if they like it, purchases the suggested product. The server then sends the order to the seller and notifies the device of the transaction status and delivery information.

[0412] Prompt Sentence Examples

[0413] "Please suggest interior design ideas to make a 30-square-meter single-person room look larger. Judging from the user's expressions, they seem surprised."

[0414] This system allows users to receive optimal interior design suggestions tailored to their own emotions and desires in real time at a physical store.

[0415] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0416] Step 1:

[0417] The user wears the smart glasses and takes three photos of the room. At this time, the user inputs their preferences, such as "I want the room to look spacious" or "I want a modern design," through the interface. In addition, the smart glasses' camera and microphone are used to collect the user's facial and voice data. The input data includes the room photo, the desired text, facial expression data, and voice data. The device then sends this data to the server.

[0418] Step 2:

[0419] The server uses image analysis algorithms (e.g., OpenCV, TensorFlow) to analyze the received photo data of the room. This identifies the room's size, the layout of existing furniture, the wall color, etc. The request text is also analyzed, and the server integrates this data and stores it in a database. The input data are the room photo and the user's request, and the output data are the analysis results, such as the recognized room size, furniture layout, and wall color.

[0420] Step 3:

[0421] The server uses an emotion recognition engine (e.g., Affectiva SDK, Microsoft Azure Cognitive Services) to analyze the user's facial and voice data. This identifies the user's emotions, such as joy, surprise, or dissatisfaction. The input data are facial and voice data, and the output data are the recognized emotions.

[0422] Step 4:

[0423] The server inputs prompts into the AI ​​model based on the image analysis results, emotion recognition results, and the user's requests. For example, a prompt such as "Please suggest interior design ideas to make a 30-square-meter room for a single person look larger. Judging from the user's facial expression, an expression of surprise appears." is generated and input into the AI ​​model. This generates a list of suitable furniture and home appliance candidates. The input data are the analysis results and the prompt, and the output data is a list of furniture and home appliance candidates.

[0424] Step 5:

[0425] Based on the generated candidate list of furniture and home appliances, the server accesses the APIs of multiple retailers to obtain product information (product name, price, size, and stock status). This allows the collection of the most up-to-date and detailed product information. The input data is the candidate list, and the output data is the obtained product information.

[0426] Step 6:

[0427] The server generates a 3D simulation model based on the acquired product information. This 3D model virtually arranges furniture in the user's room and is created using a 3D graphics library (e.g., Unity, Three.js). The input data is product information and furniture arrangement plans, and the output data is the 3D simulation model.

[0428] Step 7:

[0429] The terminal (smart glasses) displays the 3D simulation model and product information received from the server to the user. Through the smart glasses, the user can check the 3D simulation and see in real time how the virtual furniture will look when placed in a real room using AR technology. The input data is the 3D simulation model, and the output data is the user's visual confirmation.

[0430] Step 8:

[0431] If the user is satisfied with the proposed furniture or appliances, they press the "Purchase" button through the smart glasses interface to add them to their cart. The terminal then encrypts the purchase request and payment information and sends it to the server. The input data are the purchase request and payment information, and the output data is the encrypted transaction data.

[0432] Step 9:

[0433] The server accesses the seller's API to check inventory and initiate delivery procedures, and once the order is confirmed, it notifies the user's device of the transaction status. This allows the user to check the transaction progress and delivery information in real time. The input data is encrypted transaction data, and the output data is transaction status information.

[0434] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0435] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0436] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0437] [Second embodiment]

[0438] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0439] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0440] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0441] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0442] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0443] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0444] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0445] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0446] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0447] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0448] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0449] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0450] MODE FOR CARRYING OUT THE INVENTION

[0451] System Configuration

[0452] The "AI Interior Coordinator" system of this invention is primarily composed of a terminal operated by the user, a server that processes data, and a product information database that utilizes the APIs of multiple retailers. The system starts when the user takes a photo of the room and inputs their requirements, and then makes interior design suggestions based on those requirements.

[0453] Program processing

[0454] The program processing of this system will be explained in natural language below.

[0455] 1. User Input and Data Submission

[0456] The user first takes a photo of the room. It is recommended to take several photos from different angles to capture the entire room. The user also inputs their requests for the room layout in text format. For example, they might say, "I want the room to look spacious" or "I want it to have a modern feel." When the user enters this information into the app and presses the "Send" button, the data is sent to the server via their device.

[0457] 2. Data analysis and proposal generation

[0458] The server receives the photo data and requests sent from the device. The received photo data is first analyzed using an image analysis algorithm. This analysis identifies the size of the room, the layout of existing furniture, the color of the walls, and other factors. For example, the OpenCV library or the YOLO object detection algorithm can be used. Next, based on the user's requests, the data is input into a generative AI model (e.g., GPT-4) to generate optimal furniture and appliance candidates. For example, a glass table or a small sofa could be suggested to make the room appear larger.

[0459] 3. Obtaining product information and generating layouts

[0460] The server accesses the APIs of multiple retailers to obtain product information for the generated list of furniture and home appliance candidates. This product information includes product name, price, size, and stock information. Based on the obtained product information, the server generates the optimal layout for the user's room. This layout is generated as a 3D simulation model and provided to the user. For example, Unity or Three.js can be used to create the 3D model and virtually arrange the furniture in the room.

[0461] 4. View and confirm the proposal

[0462] The terminal displays the 3D simulation model and furniture list received from the server to the user. The user can view this 3D simulation via a smartphone or tablet. In addition, it is possible to virtually place furniture in a real room using AR technology and check how it will look. If the user is satisfied with the suggestions, they can add the desired items to their cart and complete the purchase process.

[0463] 5. Purchase Procedures and Transaction Notifications

[0464] If the user likes the suggested items, they press the "Purchase" button to add them to their cart. As part of the purchasing process, they enter their payment information and confirm the order. The terminal then sends the user's purchase request and payment information to the server. The server accesses the seller's API to check inventory and process delivery, and once the order is confirmed, it notifies the user's terminal of the transaction status. This notification allows the user to check the progress of the transaction and delivery information in real time.

[0465] Specific examples

[0466] Example 1: Make the room look larger

[0467] Suppose a user takes three photos of a 30-square-meter apartment for a single person and inputs the request to make it appear larger. The device sends this data to a server, which analyzes the photo data and identifies the dimensions of the room and the location of existing furniture. Based on the request, the generative AI model suggests a glass table and a small sofa, and obtains information on these products from multiple retailers.

[0468] The server generates a 3D simulation based on the acquired product information and sends it to the terminal. The user can check the 3D simulation and, if they like it, purchase the suggested product. The server then sends an order to the seller and notifies the terminal of the transaction status and delivery information.

[0469] The processing flow will be explained below.

[0470] Step 1:

[0471] The user takes three photos of the room. The photos are taken from different angles to capture information about the entire room. In addition, the user inputs preferences through the interface, such as "I want it to look spacious" or "I want a modern design."

[0472] Step 2:

[0473] The device receives the user's photo data and requests. This data is converted into the appropriate format and prepared for transmission. The device also encrypts the data to ensure security.

[0474] Step 3:

[0475] The device sends the encrypted photo data and request to the server using a secure communication protocol (e.g., HTTPS). It is recommended to use a checksum during data transmission to prevent errors.

[0476] Step 4:

[0477] The server stores the received photo data and requests in a database. To begin analyzing the photo data, it first sends the data to an image analysis algorithm, which uses libraries such as OpenCV and YOLO to identify the size of the room, the existing furniture arrangement, and the color of the walls.

[0478] Step 5:

[0479] The server inputs the analysis results and the user's requests into a generative AI model (e.g., GPT-4) to generate a list of suitable furniture and appliances. For example, furniture that makes a room appear larger might include a glass table, a small sofa, and a mirror with reflective materials.

[0480] Step 6:

[0481] Based on the generated candidate list, the server accesses the APIs of multiple retailers to obtain product information, including product name, size, price, stock status, etc. This information is then stored in a database.

[0482] Step 7:

[0483] The server generates a 3D simulation model based on the acquired product information. This simulation model virtually arranges furniture in the room layout and is created using a 3D graphics library (e.g., Unity or Three.js).

[0484] Step 8:

[0485] The server then sends the generated 3D simulation model and product information to the terminal, where the data is packetized in an appropriate format and transmitted again using a secure communication protocol.

[0486] Step 9:

[0487] The device analyzes the received data and displays a 3D simulation to the user, who can view the simulation through the interface of their smartphone or tablet. Additionally, by selecting the AR mode, it is possible to place virtual furniture in a real room.

[0488] Step 10:

[0489] The user checks the proposed furniture and appliance arrangement, and if they like it, they press the "Purchase" button, which adds the desired items to their cart and proceeds to the purchase process.

[0490] Step 11:

[0491] The terminal receives the purchase request and payment information from the user, encrypts it, and sends it to the server. The payment information is handled using a security protocol (e.g., TLS).

[0492] Step 12:

[0493] The server accesses the seller's API to check inventory and initiate the shipping process. Once the order is confirmed, the server calculates the commission and secures the revenue.

[0494] Step 13:

[0495] After the order is confirmed and the delivery procedure is completed, the server notifies the terminal of the transaction status, allowing the user to check the transaction progress and delivery information in real time.

[0496] Step 14:

[0497] The terminal displays transaction status and delivery information to the user and notifies them that the purchase has been completed.

[0498] Example 1

[0499] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0500] Conventional interior coordination systems have issues such as difficulty for users in visualizing the room layout, lack of information, and the time and effort required to make a purchasing decision. Furthermore, when acquiring product information from multiple retailers, data acquisition and management can be cumbersome, making it necessary to develop an efficient system. In addition, users lack a visual experience to confirm the proposals, making it difficult to visualize the actual layout.

[0501] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0502] In this invention, the server includes: a means for a user to acquire an image of a room and input their requirements; a means for transmitting the image data and the requirements to an information processing device; a means for analyzing the image data and identifying the dimensions and existing layout of the room; a means for generating optimal furniture and electronic device candidates based on the identification results and the requirements using a generative AI model; a means for acquiring product information from multiple vendors; a means for generating a layout based on the acquired product information and displaying a 3D simulation to the user; and a means for the user to check the 3D simulation and then proceed with purchasing the products. This allows the user to visually confirm the room layout, making it easier to make a purchasing decision through efficient information acquisition and a visual experience.

[0503] "User" refers to an individual or organization that uses the system.

[0504] "Image" refers to a digital photograph or graphic visual record of the exterior or interior of a room.

[0505] "Information processing device" refers to a computer system that receives, analyzes, generates, and transmits data.

[0506] "Analysis" refers to the process of determining room dimensions and furniture layout based on image data.

[0507] "Dimensions" refers to physical measurements related to the size of a room or furniture.

[0508] "Layout" refers to the relative positions of furniture and decorations within a room.

[0509] "Specific results" refers to data regarding the dimensions and layout of the room obtained through analysis.

[0510] "Candidates" refers to a list of furniture and electronic devices that are optimally selected based on the user's requirements.

[0511] A "generative AI model" refers to an algorithm that uses machine learning to suggest furniture and electronic devices that meet user needs.

[0512] "Product information" refers to data such as the name, price, size, and inventory of furniture and electronic devices.

[0513] "Layout" refers to a virtual layout of furniture and decorations in a room.

[0514] "3D simulation" refers to a model that visually reproduces the furniture arrangement in a room using 3D computer graphics.

[0515] "Augmented reality technology" refers to technology that overlays digital information onto real space.

[0516] "Retailer" refers to a business or store that sells furniture or electronics.

[0517] "Checkout" refers to the process by which a user selects the desired product, enters payment information, and confirms the order.

[0518] "Order" refers to a transaction request for an item that a user wishes to purchase.

[0519] MODE FOR CARRYING OUT THE INVENTION

[0520] The "AI Interior Coordinator" system of this invention begins when a user acquires a photo of a room and inputs their requirements. The photo data taken by the user and the input requirements are sent to an information processing device (server), which then performs a series of data analyses and proposes interior designs. Specific embodiments are described below.

[0521] User Actions

[0522] First, the user launches a dedicated application and takes photos of the room from multiple angles. For example, it is recommended to take four photos: one from the front, one from the left, one from the right, and one from the back. The user also enters their room-related requirements in text format (e.g., "I want the room to look spacious" or "I want a modern feel"). Once these are entered, the user presses the "Send" button in the application. This sends the photo data and requirements from the device to the information processing device (server).

[0523] Server Operation

[0524] The server receives the submitted photo data and request. First, it analyzes the photo data using the OpenCV library and the YOLO object detection algorithm to determine the room dimensions and the layout of existing furniture. Specifically, it uses the YOLO algorithm to identify furniture and obtain the position and size of each piece of furniture. Next, it uses a generative AI model (e.g., GPT-4) based on the user's request to generate optimal furniture and electronic device candidates. At this time, it inputs data into the generative AI model using prompt statements.

[0525] Prompt Sentence Examples

[0526] "Please suggest the best interior coordination to make a 30 square meter room look larger. Using glass furniture and a small sofa."

[0527] "Can you give me some ideas for modernizing this room? The current furniture arrangement is included in the images."

[0528] Obtaining product information

[0529] Based on the generated candidate list of furniture and electronic devices, the server accesses the APIs of multiple vendors to obtain corresponding product information. The obtained product information includes product name, price, size, stock information, etc. For example, the product information is obtained using the API of an internet shopping mall.

[0530] Layout generation and display

[0531] The server uses Unity or Three.js to generate a 3D simulation model based on the acquired product information and create an optimal layout. These 3D simulations include furniture virtually placed in the room. This model is sent to the device so that the user can visually check it.

[0532] Proposal confirmation and purchase process

[0533] The device displays the 3D simulation model and product list received from the server to the user. The user can view this 3D simulation through the app. Augmented reality technology using the smartphone camera also allows the user to place virtual furniture in a real room and simulate how it will look. If the user is satisfied with the suggestions, they can add the desired products to their cart and proceed with the purchase process.

[0534] Purchase Process and Transaction Notifications

[0535] The user presses the "Purchase" button, adds the product to the cart, enters payment information, and confirms the order. The terminal sends the user's purchase request and payment information to the server. The server accesses the vendor's API to check inventory and process delivery, and once the order is confirmed, it notifies the user's terminal of the transaction status and delivery information. This allows the user to check the progress of the transaction and product delivery information in real time.

[0536] In this way, the "AI interior coordinator" of the present invention provides users with an efficient and intuitive interior coordination proposal and purchasing process.

[0537] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0538] Step 1: User Input and Data Submission

[0539] The user uses a dedicated application to take photos of the room from multiple angles. As input data, they take photos of the front, left, right, and back of the room, and enter their requests in text format, such as "I want the room to look spacious" or "I want a modern feel." When they press the "Send" button, this data is sent from the device to the server. As output, the photo data and the requests in text format are sent to the server.

[0540] Step 2: Receiving and initial processing of data

[0541] The server receives the photo data and requests sent from the terminal. As input data, it receives the photo data and requests sent from the user. Based on this, it performs initial processing of the data and prepares it for data analysis. As output, it generates the photo data and requests converted into the format required for analysis.

[0542] Step 3: Data analysis

[0543] The server analyzes the photo data using the OpenCV library and the YOLO object detection algorithm. It uses the photo data formatted for processing as input data. Based on this, it identifies furniture and room dimensions in the image and extracts attribute information (position, size, color, etc.) of the identified objects. The output is the room dimensions and the existing furniture layout.

[0544] Step 4: Proposal generation using generative AI models

[0545] The server inputs the identification results and the user's requests into a generative AI model (e.g., GPT-4) and generates optimal furniture and electronic device candidates. The input data includes the room dimensions, existing furniture layout, and the user's requests (in text format). Based on the generated prompt, the AI ​​model analyzes and outputs specific furniture and interior design suggestions. The output is a list of suggested furniture and electronic devices.

[0546] Step 5: Obtaining product information using the API

[0547] The server accesses the APIs of multiple vendors to obtain product information based on the generated list of furniture and electronic device candidates. The server uses the list of proposed furniture and electronic devices as input data. Based on this, it makes API requests to obtain product names, prices, sizes, stock information, etc. The server generates a list of the obtained product information as output.

[0548] Step 6: Generate a 3D simulation model

[0549] The server uses Unity or Three.js to generate a 3D simulation model based on the acquired product information. The input data is a list of product information and the dimensions of the existing room. Based on this, it creates an optimal layout and generates a 3D model of the furniture virtually placed in the room. The generated 3D simulation model is obtained as the output.

[0550] Step 7: View and Review the Simulation

[0551] The device displays the 3D simulation model and product list received from the server to the user. The 3D simulation model sent from the server is used as input data. The user can visually check this 3D simulation through the app. It is also possible to use augmented reality technology to place virtual furniture in a real room and simulate how it will look. The output is a visual 3D simulation that the user can check.

[0552] Step 8: Checkout

[0553] Based on the suggestions they like, the user presses the "Purchase" button, adds the product to their cart, enters payment information, and confirms the order. The input data is product information and payment information. Based on this, the purchase procedure is carried out. As an output, information indicating that the user has completed the purchase procedure is sent to the server.

[0554] Step 9: Transaction Notification

[0555] The server accesses the seller's API based on the user's purchase request and payment information to check inventory and process delivery. Detailed purchase process information is used as input data. Once the order is confirmed, transaction status and delivery information is sent to the user's device. As an output, the user receives notifications that allow them to check the transaction progress and delivery information in real time.

[0556] (Application example 1)

[0557] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0558] In today's brick-and-mortar stores, customers often have difficulty visualizing the product in its actual placement when selecting the right interior product for their home. Traditionally, purchasing decisions are often made solely based on catalogs and displays, increasing the risk of making the wrong choice. Similarly, online shopping makes it difficult to confirm the product's suitability in the actual room environment. This leads to an increase in returns and exchanges after purchase, lowering customer satisfaction and increasing operational costs.

[0559] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0560] In this invention, the server includes: a means for a user to take an image of a room and input their preferences; a means for transmitting the image data and preferences to the server; a means for analyzing the image data to recognize the size of the room and the existing item layout; a means for generating optimal device and tool candidates based on the recognition results and the user's preferences; a means for acquiring product information from multiple sellers; a means for generating a layout based on the acquired product information and displaying a 3D simulation to the user; a means for the user to check the 3D simulation and then complete a product purchase procedure; and a means for displaying virtual items arranged in a real space and using augmented reality technology to confirm the display. This allows users to visually confirm the actual layout of virtual interior items in a physical store or at home. Furthermore, customers can accurately confirm the suitability of products before purchase, preventing incorrect selections and reducing returns and exchanges.

[0561] A "user" refers to an individual who uses the system to take pictures and input their preferences to coordinate the interior of a room.

[0562] "Image data" refers to photographic data of a room taken by a user, and includes information such as the size of the room and the layout of existing items.

[0563] "Wishes" are requests that the user inputs regarding the interior of the room, and refer to specific requests such as "I want the room to look spacious."

[0564] "Server" refers to a computer system that receives image data and requests sent by users and performs analysis and data processing.

[0565] "Analysis" refers to the process of identifying the size of a room and the layout of existing items based on image data, and is done using computer vision techniques and object recognition algorithms.

[0566] "Device and tool candidates" refers to a list of optimal interior products suggested by the system based on the user's preferences and analysis results.

[0567] "Sellers" refer to companies and stores that sell interior products, and the system obtains product information from them.

[0568] "Product Information" refers to detailed data about interior products provided by sellers, such as product name, price, size, and inventory information.

[0569] "Layout" refers to the layout of interior products virtually arranged in a room based on the acquired product information.

[0570] "Three-dimensional simulation" refers to three-dimensional simulation data for visually displaying the generated layout to the user.

[0571] "Augmented reality technology" refers to a technology that displays virtual objects superimposed on a real-world environment, allowing users to visually check virtual interior products in real space.

[0572] "Purchase procedure" refers to the series of operations that a user performs to actually purchase interior products suggested through the system.

[0573] This invention provides an "interior AR shopping assistant" system that allows users to try out interior products in real time in a "real store." To realize this system, the following hardware and software are required:

[0574] Hardware and software used

[0575] Hardware: Requires a smartphone (camera, display, internet connection).

[0576] Software: OpenCV, YOLO (object detection), Unity (3D simulation), generative AI model (GPT-4API), etc. are required.

[0577] System configuration and operation

[0578] 1. User Action:

[0579] Users use their smartphones to take pictures of their room from multiple angles and input their desired interior style and requirements in text format, such as "I want the room to look spacious" or "I want a modern feel."

[0580] 2. Data transmission:

[0581] The user's terminal transmits the captured image data and the desired text data to the server.

[0582] 3. Analysis of imaging data:

[0583] The server analyzes the received image data using OpenCV and YOLO to determine the size of the room and the layout of existing items. This analysis determines the floor area, the location of walls, and how existing furniture is arranged.

[0584] 4. Generate optimal furniture candidates:

[0585] The server uses a generative AI model (GPT-4) to generate optimal interior product candidates based on the analyzed data and the user's preferences. For example, if the user desires a "modern taste" for a living room measuring approximately 20 square meters, the generative AI model will suggest a simple white sofa and a glass coffee table.

[0586] Example prompt sentence:

[0587] Room features: 20 square meters, white walls, existing furniture: dark wooden table and bookshelf.

[0588] User's request: I want a modern and simple interior.

[0589] 5. Obtaining product information:

[0590] The server obtains information about the interior products listed as candidates through APIs of multiple sellers, including product names, prices, sizes, and stock information.

[0591] 6. Generation of 3D simulations:

[0592] The server uses Unity to create a 3D simulation based on the acquired product information, and the simulation is generated in a virtual room with the proposed furniture arranged.

[0593] 7. View the simulation:

[0594] The user's device displays the 3D simulation sent from the server and uses augmented reality (AR) technology to overlay virtual interior products onto the real space, allowing the user to visually confirm how the virtual interior products will actually look when placed in a physical store or at home.

[0595] Specific examples

[0596] For example, if a user wants to make a 30-square-meter living room look modern and spacious, the system analyzes images taken from various angles of the room and generates suggestions such as a white sofa and a glass table. Product information is then acquired and a 3D simulation is generated based on the acquired information. The user can then virtually place these suggested furniture items in their real room using their smartphone, see how they will look, and then purchase the items they like.

[0597] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0598] Step 1:

[0599] The user takes a picture of the room and inputs their preferences.

[0600] The user uses a smartphone to take multiple images of the room from different angles. They also input their preferences for the room's interior in text format. For example, they might input a preference such as "I want the room to look spacious." The input data is saved as image data and text data.

[0601] Step 2:

[0602] The device sends the photo data and your request to the server.

[0603] The device sends the image data and text data the user has entered to the server, which then securely transfers the data over the Internet and stores it on the server.

[0604] Step 3:

[0605] The server analyzes the photo data and recognizes the size of the room and the layout of existing items.

[0606] The server analyzes the received image data using OpenCV and YOLO object detection algorithms. This analysis identifies the size of the room, the location of walls, and the layout of existing furniture. For example, the floor area and furniture locations are automatically calculated.

[0607] Step 4:

[0608] The server generates optimal device and tool candidates based on the recognition results and user preferences.

[0609] The server uses a generative AI model (GPT-4) based on the analysis results and the user's preferences to generate optimal device and tool candidates. The analysis results and the user's preferences are input into the prompt text.

[0610] Example prompt sentence:

[0611] Room features: 20 square meters, white walls, existing furniture: dark wooden table and bookshelf.

[0612] User's wish: A modern and simple interior.

[0613] Based on this prompt, the generative AI model suggests specific interior product options, such as a simple white sofa or a glass table.

[0614] Step 5:

[0615] The server retrieves product information from multiple sellers

[0616] Based on the generated list of furniture and tool candidates, the server accesses the APIs of multiple sellers to obtain product information. The obtained product information includes product name, price, size, stock information, etc. This information is imported into the server as structured data in JSON format, etc.

[0617] Step 6:

[0618] The server generates a layout based on the product information acquired and creates a 3D simulation.

[0619] The server uses Unity to create a 3D simulation based on the acquired product information. The placement data is constructed as a 3D model, and a simulation of the virtual furniture placement in the room is generated. This simulation data is visualized for the user.

[0620] Step 7:

[0621] The server sends the 3D simulation to the terminal, which displays it.

[0622] The server sends the generated 3D simulation to the user's device, which then displays the received simulation data, allowing the user to use augmented reality (AR) technology to overlay virtual furniture onto the real space, creating a visual experience that makes it seem as if the new furniture is actually placed in the room.

[0623] Step 8:

[0624] The user completes the purchase process

[0625] The user checks the 3D simulation and selects the product they wish to purchase. As part of the purchase process, they enter payment information and confirm the order. The terminal sends the user's purchase request and payment information to the server, which then accesses the seller's API to check inventory and process delivery. Once the order is confirmed, the user's terminal is notified of the transaction status and delivery information.

[0626] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0627] MODE FOR CARRYING OUT THE INVENTION

[0628] System Configuration

[0629] The "AI Interior Coordinator" system of this invention not only proposes optimal furniture and home appliances based on user input data, but also recognizes the user's emotions and adjusts the proposals accordingly. This system is primarily composed of a terminal operated by the user, a server that performs data analysis and emotion recognition, and a product information database that utilizes the APIs of multiple retailers.

[0630] Program processing

[0631] The program processing of this system will be explained in natural language below.

[0632] 1. User Input and Data Submission

[0633] The user takes three photos of the room and inputs their requirements. It is recommended that the photos be taken from different angles to capture the entire room. The user inputs their requirements, such as "I want the room to look spacious" or "I want a modern design." In addition, the user collects their facial expressions and voice data through the interface. This data is sent to the emotion engine.

[0634] 2. Data analysis and proposal generation

[0635] The server receives the photo data, requests, facial expressions, and voice data sent from the device. The photo data is analyzed using an image analysis algorithm to determine the size of the room, the layout of existing furniture, the color of the walls, etc. The analysis results and the user's requests are input into a generative AI model to generate a list of optimal furniture and home appliances.

[0636] 3. Emotion recognition and suggestion adjustment

[0637] The server uses an emotion engine to analyze the user's facial and voice data and recognize emotions such as joy, surprise, and dissatisfaction. For example, a machine learning library is used for facial expression analysis, and a voice recognition algorithm is used for voice analysis. Based on the emotion recognition results, the generated candidate list of furniture and home appliances is re-adjusted according to the user's emotions.

[0638] 4. Obtaining product information and generating layouts

[0639] The server accesses the APIs of multiple retailers based on the adjusted candidate list to obtain product information. The obtained product information includes product name, size, price, stock status, etc., and generates a 3D simulation model based on this information. This 3D simulation virtually arranges furniture in a room and is created using a 3D graphics library (e.g., Unity or Three.js).

[0640] 5. View and confirm the proposal

[0641] The terminal displays the 3D simulation model and product information received from the server to the user. The user can check the 3D simulation through the interface of their smartphone or tablet. In addition, it is possible to use AR technology to place virtual furniture in a real room and check how it will look. If the user is satisfied with the suggestions, they can add the desired products to their cart and complete the purchase process.

[0642] 6. Purchase Procedures and Transaction Notifications

[0643] The user checks the proposed furniture and appliance arrangement, and if they like it, presses the "Purchase" button to add it to their cart. The device receives the user's purchase request and payment information, encrypts it, and sends it to the server. The server accesses the seller's API to check inventory and initiate delivery procedures, and once the order is confirmed, notifies the user's device of the transaction status. This allows the user to check the transaction progress and delivery information in real time.

[0644] Specific examples

[0645] Example 1: To make the room appear larger

[0646] A user takes three photos of a 30-square-meter apartment for a single person and inputs the request to "make it look larger." The emotion engine then collects the user's facial and voice data to recognize emotions such as joy and dissatisfaction. The server analyzes the photo data and the request to identify the room's dimensions and the location of existing furniture. Based on the request, the generative AI model suggests a glass table and a small sofa, and the suggestions are adjusted based on the emotion recognition results.

[0647] The server obtains the coordinated product information from multiple sellers, generates a 3D simulation, and sends it to the device. The user reviews the 3D simulation and, if they like it, purchases the suggested product. The server then sends the order to the seller and notifies the device of the transaction status and delivery information.

[0648] These specific processes enable optimal interior design proposals that reflect the user's emotions.

[0649] The processing flow will be explained below.

[0650] Step 1:

[0651] The user takes three photos of the room. The photos are taken from different angles to gather information about the entire room. The user also inputs requests, such as "I want the room to look larger." In addition, an interface is used to send the user's facial expressions and voice to the emotion engine.

[0652] Step 2:

[0653] The device collects the user's photos, input requests, and facial and voice data, which are then converted into a format for recognition and all data is prepared for transmission.

[0654] Step 3:

[0655] The device sends the photo data, desires, and emotion data in encrypted form to the server using a secure communication protocol (e.g., HTTPS).

[0656] Step 4:

[0657] The server stores the received photo data, requests, and emotion data in a database. The photo data is input into an image analysis algorithm (e.g., OpenCV, YOLO) to extract information such as the size of the room, the layout of existing furniture, and the color of the walls.

[0658] Step 5:

[0659] The server inputs the image analysis results and the customer's requests into a generative AI model to generate an initial list of furniture and appliance candidates. For example, a glass table or a small sofa might be selected to make the room appear larger.

[0660] Step 6:

[0661] The server inputs the user's facial expression and voice data into the emotion engine to recognize the user's emotions. Emotions such as joy, dissatisfaction, and surprise are recognized, and emotion recognition results are generated.

[0662] Step 7:

[0663] The server then adjusts the list of furniture and appliance candidates generated by the generative AI model based on the emotion recognition results. For example, if the user expresses dissatisfaction, furniture with a different design or functionality will be suggested.

[0664] Step 8:

[0665] Based on the adjusted candidate list, the server accesses the APIs of multiple sellers to retrieve product information, including product name, price, size, and availability, and stores it in a database.

[0666] Step 9:

[0667] The server generates a 3D simulation model based on the acquired product information. This simulation model virtually arranges furniture in a room and is created using a 3D graphics library (e.g., Unity, Three.js).

[0668] Step 10:

[0669] The server then transmits the generated 3D simulation model and product information to the terminal using a communication protocol. The data is then packetized and transmitted again using a secure communication protocol.

[0670] Step 11:

[0671] The device analyzes the received 3D simulation model and product information and displays it to the user. The user can view the 3D simulation via a smartphone or tablet. It is also possible to use the AR function to place virtual furniture in a real room.

[0672] Step 12:

[0673] The user checks the proposed furniture and appliance arrangement, and if satisfied, presses the "Purchase" button, which adds the desired items to the cart and proceeds with the purchase.

[0674] Step 13:

[0675] The terminal receives the purchase request and payment information from the user, encrypts it, and transmits it to the server using a security protocol.

[0676] Step 14:

[0677] The server accesses the seller's API to check inventory and initiate the shipping process. Once the order is confirmed, the server calculates the commission and secures the revenue.

[0678] Step 15:

[0679] The server completes the order confirmation and delivery procedures and notifies the terminal of the transaction status, allowing the user to check the transaction progress and delivery information in real time.

[0680] Step 16:

[0681] The terminal displays transaction status and delivery information to the user and notifies them that the purchase has been completed.

[0682] Example 2

[0683] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0684] Interior design has become an important element in modern lifestyles. However, it is not easy for users to select and appropriately arrange furniture and appliances themselves. In particular, it is difficult to obtain proposals that reflect one's own feelings and desires. This results in users spending a great deal of time and effort to create a satisfying space. A system that can solve this problem and provide optimal interior proposals more quickly and efficiently is needed.

[0685] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0686] In this invention, the server includes: means for a user to acquire an image of a room and input their requirements; means for transmitting the image data and the requirements to the server; means for analyzing the image data and recognizing the size of the space and the existing furniture layout; means for generating optimal furniture and home appliance candidates based on the recognition results and the requirements using a generative AI model; means for analyzing the user's facial expressions and voice data and recognizing emotions; means for adjusting the generated furniture and home appliance candidates based on the recognized emotions; means for acquiring product information from multiple retailers; means for generating a layout based on the acquired product information and displaying a 3D simulation to the user; and means for the user to check the 3D simulation and then proceed with purchasing the products. This allows the user to quickly and effectively obtain interior design suggestions that reflect their emotions and requirements.

[0687] "User" refers to the entity that uses the system to receive interior design proposals.

[0688] "Image" refers to visual data captured of the room situation.

[0689] "Requests" refer to the requirements for interior design desired by the user.

[0690] A "terminal" is an electronic device that allows a user to take pictures and input data.

[0691] "Server" means a central processing unit for analyzing received data, generating and managing offers.

[0692] "Image data" refers to digital information obtained from images of a room taken by a user.

[0693] "Analysis" refers to the process of extracting and understanding specific information from image data, audio data, etc.

[0694] "Space" refers to the physical size or area of ​​a room.

[0695] "Existing furniture arrangement" refers to the position and layout of furniture already placed in the room.

[0696] A "generative AI model" is an algorithm that uses artificial intelligence to generate suggestions based on user requests.

[0697] "Facial expression data" refers to digital information that captures a user's facial movements and expressions.

[0698] "Voice data" refers to digital information that records a user's speech or voice.

[0699] "Emotion" refers to information that indicates the user's psychological state.

[0700] "Adjustment" is the process of modifying the proposal to better suit the user's feelings and desires.

[0701] "Retailer" refers to a company or store that offers furniture or home appliances.

[0702] "Product information" refers to detailed product information provided by a retailer.

[0703] "Layout" refers to the arrangement plan of furniture and appliances.

[0704] "3D simulation" is a technology that visualizes the placement of furniture and home appliances in a virtual space.

[0705] "Augmented reality technology" is a technology that displays virtual elements superimposed on real scenery.

[0706] The "purchase procedure" refers to the procedure for actually purchasing the proposed product.

[0707] MODE FOR CARRYING OUT THE INVENTION

[0708] The "AI Interior Coordinator" system of this invention not only proposes optimal furniture and home appliances based on user input data, but also recognizes the user's emotions and adjusts the proposals accordingly. This system is primarily composed of a terminal operated by the user, a server that performs data analysis and emotion recognition, and a product information database that utilizes the APIs of multiple retailers.

[0709] User Input and Data Submission

[0710] The user takes three photos of the room using the device's camera and inputs their preferences, such as "I want the room to look spacious" or "I want a modern design." The photos should be taken from different angles to capture the entire room. The user also collects their facial expressions and voice data through the interface. This data is encrypted and sent to the server.

[0711] Data analysis and proposal generation

[0712] The server receives the photo data, requests, facial expressions, and voice data sent from the device. The photo data is analyzed using image analysis algorithms (e.g., OpenCV, TensorFlow). As a result of the analysis, the size of the room, the layout of existing furniture, the color of the walls, etc. are identified.

[0713] Next, the server inputs the analysis results and the user's request into a generative AI model (e.g., GPT-4, DALL-E). The prompt is in the form of "Please suggest furniture arrangements that will make a 30-square-meter room appear larger," and the AI ​​model generates a list of optimal furniture and appliance candidates.

[0714] Emotion recognition and suggestion adjustment

[0715] The server analyzes the user's facial expressions and voice data using an emotion engine (e.g., Microsoft Azure Emotion API, Google Cloud Vision API). For example, it uses face-api.js to analyze facial movements and Google Speech-to-Text API to analyze voice data. The emotion engine identifies emotions such as joy, surprise, and dissatisfaction and adjusts the generated list of furniture and appliance candidates.

[0716] Obtaining product information and generating layouts

[0717] Based on the adjusted candidate list, the server accesses APIs of multiple sellers (e.g., Amazon Product Advertising API, Rakuten API) to obtain product information. The obtained product information includes product name, size, price, and stock status. A 3D simulation model is generated based on this information. This simulation is created using a 3D graphics library (e.g., Unity, Three.js).

[0718] View and review suggestions

[0719] The device displays the 3D simulation model and product information received from the server to the user. Users can check the 3D simulation through the interface of their smartphone or tablet. Furthermore, it is possible to place virtual furniture in a real room using AR technology (e.g., ARKit, ARCore) and check how it will look.

[0720] Purchase Process and Transaction Notifications

[0721] The user checks the proposed furniture and appliance arrangement, and if they like it, they press the "Purchase" button to add it to their cart. The device receives the user's purchase request and payment information, encrypts it, and sends it to the server. The server checks inventory and initiates delivery procedures via the seller's API, and once the order is confirmed, notifies the user's device of the transaction status. This allows the user to check the transaction progress and delivery information in real time.

[0722] Examples of concrete examples and prompts

[0723] Specific examples

[0724] The user takes three photos of a 30-square-meter apartment for a single person and inputs the request to "make it look larger." The emotion engine then collects the user's facial expressions and voice data to recognize emotions. The server analyzes the photo data and the request to identify the room's dimensions and the location of existing furniture. Based on the request, the generative AI model suggests a glass table or a small sofa, and the suggestions are adjusted based on the emotion recognition results. The adjusted product information is obtained from multiple sellers, and a 3D simulation is generated and sent to the device. The user reviews the 3D simulation and, if they like it, purchases the suggested product. The server then sends the order to the seller and notifies the device of the transaction status and delivery information.

[0725] Prompt Sentence Examples

[0726] "Please suggest the best furniture layout for a 30 square meter room for a single person."

[0727] "Please tell me some interior design ideas to make a room look larger."

[0728] "Please suggest some furniture with a modern design."

[0729] This system allows users to quickly and effectively obtain interior design suggestions that reflect their own feelings and desires.

[0730] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0731] Step 1:

[0732] The user takes three photos of the room using the device's camera and enters their requirements.

[0733] Input: 3 photos of the room taken by the user, request text

[0734] How it works: The user inputs their preferences, such as "I want the room to look larger" or "I want a modern design," into the device's interface. Once the photo is taken, this data is collected and encrypted by the device.

[0735] Output: Encrypted photo data, request text

[0736] Step 2:

[0737] The terminal transmits the collected photo data and the request text to the server.

[0738] Input: Encrypted photo data, request text

[0739] How it works: The device sends this data to a server over the Internet, where it is encrypted and sent using a secure protocol.

[0740] Output: Photo data received by the server, request text

[0741] Step 3:

[0742] The server then uses image analysis algorithms to analyze the received photo data and determine the size of the room and the existing furniture layout.

[0743] Input: Photo data

[0744] How it works: The server uses image analysis software such as OpenCV and TensorFlow to process the data and determine the size of the room, the color of the walls, and the layout of the existing furniture.

[0745] Output: Data about the analyzed room size, existing furniture placement, and wall color

[0746] Step 4:

[0747] The server inputs the analyzed data and the desired text into a generative AI model to generate a list of suitable furniture and home appliances.

[0748] Input: Parsed data, requested text

[0749] Specific operation: The server uses a generative AI model (e.g., GPT-4, DALL-E) to create a prompt and input it into the model. Specifically, a prompt such as "Please suggest a furniture arrangement that will make a 30-square-meter room appear larger" is used.

[0750] Output: A list of the best furniture and appliances

[0751] Step 5:

[0752] The server uses an emotion engine to analyze the user's facial expressions and voice data and recognize the user's emotions.

[0753] Input: User's facial expression data, voice data

[0754] Specific operation: The server uses an emotion engine (e.g., Microsoft Azure Emotion API, Google Cloud Vision API) to analyze the user's facial expressions and voice to identify emotions such as joy, surprise, dissatisfaction, etc. For example, facial expression analysis is performed using face-api.js, and voice analysis is performed using Google Speech-to-Text API.

[0755] Output: Recognized emotion data

[0756] Step 6:

[0757] The server adjusts the list of furniture and home appliance candidates generated based on the emotion data.

[0758] Input: List of suitable furniture and appliances, recognized emotion data

[0759] Specific behavior: When a user shows a facial expression of joy, the candidate list is adjusted to prioritize suggestions that increase that joy.

[0760] Output: A tailored list of furniture and appliances

[0761] Step 7:

[0762] The server accesses the APIs of multiple sellers to obtain product information based on the adjusted candidate list.

[0763] Input: A tailored list of furniture and appliances

[0764] Specific operation: The server uses the API to obtain the necessary product information from multiple sellers, such as product name, size, price, and availability.

[0765] Output: Retrieved product information

[0766] Step 8:

[0767] The server generates a 3D simulation model based on the acquired product information.

[0768] Input: Retrieved product information

[0769] What it does: The server uses a 3D graphics library (e.g. Unity, Three.js) to generate a simulation of placing furniture in a virtual model of the room.

[0770] Output: 3D simulation model

[0771] Step 9:

[0772] The terminal displays the 3D simulation model and product information received from the server to the user.

[0773] Input: 3D simulation model, product information

[0774] How it works: The device displays this 3D simulation to the user through the interface of a smartphone or tablet. It is also possible to place virtual furniture in a real room using AR technology (e.g., ARKit, ARCore) and see how it will look.

[0775] Output: 3D simulation and product information displayed to the user

[0776] Step 10:

[0777] The user checks the proposed furniture and appliance arrangement, and if they like it, they press the "Purchase" button to add it to their cart. The terminal encrypts the purchase request and payment information and sends it to the server.

[0778] Input: User purchase request, payment information

[0779] Specific operation: When the user presses the purchase button, the terminal encrypts the entered payment information and sends it to the server.

[0780] Output: Purchase request sent to server, payment information

[0781] Step 11:

[0782] The server will check inventory and initiate delivery procedures via the seller's API, and once the order is confirmed, will notify the user's device of the transaction status.

[0783] Input: Purchase request, payment information

[0784] Specific operation: The server checks inventory and processes delivery, and once the order is confirmed, it notifies the user's terminal of the transaction status in real time.

[0785] Output: Transaction status reported to the user's terminal

[0786] This allows users to quickly and effectively receive interior design suggestions that reflect their feelings and desires, ensuring a consistent and smooth process leading up to the purchase.

[0787] (Application example 2)

[0788] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0789] Conventional interior design suggestion systems have had the problem of making it difficult for users to find the perfect interior design that matches their own emotions and preferences. In particular, when receiving suggestions in a physical store, it is not possible to adjust the suggestions in real time, making it difficult to provide suggestions that will satisfy the user.

[0790] The specification processing by the specification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes: means for a user to take a photo of a room and input their requirements; means for transmitting the photo data and the requirements to the server; means for analyzing the photo data and recognizing the size of the room and the layout of existing furniture; means for generating optimal furniture and home appliance candidates based on the recognition results and the requirements; means for analyzing the user's facial expressions and voice data using an emotion recognition engine and recognizing emotions; means for adjusting the proposal content based on the recognized emotions; means for acquiring product information from multiple retailers; means for generating a layout based on the acquired product information and displaying a 3D simulation to the user; and means for the user to check the 3D simulation and then proceed with the purchase of the product. This enables optimal interior design proposals to be made in real time according to the user's emotions and requirements.

[0791] "Means for users to take photos of a room and input their requirements" refers to equipment and an interface that allows users to take multiple photos of a room from different angles and input their desired interior style, furniture requirements, etc.

[0792] The "means for transmitting the photo data and request to the server" refers to a function and mechanism for transmitting the photograph taken and the request entered from the user terminal to the server via a communication line.

[0793] "Means for analyzing the photographic data and recognizing the size of the room and the existing furniture layout" refers to a device and program that uses an image analysis algorithm to identify the dimensions of a room and the current furniture layout from the photographic data taken.

[0794] The "means for generating optimal furniture and home appliance candidates based on the recognition results and user requests" refers to a generation algorithm and engine for selecting and listing appropriate furniture and home appliances based on the analyzed data and the user's requests.

[0795] "Means for recognizing emotions by analyzing a user's facial expressions and voice data using an emotion recognition engine" refers to a technology that uses emotion recognition software and machine learning models to analyze a user's facial expressions and voice to identify emotions such as joy, dissatisfaction, and surprise.

[0796] The "means for adjusting the suggestions based on the recognized emotion" refers to algorithms and programs for modifying and optimizing the furniture and appliance suggestions list depending on the user's emotional state.

[0797] "Means for obtaining product information from multiple retailers" refers to systems and programs that use APIs from various retailers to collect data such as product name, price, inventory, and size.

[0798] The "means for generating a layout based on the acquired product information and displaying a 3D simulation to the user" refers to a 3D graphics engine and user interface for creating a 3D layout using the acquired product information and visually displaying it to the user.

[0799] The "means for a user who has reviewed the 3D simulation to complete the purchase of a product" refers to an interface and process that allows a user to review the 3D simulation, select a product they like, and make a payment to purchase the product.

[0800] MODE FOR CARRYING OUT THE INVENTION

[0801] System Configuration

[0802] The "interior design proposal support system" of this invention has the function of proposing optimal furniture and home appliances based on user input data, as well as the function of recognizing the user's emotions and adjusting the proposal content based on those emotions. This system is mainly composed of a terminal operated by the user, a server that performs data analysis and emotion recognition, and a product information database that utilizes the APIs of multiple retailers.

[0803] Program action

[0804] 1. User Input and Data Submission

[0805] The user takes three photos of the room and inputs their preferences. The photos should be taken from different angles to capture the entire room. The user may input preferences such as "I want the room to look spacious" or "I want a modern design." In addition, the smart glasses collect facial and voice data through the built-in camera and microphone. This data is sent to the server.

[0806] 2. Data analysis and proposal generation

[0807] The server receives the photo data, requests, facial expressions, and voice data sent from the device. The photo data is analyzed using an image analysis algorithm to determine the size of the room, the layout of existing furniture, the color of the walls, and other information. The analysis results and the user's requests are input into a generative AI model to generate a list of optimal furniture and home appliances.

[0808] 3. Emotion recognition and suggestion adjustment

[0809] The server uses an emotion recognition engine to analyze the user's facial and voice data and recognize emotions such as joy, surprise, and dissatisfaction. For example, a machine learning library is used for facial expression analysis, and a voice recognition algorithm is used for voice analysis. Based on the emotion recognition results, the generated candidate list of furniture and home appliances is adjusted according to the user's emotions.

[0810] 4. Obtaining product information and generating layouts

[0811] The server accesses the APIs of multiple retailers based on the adjusted candidate list to obtain product information. The obtained product information includes product name, size, price, stock status, etc., and generates a 3D simulation model based on this information. This 3D simulation virtually arranges furniture in a room and is created using a 3D graphics library (e.g., Unity or Three.js).

[0812] 5. View and confirm the proposal

[0813] The terminal displays the 3D simulation model and product information received from the server to the user. The user can check the 3D simulation through an interface using smart glasses. In addition, it is possible to use AR technology to place virtual furniture in a real room and check how it will look. If the user is satisfied with the suggestions, they can add the desired products to their cart and complete the purchase process.

[0814] 6. Purchase Procedures and Transaction Notifications

[0815] The user checks the proposed furniture and appliance arrangement, and if they like it, presses the "Purchase" button to add it to their cart. The device receives the user's purchase request and payment information, encrypts it, and sends it to the server. The server accesses the seller's API to check inventory and initiate the delivery process, and once the order is confirmed, notifies the user's device of the transaction status. This allows the user to check the transaction progress and delivery information in real time.

[0816] Specific examples

[0817] Example 1: To make the room appear larger

[0818] The user takes three photos of a 30-square-meter apartment for a single person and inputs the request to "make it look larger." The emotion engine then collects the user's facial and voice data to recognize emotions such as joy and dissatisfaction. The server analyzes the photo data and the request to identify the room's dimensions and the location of existing furniture. Based on the request, the generative AI model suggests a glass table or a small sofa, and the suggestions are adjusted based on the emotion recognition results. The server then obtains the adjusted product information from multiple sellers, generates a 3D simulation, and sends it to the device. The user reviews the 3D simulation and, if they like it, purchases the suggested product. The server then sends the order to the seller and notifies the device of the transaction status and delivery information.

[0819] Prompt Sentence Examples

[0820] "Please suggest interior design ideas to make a 30-square-meter single-person room look larger. Judging from the user's expressions, they seem surprised."

[0821] This system allows users to receive optimal interior design suggestions tailored to their own emotions and desires in real time at a physical store.

[0822] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0823] Step 1:

[0824] The user wears the smart glasses and takes three photos of the room. At this time, the user inputs their preferences, such as "I want the room to look spacious" or "I want a modern design," through the interface. In addition, the smart glasses' camera and microphone are used to collect the user's facial and voice data. The input data includes the room photo, the desired text, facial expression data, and voice data. The device then sends this data to the server.

[0825] Step 2:

[0826] The server uses image analysis algorithms (e.g., OpenCV, TensorFlow) to analyze the received photo data of the room. This identifies the room's size, the layout of existing furniture, the wall color, etc. The request text is also analyzed, and the server integrates this data and stores it in a database. The input data are the room photo and the user's request, and the output data are the analysis results, such as the recognized room size, furniture layout, and wall color.

[0827] Step 3:

[0828] The server uses an emotion recognition engine (e.g., Affectiva SDK, Microsoft Azure Cognitive Services) to analyze the user's facial and voice data. This identifies the user's emotions, such as joy, surprise, or dissatisfaction. The input data are facial and voice data, and the output data are the recognized emotions.

[0829] Step 4:

[0830] The server inputs prompts into the AI ​​model based on the image analysis results, emotion recognition results, and the user's requests. For example, a prompt such as "Please suggest interior design ideas to make a 30-square-meter room for a single person look larger. Judging from the user's facial expression, an expression of surprise appears." is generated and input into the AI ​​model. This generates a list of suitable furniture and home appliance candidates. The input data are the analysis results and the prompt, and the output data is a list of furniture and home appliance candidates.

[0831] Step 5:

[0832] Based on the generated candidate list of furniture and home appliances, the server accesses the APIs of multiple retailers to obtain product information (product name, price, size, and stock status). This allows the collection of the most up-to-date and detailed product information. The input data is the candidate list, and the output data is the obtained product information.

[0833] Step 6:

[0834] The server generates a 3D simulation model based on the acquired product information. This 3D model virtually arranges furniture in the user's room and is created using a 3D graphics library (e.g., Unity, Three.js). The input data is product information and furniture arrangement plans, and the output data is the 3D simulation model.

[0835] Step 7:

[0836] The terminal (smart glasses) displays the 3D simulation model and product information received from the server to the user. Through the smart glasses, the user can check the 3D simulation and see in real time how the virtual furniture will look when placed in a real room using AR technology. The input data is the 3D simulation model, and the output data is the user's visual confirmation.

[0837] Step 8:

[0838] If the user is satisfied with the proposed furniture or appliances, they press the "Purchase" button through the smart glasses interface to add them to their cart. The terminal then encrypts the purchase request and payment information and sends it to the server. The input data are the purchase request and payment information, and the output data is the encrypted transaction data.

[0839] Step 9:

[0840] The server accesses the seller's API to check inventory and initiate delivery procedures, and once the order is confirmed, it notifies the user's device of the transaction status. This allows the user to check the transaction progress and delivery information in real time. The input data is encrypted transaction data, and the output data is transaction status information.

[0841] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0842] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0843] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0844] [Third embodiment]

[0845] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0846] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0847] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0848] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0849] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0850] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0851] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0852] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0853] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0854] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0855] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0856] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0857] MODE FOR CARRYING OUT THE INVENTION

[0858] System Configuration

[0859] The "AI Interior Coordinator" system of this invention is primarily composed of a terminal operated by the user, a server that processes data, and a product information database that utilizes the APIs of multiple retailers. The system starts when the user takes a photo of the room and inputs their requirements, and then makes interior design suggestions based on those requirements.

[0860] Program processing

[0861] The program processing of this system will be explained in natural language below.

[0862] 1. User Input and Data Submission

[0863] The user first takes a photo of the room. It is recommended to take several photos from different angles to capture the entire room. The user also inputs their requests for the room layout in text format. For example, they might say, "I want the room to look spacious" or "I want it to have a modern feel." When the user enters this information into the app and presses the "Send" button, the data is sent to the server via their device.

[0864] 2. Data analysis and proposal generation

[0865] The server receives the photo data and requests sent from the device. The received photo data is first analyzed using an image analysis algorithm. This analysis identifies the size of the room, the layout of existing furniture, the color of the walls, and other factors. For example, the OpenCV library or the YOLO object detection algorithm can be used. Next, based on the user's requests, the data is input into a generative AI model (e.g., GPT-4) to generate optimal furniture and appliance candidates. For example, a glass table or a small sofa could be suggested to make the room appear larger.

[0866] 3. Obtaining product information and generating layouts

[0867] The server accesses the APIs of multiple retailers to obtain product information for the generated list of furniture and home appliance candidates. This product information includes product name, price, size, and stock information. Based on the obtained product information, the server generates the optimal layout for the user's room. This layout is generated as a 3D simulation model and provided to the user. For example, Unity or Three.js can be used to create the 3D model and virtually arrange the furniture in the room.

[0868] 4. View and confirm the proposal

[0869] The terminal displays the 3D simulation model and furniture list received from the server to the user. The user can view this 3D simulation via a smartphone or tablet. In addition, it is possible to virtually place furniture in a real room using AR technology and check how it will look. If the user is satisfied with the suggestions, they can add the desired items to their cart and complete the purchase process.

[0870] 5. Purchase Procedures and Transaction Notifications

[0871] If the user likes the suggested items, they press the "Purchase" button to add them to their cart. As part of the purchasing process, they enter their payment information and confirm the order. The terminal then sends the user's purchase request and payment information to the server. The server accesses the seller's API to check inventory and process delivery, and once the order is confirmed, it notifies the user's terminal of the transaction status. This notification allows the user to check the progress of the transaction and delivery information in real time.

[0872] Specific examples

[0873] Example 1: Make the room look larger

[0874] Suppose a user takes three photos of a 30-square-meter apartment for a single person and inputs the request to make it appear larger. The device sends this data to a server, which analyzes the photo data and identifies the dimensions of the room and the location of existing furniture. Based on the request, the generative AI model suggests a glass table and a small sofa, and obtains information on these products from multiple retailers.

[0875] The server generates a 3D simulation based on the acquired product information and sends it to the terminal. The user can check the 3D simulation and, if they like it, purchase the suggested product. The server then sends an order to the seller and notifies the terminal of the transaction status and delivery information.

[0876] The processing flow will be explained below.

[0877] Step 1:

[0878] The user takes three photos of the room. The photos are taken from different angles to capture information about the entire room. In addition, the user inputs preferences through the interface, such as "I want it to look spacious" or "I want a modern design."

[0879] Step 2:

[0880] The device receives the user's photo data and requests. This data is converted into the appropriate format and prepared for transmission. The device also encrypts the data to ensure security.

[0881] Step 3:

[0882] The device sends the encrypted photo data and request to the server using a secure communication protocol (e.g., HTTPS). It is recommended to use a checksum during data transmission to prevent errors.

[0883] Step 4:

[0884] The server stores the received photo data and requests in a database. To begin analyzing the photo data, it first sends the data to an image analysis algorithm, which uses libraries such as OpenCV and YOLO to identify the size of the room, the existing furniture arrangement, and the color of the walls.

[0885] Step 5:

[0886] The server inputs the analysis results and the user's requests into a generative AI model (e.g., GPT-4) to generate a list of suitable furniture and appliances. For example, furniture that makes a room appear larger might include a glass table, a small sofa, and a mirror with reflective materials.

[0887] Step 6:

[0888] Based on the generated candidate list, the server accesses the APIs of multiple retailers to obtain product information, including product name, size, price, stock status, etc. This information is then stored in a database.

[0889] Step 7:

[0890] The server generates a 3D simulation model based on the acquired product information. This simulation model virtually arranges furniture in the room layout and is created using a 3D graphics library (e.g., Unity or Three.js).

[0891] Step 8:

[0892] The server then sends the generated 3D simulation model and product information to the terminal, where the data is packetized in an appropriate format and transmitted again using a secure communication protocol.

[0893] Step 9:

[0894] The device analyzes the received data and displays a 3D simulation to the user, who can view the simulation through the interface of their smartphone or tablet. Additionally, by selecting the AR mode, it is possible to place virtual furniture in a real room.

[0895] Step 10:

[0896] The user checks the proposed furniture and appliance arrangement, and if they like it, they press the "Purchase" button, which adds the desired items to their cart and proceeds to the purchase process.

[0897] Step 11:

[0898] The terminal receives the purchase request and payment information from the user, encrypts it, and sends it to the server. The payment information is handled using a security protocol (e.g., TLS).

[0899] Step 12:

[0900] The server accesses the seller's API to check inventory and initiate the shipping process. Once the order is confirmed, the server calculates the commission and secures the revenue.

[0901] Step 13:

[0902] After the order is confirmed and the delivery procedure is completed, the server notifies the terminal of the transaction status, allowing the user to check the transaction progress and delivery information in real time.

[0903] Step 14:

[0904] The terminal displays transaction status and delivery information to the user and notifies them that the purchase has been completed.

[0905] Example 1

[0906] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0907] Conventional interior coordination systems have issues such as difficulty for users in visualizing the room layout, lack of information, and the time and effort required to make a purchasing decision. Furthermore, when acquiring product information from multiple retailers, data acquisition and management can be cumbersome, making it necessary to develop an efficient system. In addition, users lack a visual experience to confirm the proposals, making it difficult to visualize the actual layout.

[0908] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0909] In this invention, the server includes: a means for a user to acquire an image of a room and input their requirements; a means for transmitting the image data and the requirements to an information processing device; a means for analyzing the image data and identifying the dimensions and existing layout of the room; a means for generating optimal furniture and electronic device candidates based on the identification results and the requirements using a generative AI model; a means for acquiring product information from multiple vendors; a means for generating a layout based on the acquired product information and displaying a 3D simulation to the user; and a means for the user to check the 3D simulation and then proceed with purchasing the products. This allows the user to visually confirm the room layout, making it easier to make a purchasing decision through efficient information acquisition and a visual experience.

[0910] "User" refers to an individual or organization that uses the system.

[0911] "Image" refers to a digital photograph or graphic visual record of the exterior or interior of a room.

[0912] "Information processing device" refers to a computer system that receives, analyzes, generates, and transmits data.

[0913] "Analysis" refers to the process of determining room dimensions and furniture layout based on image data.

[0914] "Dimensions" refers to physical measurements related to the size of a room or furniture.

[0915] "Layout" refers to the relative positions of furniture and decorations within a room.

[0916] "Specific results" refers to data regarding the dimensions and layout of the room obtained through analysis.

[0917] "Candidates" refers to a list of furniture and electronic devices that are optimally selected based on the user's requirements.

[0918] A "generative AI model" refers to an algorithm that uses machine learning to suggest furniture and electronic devices that meet user needs.

[0919] "Product information" refers to data such as the name, price, size, and inventory of furniture and electronic devices.

[0920] "Layout" refers to a virtual layout of furniture and decorations in a room.

[0921] "3D simulation" refers to a model that visually reproduces the furniture arrangement in a room using 3D computer graphics.

[0922] "Augmented reality technology" refers to technology that overlays digital information onto real space.

[0923] "Retailer" refers to a business or store that sells furniture or electronics.

[0924] "Checkout" refers to the process by which a user selects the desired product, enters payment information, and confirms the order.

[0925] "Order" refers to a transaction request for an item that a user wishes to purchase.

[0926] MODE FOR CARRYING OUT THE INVENTION

[0927] The "AI Interior Coordinator" system of this invention begins when a user acquires a photo of a room and inputs their requirements. The photo data taken by the user and the input requirements are sent to an information processing device (server), which then performs a series of data analyses and proposes interior designs. Specific embodiments are described below.

[0928] User Actions

[0929] First, the user launches a dedicated application and takes photos of the room from multiple angles. For example, it is recommended to take four photos: one from the front, one from the left, one from the right, and one from the back. The user also enters their room-related requirements in text format (e.g., "I want the room to look spacious" or "I want a modern feel"). Once these are entered, the user presses the "Send" button in the application. This sends the photo data and requirements from the device to the information processing device (server).

[0930] Server Operation

[0931] The server receives the submitted photo data and request. First, it analyzes the photo data using the OpenCV library and the YOLO object detection algorithm to determine the room dimensions and the layout of existing furniture. Specifically, it uses the YOLO algorithm to identify furniture and obtain the position and size of each piece of furniture. Next, it uses a generative AI model (e.g., GPT-4) based on the user's request to generate optimal furniture and electronic device candidates. At this time, it inputs data into the generative AI model using prompt statements.

[0932] Prompt Sentence Examples

[0933] "Please suggest the best interior coordination to make a 30 square meter room look larger. Using glass furniture and a small sofa."

[0934] "Can you give me some ideas for modernizing this room? The current furniture arrangement is included in the images."

[0935] Obtaining product information

[0936] Based on the generated candidate list of furniture and electronic devices, the server accesses the APIs of multiple vendors to obtain corresponding product information. The obtained product information includes product name, price, size, stock information, etc. For example, the product information is obtained using the API of an internet shopping mall.

[0937] Layout generation and display

[0938] The server uses Unity or Three.js to generate a 3D simulation model based on the acquired product information and create an optimal layout. These 3D simulations include furniture virtually placed in the room. This model is sent to the device so that the user can visually check it.

[0939] Proposal confirmation and purchase process

[0940] The device displays the 3D simulation model and product list received from the server to the user. The user can view this 3D simulation through the app. Augmented reality technology using the smartphone camera also allows the user to place virtual furniture in a real room and simulate how it will look. If the user is satisfied with the suggestions, they can add the desired products to their cart and proceed with the purchase process.

[0941] Purchase Process and Transaction Notifications

[0942] The user presses the "Purchase" button, adds the product to the cart, enters payment information, and confirms the order. The terminal sends the user's purchase request and payment information to the server. The server accesses the vendor's API to check inventory and process delivery, and once the order is confirmed, it notifies the user's terminal of the transaction status and delivery information. This allows the user to check the progress of the transaction and product delivery information in real time.

[0943] In this way, the "AI interior coordinator" of the present invention provides users with an efficient and intuitive interior coordination proposal and purchasing process.

[0944] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0945] Step 1: User Input and Data Submission

[0946] The user uses a dedicated application to take photos of the room from multiple angles. As input data, they take photos of the front, left, right, and back of the room, and enter their requests in text format, such as "I want the room to look spacious" or "I want a modern feel." When they press the "Send" button, this data is sent from the device to the server. As output, the photo data and the requests in text format are sent to the server.

[0947] Step 2: Receiving and initial processing of data

[0948] The server receives the photo data and requests sent from the terminal. As input data, it receives the photo data and requests sent from the user. Based on this, it performs initial processing of the data and prepares it for data analysis. As output, it generates the photo data and requests converted into the format required for analysis.

[0949] Step 3: Data analysis

[0950] The server analyzes the photo data using the OpenCV library and the YOLO object detection algorithm. It uses the photo data formatted for processing as input data. Based on this, it identifies furniture and room dimensions in the image and extracts attribute information (position, size, color, etc.) of the identified objects. The output is the room dimensions and the existing furniture layout.

[0951] Step 4: Proposal generation using generative AI models

[0952] The server inputs the identification results and the user's requests into a generative AI model (e.g., GPT-4) and generates optimal furniture and electronic device candidates. The input data includes the room dimensions, existing furniture layout, and the user's requests (in text format). Based on the generated prompt, the AI ​​model analyzes and outputs specific furniture and interior design suggestions. The output is a list of suggested furniture and electronic devices.

[0953] Step 5: Obtaining product information using the API

[0954] The server accesses the APIs of multiple vendors to obtain product information based on the generated list of furniture and electronic device candidates. The server uses the list of proposed furniture and electronic devices as input data. Based on this, it makes API requests to obtain product names, prices, sizes, stock information, etc. The server generates a list of the obtained product information as output.

[0955] Step 6: Generate a 3D simulation model

[0956] The server uses Unity or Three.js to generate a 3D simulation model based on the acquired product information. The input data is a list of product information and the dimensions of the existing room. Based on this, it creates an optimal layout and generates a 3D model of the furniture virtually placed in the room. The generated 3D simulation model is obtained as the output.

[0957] Step 7: View and Review the Simulation

[0958] The device displays the 3D simulation model and product list received from the server to the user. The 3D simulation model sent from the server is used as input data. The user can visually check this 3D simulation through the app. It is also possible to use augmented reality technology to place virtual furniture in a real room and simulate how it will look. The output is a visual 3D simulation that the user can check.

[0959] Step 8: Checkout

[0960] Based on the suggestions they like, the user presses the "Purchase" button, adds the product to their cart, enters payment information, and confirms the order. The input data is product information and payment information. Based on this, the purchase procedure is carried out. As an output, information indicating that the user has completed the purchase procedure is sent to the server.

[0961] Step 9: Transaction Notification

[0962] The server accesses the seller's API based on the user's purchase request and payment information to check inventory and process delivery. Detailed purchase process information is used as input data. Once the order is confirmed, transaction status and delivery information is sent to the user's device. As an output, the user receives notifications that allow them to check the transaction progress and delivery information in real time.

[0963] (Application example 1)

[0964] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0965] In today's brick-and-mortar stores, customers often have difficulty visualizing the product in its actual placement when selecting the right interior product for their home. Traditionally, purchasing decisions are often made solely based on catalogs and displays, increasing the risk of making the wrong choice. Similarly, online shopping makes it difficult to confirm the product's suitability in the actual room environment. This leads to an increase in returns and exchanges after purchase, lowering customer satisfaction and increasing operational costs.

[0966] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0967] In this invention, the server includes: a means for a user to take an image of a room and input their preferences; a means for transmitting the image data and preferences to the server; a means for analyzing the image data to recognize the size of the room and the existing item layout; a means for generating optimal device and tool candidates based on the recognition results and the user's preferences; a means for acquiring product information from multiple sellers; a means for generating a layout based on the acquired product information and displaying a 3D simulation to the user; a means for the user to check the 3D simulation and then complete a product purchase procedure; and a means for displaying virtual items arranged in a real space and using augmented reality technology to confirm the display. This allows users to visually confirm the actual layout of virtual interior items in a physical store or at home. Furthermore, customers can accurately confirm the suitability of products before purchase, preventing incorrect selections and reducing returns and exchanges.

[0968] A "user" refers to an individual who uses the system to take pictures and input their preferences to coordinate the interior of a room.

[0969] "Image data" refers to photographic data of a room taken by a user, and includes information such as the size of the room and the layout of existing items.

[0970] "Wishes" are requests that the user inputs regarding the interior of the room, and refer to specific requests such as "I want the room to look spacious."

[0971] "Server" refers to a computer system that receives image data and requests sent by users and performs analysis and data processing.

[0972] "Analysis" refers to the process of identifying the size of a room and the layout of existing items based on image data, and is done using computer vision techniques and object recognition algorithms.

[0973] "Device and tool candidates" refers to a list of optimal interior products suggested by the system based on the user's preferences and analysis results.

[0974] "Sellers" refer to companies and stores that sell interior products, and the system obtains product information from them.

[0975] "Product Information" refers to detailed data about interior products provided by sellers, such as product name, price, size, and inventory information.

[0976] "Layout" refers to the layout of interior products virtually arranged in a room based on the acquired product information.

[0977] "Three-dimensional simulation" refers to three-dimensional simulation data for visually displaying the generated layout to the user.

[0978] "Augmented reality technology" refers to a technology that displays virtual objects superimposed on a real-world environment, allowing users to visually check virtual interior products in real space.

[0979] "Purchase procedure" refers to the series of operations that a user performs to actually purchase interior products suggested through the system.

[0980] This invention provides an "interior AR shopping assistant" system that allows users to try out interior products in real time in a "real store." To realize this system, the following hardware and software are required:

[0981] Hardware and software used

[0982] Hardware: Requires a smartphone (camera, display, internet connection).

[0983] Software: OpenCV, YOLO (object detection), Unity (3D simulation), generative AI model (GPT-4API), etc. are required.

[0984] System configuration and operation

[0985] 1. User Action:

[0986] Users use their smartphones to take pictures of their room from multiple angles and input their desired interior style and requirements in text format, such as "I want the room to look spacious" or "I want a modern feel."

[0987] 2. Data transmission:

[0988] The user's terminal transmits the captured image data and the desired text data to the server.

[0989] 3. Analysis of imaging data:

[0990] The server analyzes the received image data using OpenCV and YOLO to determine the size of the room and the layout of existing items. This analysis determines the floor area, the location of walls, and how existing furniture is arranged.

[0991] 4. Generate optimal furniture candidates:

[0992] The server uses a generative AI model (GPT-4) to generate optimal interior product candidates based on the analyzed data and the user's preferences. For example, if the user desires a "modern taste" for a living room measuring approximately 20 square meters, the generative AI model will suggest a simple white sofa and a glass coffee table.

[0993] Example prompt sentence:

[0994] Room features: 20 square meters, white walls, existing furniture: dark wooden table and bookshelf.

[0995] User's request: I want a modern and simple interior.

[0996] 5. Obtaining product information:

[0997] The server obtains information about the interior products listed as candidates through APIs of multiple sellers, including product names, prices, sizes, and stock information.

[0998] 6. Generation of 3D simulations:

[0999] The server uses Unity to create a 3D simulation based on the acquired product information, and the simulation is generated in a virtual room with the proposed furniture arranged.

[1000] 7. View the simulation:

[1001] The user's device displays the 3D simulation sent from the server and uses augmented reality (AR) technology to overlay virtual interior products onto the real space, allowing the user to visually confirm how the virtual interior products will actually look when placed in a physical store or at home.

[1002] Specific examples

[1003] For example, if a user wants to make a 30-square-meter living room look modern and spacious, the system analyzes images taken from various angles of the room and generates suggestions such as a white sofa and a glass table. Product information is then acquired and a 3D simulation is generated based on the acquired information. The user can then virtually place these suggested furniture items in their real room using their smartphone, see how they will look, and then purchase the items they like.

[1004] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1005] Step 1:

[1006] The user takes a picture of the room and inputs their preferences.

[1007] The user uses a smartphone to take multiple images of the room from different angles. They also input their preferences for the room's interior in text format. For example, they might input a preference such as "I want the room to look spacious." The input data is saved as image data and text data.

[1008] Step 2:

[1009] The device sends the photo data and your request to the server.

[1010] The device sends the image data and text data the user has entered to the server, which then securely transfers the data over the Internet and stores it on the server.

[1011] Step 3:

[1012] The server analyzes the photo data and recognizes the size of the room and the layout of existing items.

[1013] The server analyzes the received image data using OpenCV and YOLO object detection algorithms. This analysis identifies the size of the room, the location of walls, and the layout of existing furniture. For example, the floor area and furniture locations are automatically calculated.

[1014] Step 4:

[1015] The server generates optimal device and tool candidates based on the recognition results and user preferences.

[1016] The server uses a generative AI model (GPT-4) based on the analysis results and the user's preferences to generate optimal device and tool candidates. The analysis results and the user's preferences are input into the prompt text.

[1017] Example prompt sentence:

[1018] Room features: 20 square meters, white walls, existing furniture: dark wooden table and bookshelf.

[1019] User's wish: A modern and simple interior.

[1020] Based on this prompt, the generative AI model suggests specific interior product options, such as a simple white sofa or a glass table.

[1021] Step 5:

[1022] The server retrieves product information from multiple sellers

[1023] Based on the generated list of furniture and tool candidates, the server accesses the APIs of multiple sellers to obtain product information. The obtained product information includes product name, price, size, stock information, etc. This information is imported into the server as structured data in JSON format, etc.

[1024] Step 6:

[1025] The server generates a layout based on the product information acquired and creates a 3D simulation.

[1026] The server uses Unity to create a 3D simulation based on the acquired product information. The placement data is constructed as a 3D model, and a simulation of the virtual furniture placement in the room is generated. This simulation data is visualized for the user.

[1027] Step 7:

[1028] The server sends the 3D simulation to the terminal, which displays it.

[1029] The server sends the generated 3D simulation to the user's device, which then displays the received simulation data, allowing the user to use augmented reality (AR) technology to overlay virtual furniture onto the real space, creating a visual experience that makes it seem as if the new furniture is actually placed in the room.

[1030] Step 8:

[1031] The user completes the purchase process

[1032] The user checks the 3D simulation and selects the product they wish to purchase. As part of the purchase process, they enter payment information and confirm the order. The terminal sends the user's purchase request and payment information to the server, which then accesses the seller's API to check inventory and process delivery. Once the order is confirmed, the user's terminal is notified of the transaction status and delivery information.

[1033] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1034] MODE FOR CARRYING OUT THE INVENTION

[1035] System Configuration

[1036] The "AI Interior Coordinator" system of this invention not only proposes optimal furniture and home appliances based on user input data, but also recognizes the user's emotions and adjusts the proposals accordingly. This system is primarily composed of a terminal operated by the user, a server that performs data analysis and emotion recognition, and a product information database that utilizes the APIs of multiple retailers.

[1037] Program processing

[1038] The program processing of this system will be explained in natural language below.

[1039] 1. User Input and Data Submission

[1040] The user takes three photos of the room and inputs their requirements. It is recommended that the photos be taken from different angles to capture the entire room. The user inputs their requirements, such as "I want the room to look spacious" or "I want a modern design." In addition, the user collects their facial expressions and voice data through the interface. This data is sent to the emotion engine.

[1041] 2. Data analysis and proposal generation

[1042] The server receives the photo data, requests, facial expressions, and voice data sent from the device. The photo data is analyzed using an image analysis algorithm to determine the size of the room, the layout of existing furniture, the color of the walls, etc. The analysis results and the user's requests are input into a generative AI model to generate a list of optimal furniture and home appliances.

[1043] 3. Emotion recognition and suggestion adjustment

[1044] The server uses an emotion engine to analyze the user's facial and voice data and recognize emotions such as joy, surprise, and dissatisfaction. For example, a machine learning library is used for facial expression analysis, and a voice recognition algorithm is used for voice analysis. Based on the emotion recognition results, the generated candidate list of furniture and home appliances is re-adjusted according to the user's emotions.

[1045] 4. Obtaining product information and generating layouts

[1046] The server accesses the APIs of multiple retailers based on the adjusted candidate list to obtain product information. The obtained product information includes product name, size, price, stock status, etc., and generates a 3D simulation model based on this information. This 3D simulation virtually arranges furniture in a room and is created using a 3D graphics library (e.g., Unity or Three.js).

[1047] 5. View and confirm the proposal

[1048] The terminal displays the 3D simulation model and product information received from the server to the user. The user can check the 3D simulation through the interface of their smartphone or tablet. In addition, it is possible to use AR technology to place virtual furniture in a real room and check how it will look. If the user is satisfied with the suggestions, they can add the desired products to their cart and complete the purchase process.

[1049] 6. Purchase Procedures and Transaction Notifications

[1050] The user checks the proposed furniture and appliance arrangement, and if they like it, presses the "Purchase" button to add it to their cart. The device receives the user's purchase request and payment information, encrypts it, and sends it to the server. The server accesses the seller's API to check inventory and initiate delivery procedures, and once the order is confirmed, notifies the user's device of the transaction status. This allows the user to check the transaction progress and delivery information in real time.

[1051] Specific examples

[1052] Example 1: To make the room appear larger

[1053] A user takes three photos of a 30-square-meter apartment for a single person and inputs the request to "make it look larger." The emotion engine then collects the user's facial and voice data to recognize emotions such as joy and dissatisfaction. The server analyzes the photo data and the request to identify the room's dimensions and the location of existing furniture. Based on the request, the generative AI model suggests a glass table and a small sofa, and the suggestions are adjusted based on the emotion recognition results.

[1054] The server obtains the coordinated product information from multiple sellers, generates a 3D simulation, and sends it to the device. The user reviews the 3D simulation and, if they like it, purchases the suggested product. The server then sends the order to the seller and notifies the device of the transaction status and delivery information.

[1055] These specific processes enable optimal interior design proposals that reflect the user's emotions.

[1056] The processing flow will be explained below.

[1057] Step 1:

[1058] The user takes three photos of the room. The photos are taken from different angles to gather information about the entire room. The user also inputs requests, such as "I want the room to look larger." In addition, an interface is used to send the user's facial expressions and voice to the emotion engine.

[1059] Step 2:

[1060] The device collects the user's photos, input requests, and facial and voice data, which are then converted into a format for recognition and all data is prepared for transmission.

[1061] Step 3:

[1062] The device sends the photo data, desires, and emotion data in encrypted form to the server using a secure communication protocol (e.g., HTTPS).

[1063] Step 4:

[1064] The server stores the received photo data, requests, and emotion data in a database. The photo data is input into an image analysis algorithm (e.g., OpenCV, YOLO) to extract information such as the size of the room, the layout of existing furniture, and the color of the walls.

[1065] Step 5:

[1066] The server inputs the image analysis results and the customer's requests into a generative AI model to generate an initial list of furniture and appliance candidates. For example, a glass table or a small sofa might be selected to make the room appear larger.

[1067] Step 6:

[1068] The server inputs the user's facial expression and voice data into the emotion engine to recognize the user's emotions. Emotions such as joy, dissatisfaction, and surprise are recognized, and emotion recognition results are generated.

[1069] Step 7:

[1070] The server then adjusts the list of furniture and appliance candidates generated by the generative AI model based on the emotion recognition results. For example, if the user expresses dissatisfaction, furniture with a different design or functionality will be suggested.

[1071] Step 8:

[1072] Based on the adjusted candidate list, the server accesses the APIs of multiple sellers to retrieve product information, including product name, price, size, and availability, and stores it in a database.

[1073] Step 9:

[1074] The server generates a 3D simulation model based on the acquired product information. This simulation model virtually arranges furniture in a room and is created using a 3D graphics library (e.g., Unity, Three.js).

[1075] Step 10:

[1076] The server then transmits the generated 3D simulation model and product information to the terminal using a communication protocol. The data is then packetized and transmitted again using a secure communication protocol.

[1077] Step 11:

[1078] The device analyzes the received 3D simulation model and product information and displays it to the user. The user can view the 3D simulation via a smartphone or tablet. It is also possible to use the AR function to place virtual furniture in a real room.

[1079] Step 12:

[1080] The user checks the proposed furniture and appliance arrangement, and if satisfied, presses the "Purchase" button, which adds the desired items to the cart and proceeds with the purchase.

[1081] Step 13:

[1082] The terminal receives the purchase request and payment information from the user, encrypts it, and transmits it to the server using a security protocol.

[1083] Step 14:

[1084] The server accesses the seller's API to check inventory and initiate the shipping process. Once the order is confirmed, the server calculates the commission and secures the revenue.

[1085] Step 15:

[1086] The server completes the order confirmation and delivery procedures and notifies the terminal of the transaction status, allowing the user to check the transaction progress and delivery information in real time.

[1087] Step 16:

[1088] The terminal displays transaction status and delivery information to the user and notifies them that the purchase has been completed.

[1089] Example 2

[1090] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1091] Interior design has become an important element in modern lifestyles. However, it is not easy for users to select and appropriately arrange furniture and appliances themselves. In particular, it is difficult to obtain proposals that reflect one's own feelings and desires. This results in users spending a great deal of time and effort to create a satisfying space. A system that can solve this problem and provide optimal interior proposals more quickly and efficiently is needed.

[1092] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1093] In this invention, the server includes: means for a user to acquire an image of a room and input their requirements; means for transmitting the image data and the requirements to the server; means for analyzing the image data and recognizing the size of the space and the existing furniture layout; means for generating optimal furniture and home appliance candidates based on the recognition results and the requirements using a generative AI model; means for analyzing the user's facial expressions and voice data and recognizing emotions; means for adjusting the generated furniture and home appliance candidates based on the recognized emotions; means for acquiring product information from multiple retailers; means for generating a layout based on the acquired product information and displaying a 3D simulation to the user; and means for the user to check the 3D simulation and then proceed with purchasing the products. This allows the user to quickly and effectively obtain interior design suggestions that reflect their emotions and requirements.

[1094] "User" refers to the entity that uses the system to receive interior design proposals.

[1095] "Image" refers to visual data captured of the room situation.

[1096] "Requests" refer to the requirements for interior design desired by the user.

[1097] A "terminal" is an electronic device that allows a user to take pictures and input data.

[1098] "Server" means a central processing unit for analyzing received data, generating and managing offers.

[1099] "Image data" refers to digital information obtained from images of a room taken by a user.

[1100] "Analysis" refers to the process of extracting and understanding specific information from image data, audio data, etc.

[1101] "Space" refers to the physical size or area of ​​a room.

[1102] "Existing furniture arrangement" refers to the position and layout of furniture already placed in the room.

[1103] A "generative AI model" is an algorithm that uses artificial intelligence to generate suggestions based on user requests.

[1104] "Facial expression data" refers to digital information that captures a user's facial movements and expressions.

[1105] "Voice data" refers to digital information that records a user's speech or voice.

[1106] "Emotion" refers to information that indicates the user's psychological state.

[1107] "Adjustment" is the process of modifying the proposal to better suit the user's feelings and desires.

[1108] "Retailer" refers to a company or store that offers furniture or home appliances.

[1109] "Product information" refers to detailed product information provided by a retailer.

[1110] "Layout" refers to the arrangement plan of furniture and appliances.

[1111] "3D simulation" is a technology that visualizes the placement of furniture and home appliances in a virtual space.

[1112] "Augmented reality technology" is a technology that displays virtual elements superimposed on real scenery.

[1113] The "purchase procedure" refers to the procedure for actually purchasing the proposed product.

[1114] MODE FOR CARRYING OUT THE INVENTION

[1115] The "AI Interior Coordinator" system of this invention not only proposes optimal furniture and home appliances based on user input data, but also recognizes the user's emotions and adjusts the proposals accordingly. This system is primarily composed of a terminal operated by the user, a server that performs data analysis and emotion recognition, and a product information database that utilizes the APIs of multiple retailers.

[1116] User Input and Data Submission

[1117] The user takes three photos of the room using the device's camera and inputs their preferences, such as "I want the room to look spacious" or "I want a modern design." The photos should be taken from different angles to capture the entire room. The user also collects their facial expressions and voice data through the interface. This data is encrypted and sent to the server.

[1118] Data analysis and proposal generation

[1119] The server receives the photo data, requests, facial expressions, and voice data sent from the device. The photo data is analyzed using image analysis algorithms (e.g., OpenCV, TensorFlow). As a result of the analysis, the size of the room, the layout of existing furniture, the color of the walls, etc. are identified.

[1120] Next, the server inputs the analysis results and the user's request into a generative AI model (e.g., GPT-4, DALL-E). The prompt is in the form of "Please suggest furniture arrangements that will make a 30-square-meter room appear larger," and the AI ​​model generates a list of optimal furniture and appliance candidates.

[1121] Emotion recognition and suggestion adjustment

[1122] The server analyzes the user's facial expressions and voice data using an emotion engine (e.g., Microsoft Azure Emotion API, Google Cloud Vision API). For example, it uses face-api.js to analyze facial movements and Google Speech-to-Text API to analyze voice data. The emotion engine identifies emotions such as joy, surprise, and dissatisfaction and adjusts the generated list of furniture and appliance candidates.

[1123] Obtaining product information and generating layouts

[1124] Based on the adjusted candidate list, the server accesses APIs of multiple sellers (e.g., Amazon Product Advertising API, Rakuten API) to obtain product information. The obtained product information includes product name, size, price, and stock status. A 3D simulation model is generated based on this information. This simulation is created using a 3D graphics library (e.g., Unity, Three.js).

[1125] View and review suggestions

[1126] The device displays the 3D simulation model and product information received from the server to the user. Users can check the 3D simulation through the interface of their smartphone or tablet. Furthermore, it is possible to place virtual furniture in a real room using AR technology (e.g., ARKit, ARCore) and check how it will look.

[1127] Purchase Process and Transaction Notifications

[1128] The user checks the proposed furniture and appliance arrangement, and if they like it, they press the "Purchase" button to add it to their cart. The device receives the user's purchase request and payment information, encrypts it, and sends it to the server. The server checks inventory and initiates delivery procedures via the seller's API, and once the order is confirmed, notifies the user's device of the transaction status. This allows the user to check the transaction progress and delivery information in real time.

[1129] Examples of concrete examples and prompts

[1130] Specific examples

[1131] The user takes three photos of a 30-square-meter apartment for a single person and inputs the request to "make it look larger." The emotion engine then collects the user's facial expressions and voice data to recognize emotions. The server analyzes the photo data and the request to identify the room's dimensions and the location of existing furniture. Based on the request, the generative AI model suggests a glass table or a small sofa, and the suggestions are adjusted based on the emotion recognition results. The adjusted product information is obtained from multiple sellers, and a 3D simulation is generated and sent to the device. The user reviews the 3D simulation and, if they like it, purchases the suggested product. The server then sends the order to the seller and notifies the device of the transaction status and delivery information.

[1132] Prompt Sentence Examples

[1133] "Please suggest the best furniture layout for a 30 square meter room for a single person."

[1134] "Please tell me some interior design ideas to make a room look larger."

[1135] "Please suggest some furniture with a modern design."

[1136] This system allows users to quickly and effectively obtain interior design suggestions that reflect their own feelings and desires.

[1137] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1138] Step 1:

[1139] The user takes three photos of the room using the device's camera and enters their requirements.

[1140] Input: 3 photos of the room taken by the user, request text

[1141] How it works: The user inputs their preferences, such as "I want the room to look larger" or "I want a modern design," into the device's interface. Once the photo is taken, this data is collected and encrypted by the device.

[1142] Output: Encrypted photo data, request text

[1143] Step 2:

[1144] The terminal transmits the collected photo data and the request text to the server.

[1145] Input: Encrypted photo data, request text

[1146] How it works: The device sends this data to a server over the Internet, where it is encrypted and sent using a secure protocol.

[1147] Output: Photo data received by the server, request text

[1148] Step 3:

[1149] The server then uses image analysis algorithms to analyze the received photo data and determine the size of the room and the existing furniture layout.

[1150] Input: Photo data

[1151] How it works: The server uses image analysis software such as OpenCV and TensorFlow to process the data and determine the size of the room, the color of the walls, and the layout of the existing furniture.

[1152] Output: Data about the analyzed room size, existing furniture placement, and wall color

[1153] Step 4:

[1154] The server inputs the analyzed data and the desired text into a generative AI model to generate a list of suitable furniture and home appliances.

[1155] Input: Parsed data, requested text

[1156] Specific operation: The server uses a generative AI model (e.g., GPT-4, DALL-E) to create a prompt and input it into the model. Specifically, a prompt such as "Please suggest a furniture arrangement that will make a 30-square-meter room appear larger" is used.

[1157] Output: A list of the best furniture and appliances

[1158] Step 5:

[1159] The server uses an emotion engine to analyze the user's facial expressions and voice data and recognize the user's emotions.

[1160] Input: User's facial expression data, voice data

[1161] Specific operation: The server uses an emotion engine (e.g., Microsoft Azure Emotion API, Google Cloud Vision API) to analyze the user's facial expressions and voice to identify emotions such as joy, surprise, dissatisfaction, etc. For example, facial expression analysis is performed using face-api.js, and voice analysis is performed using Google Speech-to-Text API.

[1162] Output: Recognized emotion data

[1163] Step 6:

[1164] The server adjusts the list of furniture and home appliance candidates generated based on the emotion data.

[1165] Input: List of suitable furniture and appliances, recognized emotion data

[1166] Specific behavior: When a user shows a facial expression of joy, the candidate list is adjusted to prioritize suggestions that increase that joy.

[1167] Output: A tailored list of furniture and appliances

[1168] Step 7:

[1169] The server accesses the APIs of multiple sellers to obtain product information based on the adjusted candidate list.

[1170] Input: A tailored list of furniture and appliances

[1171] Specific operation: The server uses the API to obtain the necessary product information from multiple sellers, such as product name, size, price, and availability.

[1172] Output: Retrieved product information

[1173] Step 8:

[1174] The server generates a 3D simulation model based on the acquired product information.

[1175] Input: Retrieved product information

[1176] What it does: The server uses a 3D graphics library (e.g. Unity, Three.js) to generate a simulation of placing furniture in a virtual model of the room.

[1177] Output: 3D simulation model

[1178] Step 9:

[1179] The terminal displays the 3D simulation model and product information received from the server to the user.

[1180] Input: 3D simulation model, product information

[1181] How it works: The device displays this 3D simulation to the user through the interface of a smartphone or tablet. It is also possible to place virtual furniture in a real room using AR technology (e.g., ARKit, ARCore) and see how it will look.

[1182] Output: 3D simulation and product information displayed to the user

[1183] Step 10:

[1184] The user checks the proposed furniture and appliance arrangement, and if they like it, they press the "Purchase" button to add it to their cart. The terminal encrypts the purchase request and payment information and sends it to the server.

[1185] Input: User purchase request, payment information

[1186] Specific operation: When the user presses the purchase button, the terminal encrypts the entered payment information and sends it to the server.

[1187] Output: Purchase request sent to server, payment information

[1188] Step 11:

[1189] The server will check inventory and initiate delivery procedures via the seller's API, and once the order is confirmed, will notify the user's device of the transaction status.

[1190] Input: Purchase request, payment information

[1191] Specific operation: The server checks inventory and processes delivery, and once the order is confirmed, it notifies the user's terminal of the transaction status in real time.

[1192] Output: Transaction status reported to the user's terminal

[1193] This allows users to quickly and effectively receive interior design suggestions that reflect their feelings and desires, ensuring a consistent and smooth process leading up to the purchase.

[1194] (Application example 2)

[1195] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1196] Conventional interior design suggestion systems have had the problem of making it difficult for users to find the perfect interior design that matches their own emotions and preferences. In particular, when receiving suggestions in a physical store, it is not possible to adjust the suggestions in real time, making it difficult to provide suggestions that will satisfy the user.

[1197] The specification processing by the specification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes: means for a user to take a photo of a room and input their requirements; means for transmitting the photo data and the requirements to the server; means for analyzing the photo data and recognizing the size of the room and the layout of existing furniture; means for generating optimal furniture and home appliance candidates based on the recognition results and the requirements; means for analyzing the user's facial expressions and voice data using an emotion recognition engine and recognizing emotions; means for adjusting the proposal content based on the recognized emotions; means for acquiring product information from multiple retailers; means for generating a layout based on the acquired product information and displaying a 3D simulation to the user; and means for the user to check the 3D simulation and then proceed with the purchase of the product. This enables optimal interior design proposals to be made in real time according to the user's emotions and requirements.

[1198] "Means for users to take photos of a room and input their requirements" refers to equipment and an interface that allows users to take multiple photos of a room from different angles and input their desired interior style, furniture requirements, etc.

[1199] The "means for transmitting the photo data and request to the server" refers to a function and mechanism for transmitting the photograph taken and the request entered from the user terminal to the server via a communication line.

[1200] "Means for analyzing the photographic data and recognizing the size of the room and the existing furniture layout" refers to a device and program that uses an image analysis algorithm to identify the dimensions of a room and the current furniture layout from the photographic data taken.

[1201] The "means for generating optimal furniture and home appliance candidates based on the recognition results and user requests" refers to a generation algorithm and engine for selecting and listing appropriate furniture and home appliances based on the analyzed data and the user's requests.

[1202] "Means for recognizing emotions by analyzing a user's facial expressions and voice data using an emotion recognition engine" refers to a technology that uses emotion recognition software and machine learning models to analyze a user's facial expressions and voice to identify emotions such as joy, dissatisfaction, and surprise.

[1203] The "means for adjusting the suggestions based on the recognized emotion" refers to algorithms and programs for modifying and optimizing the furniture and appliance suggestions list depending on the user's emotional state.

[1204] "Means for obtaining product information from multiple retailers" refers to systems and programs that use APIs from various retailers to collect data such as product name, price, inventory, and size.

[1205] The "means for generating a layout based on the acquired product information and displaying a 3D simulation to the user" refers to a 3D graphics engine and user interface for creating a 3D layout using the acquired product information and visually displaying it to the user.

[1206] The "means for a user who has reviewed the 3D simulation to complete the purchase of a product" refers to an interface and process that allows a user to review the 3D simulation, select a product they like, and make a payment to purchase the product.

[1207] MODE FOR CARRYING OUT THE INVENTION

[1208] System Configuration

[1209] The "interior design proposal support system" of this invention has the function of proposing optimal furniture and home appliances based on user input data, as well as the function of recognizing the user's emotions and adjusting the proposal content based on those emotions. This system is mainly composed of a terminal operated by the user, a server that performs data analysis and emotion recognition, and a product information database that utilizes the APIs of multiple retailers.

[1210] Program action

[1211] 1. User Input and Data Submission

[1212] The user takes three photos of the room and inputs their preferences. The photos should be taken from different angles to capture the entire room. The user may input preferences such as "I want the room to look spacious" or "I want a modern design." In addition, the smart glasses collect facial and voice data through the built-in camera and microphone. This data is sent to the server.

[1213] 2. Data analysis and proposal generation

[1214] The server receives the photo data, requests, facial expressions, and voice data sent from the device. The photo data is analyzed using an image analysis algorithm to determine the size of the room, the layout of existing furniture, the color of the walls, and other information. The analysis results and the user's requests are input into a generative AI model to generate a list of optimal furniture and home appliances.

[1215] 3. Emotion recognition and suggestion adjustment

[1216] The server uses an emotion recognition engine to analyze the user's facial and voice data and recognize emotions such as joy, surprise, and dissatisfaction. For example, a machine learning library is used for facial expression analysis, and a voice recognition algorithm is used for voice analysis. Based on the emotion recognition results, the generated candidate list of furniture and home appliances is adjusted according to the user's emotions.

[1217] 4. Obtaining product information and generating layouts

[1218] The server accesses the APIs of multiple retailers based on the adjusted candidate list to obtain product information. The obtained product information includes product name, size, price, stock status, etc., and generates a 3D simulation model based on this information. This 3D simulation virtually arranges furniture in a room and is created using a 3D graphics library (e.g., Unity or Three.js).

[1219] 5. View and confirm the proposal

[1220] The terminal displays the 3D simulation model and product information received from the server to the user. The user can check the 3D simulation through an interface using smart glasses. In addition, it is possible to use AR technology to place virtual furniture in a real room and check how it will look. If the user is satisfied with the suggestions, they can add the desired products to their cart and complete the purchase process.

[1221] 6. Purchase Procedures and Transaction Notifications

[1222] The user checks the proposed furniture and appliance arrangement, and if they like it, presses the "Purchase" button to add it to their cart. The device receives the user's purchase request and payment information, encrypts it, and sends it to the server. The server accesses the seller's API to check inventory and initiate the delivery process, and once the order is confirmed, notifies the user's device of the transaction status. This allows the user to check the transaction progress and delivery information in real time.

[1223] Specific examples

[1224] Example 1: To make the room appear larger

[1225] The user takes three photos of a 30-square-meter apartment for a single person and inputs the request to "make it look larger." The emotion engine then collects the user's facial and voice data to recognize emotions such as joy and dissatisfaction. The server analyzes the photo data and the request to identify the room's dimensions and the location of existing furniture. Based on the request, the generative AI model suggests a glass table or a small sofa, and the suggestions are adjusted based on the emotion recognition results. The server then obtains the adjusted product information from multiple sellers, generates a 3D simulation, and sends it to the device. The user reviews the 3D simulation and, if they like it, purchases the suggested product. The server then sends the order to the seller and notifies the device of the transaction status and delivery information.

[1226] Prompt Sentence Examples

[1227] "Please suggest interior design ideas to make a 30-square-meter single-person room look larger. Judging from the user's expressions, they seem surprised."

[1228] This system allows users to receive optimal interior design suggestions tailored to their own emotions and desires in real time at a physical store.

[1229] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1230] Step 1:

[1231] The user wears the smart glasses and takes three photos of the room. At this time, the user inputs their preferences, such as "I want the room to look spacious" or "I want a modern design," through the interface. In addition, the smart glasses' camera and microphone are used to collect the user's facial and voice data. The input data includes the room photo, the desired text, facial expression data, and voice data. The device then sends this data to the server.

[1232] Step 2:

[1233] The server uses image analysis algorithms (e.g., OpenCV, TensorFlow) to analyze the received photo data of the room. This identifies the room's size, the layout of existing furniture, the wall color, etc. The request text is also analyzed, and the server integrates this data and stores it in a database. The input data are the room photo and the user's request, and the output data are the analysis results, such as the recognized room size, furniture layout, and wall color.

[1234] Step 3:

[1235] The server uses an emotion recognition engine (e.g., Affectiva SDK, Microsoft Azure Cognitive Services) to analyze the user's facial and voice data. This identifies the user's emotions, such as joy, surprise, or dissatisfaction. The input data are facial and voice data, and the output data are the recognized emotions.

[1236] Step 4:

[1237] The server inputs prompts into the AI ​​model based on the image analysis results, emotion recognition results, and the user's requests. For example, a prompt such as "Please suggest interior design ideas to make a 30-square-meter room for a single person look larger. Judging from the user's facial expression, an expression of surprise appears." is generated and input into the AI ​​model. This generates a list of suitable furniture and home appliance candidates. The input data are the analysis results and the prompt, and the output data is a list of furniture and home appliance candidates.

[1238] Step 5:

[1239] Based on the generated candidate list of furniture and home appliances, the server accesses the APIs of multiple retailers to obtain product information (product name, price, size, and stock status). This allows the collection of the most up-to-date and detailed product information. The input data is the candidate list, and the output data is the obtained product information.

[1240] Step 6:

[1241] The server generates a 3D simulation model based on the acquired product information. This 3D model virtually arranges furniture in the user's room and is created using a 3D graphics library (e.g., Unity, Three.js). The input data is product information and furniture arrangement plans, and the output data is the 3D simulation model.

[1242] Step 7:

[1243] The terminal (smart glasses) displays the 3D simulation model and product information received from the server to the user. Through the smart glasses, the user can check the 3D simulation and see in real time how the virtual furniture will look when placed in a real room using AR technology. The input data is the 3D simulation model, and the output data is the user's visual confirmation.

[1244] Step 8:

[1245] If the user is satisfied with the proposed furniture or appliances, they press the "Purchase" button through the smart glasses interface to add them to their cart. The terminal then encrypts the purchase request and payment information and sends it to the server. The input data are the purchase request and payment information, and the output data is the encrypted transaction data.

[1246] Step 9:

[1247] The server accesses the seller's API to check inventory and initiate delivery procedures, and once the order is confirmed, it notifies the user's device of the transaction status. This allows the user to check the transaction progress and delivery information in real time. The input data is encrypted transaction data, and the output data is transaction status information.

[1248] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1249] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1250] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1251] [Fourth embodiment]

[1252] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1253] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1254] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1255] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1256] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1257] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1258] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1259] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1260] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1261] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1262] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1263] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1264] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1265] MODE FOR CARRYING OUT THE INVENTION

[1266] System Configuration

[1267] The "AI Interior Coordinator" system of this invention is primarily composed of a terminal operated by the user, a server that processes data, and a product information database that utilizes the APIs of multiple retailers. The system starts when the user takes a photo of the room and inputs their requirements, and then makes interior design suggestions based on those requirements.

[1268] Program processing

[1269] The program processing of this system will be explained in natural language below.

[1270] 1. User Input and Data Submission

[1271] The user first takes a photo of the room. It is recommended to take several photos from different angles to capture the entire room. The user also inputs their requests for the room layout in text format. For example, they might say, "I want the room to look spacious" or "I want it to have a modern feel." When the user enters this information into the app and presses the "Send" button, the data is sent to the server via their device.

[1272] 2. Data analysis and proposal generation

[1273] The server receives the photo data and requests sent from the device. The received photo data is first analyzed using an image analysis algorithm. This analysis identifies the size of the room, the layout of existing furniture, the color of the walls, and other factors. For example, the OpenCV library or the YOLO object detection algorithm can be used. Next, based on the user's requests, the data is input into a generative AI model (e.g., GPT-4) to generate optimal furniture and appliance candidates. For example, a glass table or a small sofa could be suggested to make the room appear larger.

[1274] 3. Obtaining product information and generating layouts

[1275] The server accesses the APIs of multiple retailers to obtain product information for the generated list of furniture and home appliance candidates. This product information includes product name, price, size, and stock information. Based on the obtained product information, the server generates the optimal layout for the user's room. This layout is generated as a 3D simulation model and provided to the user. For example, Unity or Three.js can be used to create the 3D model and virtually arrange the furniture in the room.

[1276] 4. View and confirm the proposal

[1277] The terminal displays the 3D simulation model and furniture list received from the server to the user. The user can view this 3D simulation via a smartphone or tablet. In addition, it is possible to virtually place furniture in a real room using AR technology and check how it will look. If the user is satisfied with the suggestions, they can add the desired items to their cart and complete the purchase process.

[1278] 5. Purchase Procedures and Transaction Notifications

[1279] If the user likes the suggested items, they press the "Purchase" button to add them to their cart. As part of the purchasing process, they enter their payment information and confirm the order. The terminal then sends the user's purchase request and payment information to the server. The server accesses the seller's API to check inventory and process delivery, and once the order is confirmed, it notifies the user's terminal of the transaction status. This notification allows the user to check the progress of the transaction and delivery information in real time.

[1280] Specific examples

[1281] Example 1: Make the room look larger

[1282] Suppose a user takes three photos of a 30-square-meter apartment for a single person and inputs the request to make it appear larger. The device sends this data to a server, which analyzes the photo data and identifies the dimensions of the room and the location of existing furniture. Based on the request, the generative AI model suggests a glass table and a small sofa, and obtains information on these products from multiple retailers.

[1283] The server generates a 3D simulation based on the acquired product information and sends it to the terminal. The user can check the 3D simulation and, if they like it, purchase the suggested product. The server then sends an order to the seller and notifies the terminal of the transaction status and delivery information.

[1284] The processing flow will be explained below.

[1285] Step 1:

[1286] The user takes three photos of the room. The photos are taken from different angles to capture information about the entire room. In addition, the user inputs preferences through the interface, such as "I want it to look spacious" or "I want a modern design."

[1287] Step 2:

[1288] The device receives the user's photo data and requests. This data is converted into the appropriate format and prepared for transmission. The device also encrypts the data to ensure security.

[1289] Step 3:

[1290] The device sends the encrypted photo data and request to the server using a secure communication protocol (e.g., HTTPS). It is recommended to use a checksum during data transmission to prevent errors.

[1291] Step 4:

[1292] The server stores the received photo data and requests in a database. To begin analyzing the photo data, it first sends the data to an image analysis algorithm, which uses libraries such as OpenCV and YOLO to identify the size of the room, the existing furniture arrangement, and the color of the walls.

[1293] Step 5:

[1294] The server inputs the analysis results and the user's requests into a generative AI model (e.g., GPT-4) to generate a list of suitable furniture and appliances. For example, furniture that makes a room appear larger might include a glass table, a small sofa, and a mirror with reflective materials.

[1295] Step 6:

[1296] Based on the generated candidate list, the server accesses the APIs of multiple retailers to obtain product information, including product name, size, price, stock status, etc. This information is then stored in a database.

[1297] Step 7:

[1298] The server generates a 3D simulation model based on the acquired product information. This simulation model virtually arranges furniture in the room layout and is created using a 3D graphics library (e.g., Unity or Three.js).

[1299] Step 8:

[1300] The server then sends the generated 3D simulation model and product information to the terminal, where the data is packetized in an appropriate format and transmitted again using a secure communication protocol.

[1301] Step 9:

[1302] The device analyzes the received data and displays a 3D simulation to the user, who can view the simulation through the interface of their smartphone or tablet. Additionally, by selecting the AR mode, it is possible to place virtual furniture in a real room.

[1303] Step 10:

[1304] The user checks the proposed furniture and appliance arrangement, and if they like it, they press the "Purchase" button, which adds the desired items to their cart and proceeds to the purchase process.

[1305] Step 11:

[1306] The terminal receives the purchase request and payment information from the user, encrypts it, and sends it to the server. The payment information is handled using a security protocol (e.g., TLS).

[1307] Step 12:

[1308] The server accesses the seller's API to check inventory and initiate the shipping process. Once the order is confirmed, the server calculates the commission and secures the revenue.

[1309] Step 13:

[1310] After the order is confirmed and the delivery procedure is completed, the server notifies the terminal of the transaction status, allowing the user to check the transaction progress and delivery information in real time.

[1311] Step 14:

[1312] The terminal displays transaction status and delivery information to the user and notifies them that the purchase has been completed.

[1313] Example 1

[1314] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1315] Conventional interior coordination systems have issues such as difficulty for users in visualizing the room layout, lack of information, and the time and effort required to make a purchasing decision. Furthermore, when acquiring product information from multiple retailers, data acquisition and management can be cumbersome, making it necessary to develop an efficient system. In addition, users lack a visual experience to confirm the proposals, making it difficult to visualize the actual layout.

[1316] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1317] In this invention, the server includes: a means for a user to acquire an image of a room and input their requirements; a means for transmitting the image data and the requirements to an information processing device; a means for analyzing the image data and identifying the dimensions and existing layout of the room; a means for generating optimal furniture and electronic device candidates based on the identification results and the requirements using a generative AI model; a means for acquiring product information from multiple vendors; a means for generating a layout based on the acquired product information and displaying a 3D simulation to the user; and a means for the user to check the 3D simulation and then proceed with purchasing the products. This allows the user to visually confirm the room layout, making it easier to make a purchasing decision through efficient information acquisition and a visual experience.

[1318] "User" refers to an individual or organization that uses the system.

[1319] "Image" refers to a digital photograph or graphic visual record of the exterior or interior of a room.

[1320] "Information processing device" refers to a computer system that receives, analyzes, generates, and transmits data.

[1321] "Analysis" refers to the process of determining room dimensions and furniture layout based on image data.

[1322] "Dimensions" refers to physical measurements related to the size of a room or furniture.

[1323] "Layout" refers to the relative positions of furniture and decorations within a room.

[1324] "Specific results" refers to data regarding the dimensions and layout of the room obtained through analysis.

[1325] "Candidates" refers to a list of furniture and electronic devices that are optimally selected based on the user's requirements.

[1326] A "generative AI model" refers to an algorithm that uses machine learning to suggest furniture and electronic devices that meet user needs.

[1327] "Product information" refers to data such as the name, price, size, and inventory of furniture and electronic devices.

[1328] "Layout" refers to a virtual layout of furniture and decorations in a room.

[1329] "3D simulation" refers to a model that visually reproduces the furniture arrangement in a room using 3D computer graphics.

[1330] "Augmented reality technology" refers to technology that overlays digital information onto real space.

[1331] "Retailer" refers to a business or store that sells furniture or electronics.

[1332] "Checkout" refers to the process by which a user selects the desired product, enters payment information, and confirms the order.

[1333] "Order" refers to a transaction request for an item that a user wishes to purchase.

[1334] MODE FOR CARRYING OUT THE INVENTION

[1335] The "AI Interior Coordinator" system of this invention begins when a user acquires a photo of a room and inputs their requirements. The photo data taken by the user and the input requirements are sent to an information processing device (server), which then performs a series of data analyses and proposes interior designs. Specific embodiments are described below.

[1336] User Actions

[1337] First, the user launches a dedicated application and takes photos of the room from multiple angles. For example, it is recommended to take four photos: one from the front, one from the left, one from the right, and one from the back. The user also enters their room-related requirements in text format (e.g., "I want the room to look spacious" or "I want a modern feel"). Once these are entered, the user presses the "Send" button in the application. This sends the photo data and requirements from the device to the information processing device (server).

[1338] Server Operation

[1339] The server receives the submitted photo data and request. First, it analyzes the photo data using the OpenCV library and the YOLO object detection algorithm to determine the room dimensions and the layout of existing furniture. Specifically, it uses the YOLO algorithm to identify furniture and obtain the position and size of each piece of furniture. Next, it uses a generative AI model (e.g., GPT-4) based on the user's request to generate optimal furniture and electronic device candidates. At this time, it inputs data into the generative AI model using prompt statements.

[1340] Prompt Sentence Examples

[1341] "Please suggest the best interior coordination to make a 30 square meter room look larger. Using glass furniture and a small sofa."

[1342] "Can you give me some ideas for modernizing this room? The current furniture arrangement is included in the images."

[1343] Obtaining product information

[1344] Based on the generated candidate list of furniture and electronic devices, the server accesses the APIs of multiple vendors to obtain corresponding product information. The obtained product information includes product name, price, size, stock information, etc. For example, the product information is obtained using the API of an internet shopping mall.

[1345] Layout generation and display

[1346] The server uses Unity or Three.js to generate a 3D simulation model based on the acquired product information and create an optimal layout. These 3D simulations include furniture virtually placed in the room. This model is sent to the device so that the user can visually check it.

[1347] Proposal confirmation and purchase process

[1348] The device displays the 3D simulation model and product list received from the server to the user. The user can view this 3D simulation through the app. Augmented reality technology using the smartphone camera also allows the user to place virtual furniture in a real room and simulate how it will look. If the user is satisfied with the suggestions, they can add the desired products to their cart and proceed with the purchase process.

[1349] Purchase Process and Transaction Notifications

[1350] The user presses the "Purchase" button, adds the product to the cart, enters payment information, and confirms the order. The terminal sends the user's purchase request and payment information to the server. The server accesses the vendor's API to check inventory and process delivery, and once the order is confirmed, it notifies the user's terminal of the transaction status and delivery information. This allows the user to check the progress of the transaction and product delivery information in real time.

[1351] In this way, the "AI interior coordinator" of the present invention provides users with an efficient and intuitive interior coordination proposal and purchasing process.

[1352] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1353] Step 1: User Input and Data Submission

[1354] The user uses a dedicated application to take photos of the room from multiple angles. As input data, they take photos of the front, left, right, and back of the room, and enter their requests in text format, such as "I want the room to look spacious" or "I want a modern feel." When they press the "Send" button, this data is sent from the device to the server. As output, the photo data and the requests in text format are sent to the server.

[1355] Step 2: Receiving and initial processing of data

[1356] The server receives the photo data and requests sent from the terminal. As input data, it receives the photo data and requests sent from the user. Based on this, it performs initial processing of the data and prepares it for data analysis. As output, it generates the photo data and requests converted into the format required for analysis.

[1357] Step 3: Data analysis

[1358] The server analyzes the photo data using the OpenCV library and the YOLO object detection algorithm. It uses the photo data formatted for processing as input data. Based on this, it identifies furniture and room dimensions in the image and extracts attribute information (position, size, color, etc.) of the identified objects. The output is the room dimensions and the existing furniture layout.

[1359] Step 4: Proposal generation using generative AI models

[1360] The server inputs the identification results and the user's requests into a generative AI model (e.g., GPT-4) and generates optimal furniture and electronic device candidates. The input data includes the room dimensions, existing furniture layout, and the user's requests (in text format). Based on the generated prompt, the AI ​​model analyzes and outputs specific furniture and interior design suggestions. The output is a list of suggested furniture and electronic devices.

[1361] Step 5: Obtaining product information using the API

[1362] The server accesses the APIs of multiple vendors to obtain product information based on the generated list of furniture and electronic device candidates. The server uses the list of proposed furniture and electronic devices as input data. Based on this, it makes API requests to obtain product names, prices, sizes, stock information, etc. The server generates a list of the obtained product information as output.

[1363] Step 6: Generate a 3D simulation model

[1364] The server uses Unity or Three.js to generate a 3D simulation model based on the acquired product information. The input data is a list of product information and the dimensions of the existing room. Based on this, it creates an optimal layout and generates a 3D model of the furniture virtually placed in the room. The generated 3D simulation model is obtained as the output.

[1365] Step 7: View and Review the Simulation

[1366] The device displays the 3D simulation model and product list received from the server to the user. The 3D simulation model sent from the server is used as input data. The user can visually check this 3D simulation through the app. It is also possible to use augmented reality technology to place virtual furniture in a real room and simulate how it will look. The output is a visual 3D simulation that the user can check.

[1367] Step 8: Checkout

[1368] Based on the suggestions they like, the user presses the "Purchase" button, adds the product to their cart, enters payment information, and confirms the order. The input data is product information and payment information. Based on this, the purchase procedure is carried out. As an output, information indicating that the user has completed the purchase procedure is sent to the server.

[1369] Step 9: Transaction Notification

[1370] The server accesses the seller's API based on the user's purchase request and payment information to check inventory and process delivery. Detailed purchase process information is used as input data. Once the order is confirmed, transaction status and delivery information is sent to the user's device. As an output, the user receives notifications that allow them to check the transaction progress and delivery information in real time.

[1371] (Application example 1)

[1372] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1373] In today's brick-and-mortar stores, customers often have difficulty visualizing the product in its actual placement when selecting the right interior product for their home. Traditionally, purchasing decisions are often made solely based on catalogs and displays, increasing the risk of making the wrong choice. Similarly, online shopping makes it difficult to confirm the product's suitability in the actual room environment. This leads to an increase in returns and exchanges after purchase, lowering customer satisfaction and increasing operational costs.

[1374] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1375] In this invention, the server includes: a means for a user to take an image of a room and input their preferences; a means for transmitting the image data and preferences to the server; a means for analyzing the image data to recognize the size of the room and the existing item layout; a means for generating optimal device and tool candidates based on the recognition results and the user's preferences; a means for acquiring product information from multiple sellers; a means for generating a layout based on the acquired product information and displaying a 3D simulation to the user; a means for the user to check the 3D simulation and then complete a product purchase procedure; and a means for displaying virtual items arranged in a real space and using augmented reality technology to confirm the display. This allows users to visually confirm the actual layout of virtual interior items in a physical store or at home. Furthermore, customers can accurately confirm the suitability of products before purchase, preventing incorrect selections and reducing returns and exchanges.

[1376] A "user" refers to an individual who uses the system to take pictures and input their preferences to coordinate the interior of a room.

[1377] "Image data" refers to photographic data of a room taken by a user, and includes information such as the size of the room and the layout of existing items.

[1378] "Wishes" are requests that the user inputs regarding the interior of the room, and refer to specific requests such as "I want the room to look spacious."

[1379] "Server" refers to a computer system that receives image data and requests sent by users and performs analysis and data processing.

[1380] "Analysis" refers to the process of identifying the size of a room and the layout of existing items based on image data, and is done using computer vision techniques and object recognition algorithms.

[1381] "Device and tool candidates" refers to a list of optimal interior products suggested by the system based on the user's preferences and analysis results.

[1382] "Sellers" refer to companies and stores that sell interior products, and the system obtains product information from them.

[1383] "Product Information" refers to detailed data about interior products provided by sellers, such as product name, price, size, and inventory information.

[1384] "Layout" refers to the layout of interior products virtually arranged in a room based on the acquired product information.

[1385] "Three-dimensional simulation" refers to three-dimensional simulation data for visually displaying the generated layout to the user.

[1386] "Augmented reality technology" refers to a technology that displays virtual objects superimposed on a real-world environment, allowing users to visually check virtual interior products in real space.

[1387] "Purchase procedure" refers to the series of operations that a user performs to actually purchase interior products suggested through the system.

[1388] This invention provides an "interior AR shopping assistant" system that allows users to try out interior products in real time in a "real store." To realize this system, the following hardware and software are required:

[1389] Hardware and software used

[1390] Hardware: Requires a smartphone (camera, display, internet connection).

[1391] Software: OpenCV, YOLO (object detection), Unity (3D simulation), generative AI model (GPT-4API), etc. are required.

[1392] System configuration and operation

[1393] 1. User Action:

[1394] Users use their smartphones to take pictures of their room from multiple angles and input their desired interior style and requirements in text format, such as "I want the room to look spacious" or "I want a modern feel."

[1395] 2. Data transmission:

[1396] The user's terminal transmits the captured image data and the desired text data to the server.

[1397] 3. Analysis of imaging data:

[1398] The server analyzes the received image data using OpenCV and YOLO to determine the size of the room and the layout of existing items. This analysis determines the floor area, the location of walls, and how existing furniture is arranged.

[1399] 4. Generate optimal furniture candidates:

[1400] The server uses a generative AI model (GPT-4) to generate optimal interior product candidates based on the analyzed data and the user's preferences. For example, if the user desires a "modern taste" for a living room measuring approximately 20 square meters, the generative AI model will suggest a simple white sofa and a glass coffee table.

[1401] Example prompt sentence:

[1402] Room features: 20 square meters, white walls, existing furniture: dark wooden table and bookshelf.

[1403] User's request: I want a modern and simple interior.

[1404] 5. Obtaining product information:

[1405] The server obtains information about the interior products listed as candidates through APIs of multiple sellers, including product names, prices, sizes, and stock information.

[1406] 6. Generation of 3D simulations:

[1407] The server uses Unity to create a 3D simulation based on the acquired product information, and the simulation is generated in a virtual room with the proposed furniture arranged.

[1408] 7. View the simulation:

[1409] The user's device displays the 3D simulation sent from the server and uses augmented reality (AR) technology to overlay virtual interior products onto the real space, allowing the user to visually confirm how the virtual interior products will actually look when placed in a physical store or at home.

[1410] Specific examples

[1411] For example, if a user wants to make a 30-square-meter living room look modern and spacious, the system analyzes images taken from various angles of the room and generates suggestions such as a white sofa and a glass table. Product information is then acquired and a 3D simulation is generated based on the acquired information. The user can then virtually place these suggested furniture items in their real room using their smartphone, see how they will look, and then purchase the items they like.

[1412] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1413] Step 1:

[1414] The user takes a picture of the room and inputs their preferences.

[1415] The user uses a smartphone to take multiple images of the room from different angles. They also input their preferences for the room's interior in text format. For example, they might input a preference such as "I want the room to look spacious." The input data is saved as image data and text data.

[1416] Step 2:

[1417] The device sends the photo data and your request to the server.

[1418] The device sends the image data and text data the user has entered to the server, which then securely transfers the data over the Internet and stores it on the server.

[1419] Step 3:

[1420] The server analyzes the photo data and recognizes the size of the room and the layout of existing items.

[1421] The server analyzes the received image data using OpenCV and YOLO object detection algorithms. This analysis identifies the size of the room, the location of walls, and the layout of existing furniture. For example, the floor area and furniture locations are automatically calculated.

[1422] Step 4:

[1423] The server generates optimal device and tool candidates based on the recognition results and user preferences.

[1424] The server uses a generative AI model (GPT-4) based on the analysis results and the user's preferences to generate optimal device and tool candidates. The analysis results and the user's preferences are input into the prompt text.

[1425] Example prompt sentence:

[1426] Room features: 20 square meters, white walls, existing furniture: dark wooden table and bookshelf.

[1427] User's wish: A modern and simple interior.

[1428] Based on this prompt, the generative AI model suggests specific interior product options, such as a simple white sofa or a glass table.

[1429] Step 5:

[1430] The server retrieves product information from multiple sellers

[1431] Based on the generated list of furniture and tool candidates, the server accesses the APIs of multiple sellers to obtain product information. The obtained product information includes product name, price, size, stock information, etc. This information is imported into the server as structured data in JSON format, etc.

[1432] Step 6:

[1433] The server generates a layout based on the product information acquired and creates a 3D simulation.

[1434] The server uses Unity to create a 3D simulation based on the acquired product information. The placement data is constructed as a 3D model, and a simulation of the virtual furniture placement in the room is generated. This simulation data is visualized for the user.

[1435] Step 7:

[1436] The server sends the 3D simulation to the terminal, which displays it.

[1437] The server sends the generated 3D simulation to the user's device, which then displays the received simulation data, allowing the user to use augmented reality (AR) technology to overlay virtual furniture onto the real space, creating a visual experience that makes it seem as if the new furniture is actually placed in the room.

[1438] Step 8:

[1439] The user completes the purchase process

[1440] The user checks the 3D simulation and selects the product they wish to purchase. As part of the purchase process, they enter payment information and confirm the order. The terminal sends the user's purchase request and payment information to the server, which then accesses the seller's API to check inventory and process delivery. Once the order is confirmed, the user's terminal is notified of the transaction status and delivery information.

[1441] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1442] MODE FOR CARRYING OUT THE INVENTION

[1443] System Configuration

[1444] The "AI Interior Coordinator" system of this invention not only proposes optimal furniture and home appliances based on user input data, but also recognizes the user's emotions and adjusts the proposals accordingly. This system is primarily composed of a terminal operated by the user, a server that performs data analysis and emotion recognition, and a product information database that utilizes the APIs of multiple retailers.

[1445] Program processing

[1446] The program processing of this system will be explained in natural language below.

[1447] 1. User Input and Data Submission

[1448] The user takes three photos of the room and inputs their requirements. It is recommended that the photos be taken from different angles to capture the entire room. The user inputs their requirements, such as "I want the room to look spacious" or "I want a modern design." In addition, the user collects their facial expressions and voice data through the interface. This data is sent to the emotion engine.

[1449] 2. Data analysis and proposal generation

[1450] The server receives the photo data, requests, facial expressions, and voice data sent from the device. The photo data is analyzed using an image analysis algorithm to determine the size of the room, the layout of existing furniture, the color of the walls, etc. The analysis results and the user's requests are input into a generative AI model to generate a list of optimal furniture and home appliances.

[1451] 3. Emotion recognition and suggestion adjustment

[1452] The server uses an emotion engine to analyze the user's facial and voice data and recognize emotions such as joy, surprise, and dissatisfaction. For example, a machine learning library is used for facial expression analysis, and a voice recognition algorithm is used for voice analysis. Based on the emotion recognition results, the generated candidate list of furniture and home appliances is re-adjusted according to the user's emotions.

[1453] 4. Obtaining product information and generating layouts

[1454] The server accesses the APIs of multiple retailers based on the adjusted candidate list to obtain product information. The obtained product information includes product name, size, price, stock status, etc., and generates a 3D simulation model based on this information. This 3D simulation virtually arranges furniture in a room and is created using a 3D graphics library (e.g., Unity or Three.js).

[1455] 5. View and confirm the proposal

[1456] The terminal displays the 3D simulation model and product information received from the server to the user. The user can check the 3D simulation through the interface of their smartphone or tablet. In addition, it is possible to use AR technology to place virtual furniture in a real room and check how it will look. If the user is satisfied with the suggestions, they can add the desired products to their cart and complete the purchase process.

[1457] 6. Purchase Procedures and Transaction Notifications

[1458] The user checks the proposed furniture and appliance arrangement, and if they like it, presses the "Purchase" button to add it to their cart. The device receives the user's purchase request and payment information, encrypts it, and sends it to the server. The server accesses the seller's API to check inventory and initiate delivery procedures, and once the order is confirmed, notifies the user's device of the transaction status. This allows the user to check the transaction progress and delivery information in real time.

[1459] Specific examples

[1460] Example 1: To make the room appear larger

[1461] A user takes three photos of a 30-square-meter apartment for a single person and inputs the request to "make it look larger." The emotion engine then collects the user's facial and voice data to recognize emotions such as joy and dissatisfaction. The server analyzes the photo data and the request to identify the room's dimensions and the location of existing furniture. Based on the request, the generative AI model suggests a glass table and a small sofa, and the suggestions are adjusted based on the emotion recognition results.

[1462] The server obtains the coordinated product information from multiple sellers, generates a 3D simulation, and sends it to the device. The user reviews the 3D simulation and, if they like it, purchases the suggested product. The server then sends the order to the seller and notifies the device of the transaction status and delivery information.

[1463] These specific processes enable optimal interior design proposals that reflect the user's emotions.

[1464] The processing flow will be explained below.

[1465] Step 1:

[1466] The user takes three photos of the room. The photos are taken from different angles to gather information about the entire room. The user also inputs requests, such as "I want the room to look larger." In addition, an interface is used to send the user's facial expressions and voice to the emotion engine.

[1467] Step 2:

[1468] The device collects the user's photos, input requests, and facial and voice data, which are then converted into a format for recognition and all data is prepared for transmission.

[1469] Step 3:

[1470] The device sends the photo data, desires, and emotion data in encrypted form to the server using a secure communication protocol (e.g., HTTPS).

[1471] Step 4:

[1472] The server stores the received photo data, requests, and emotion data in a database. The photo data is input into an image analysis algorithm (e.g., OpenCV, YOLO) to extract information such as the size of the room, the layout of existing furniture, and the color of the walls.

[1473] Step 5:

[1474] The server inputs the image analysis results and the customer's requests into a generative AI model to generate an initial list of furniture and appliance candidates. For example, a glass table or a small sofa might be selected to make the room appear larger.

[1475] Step 6:

[1476] The server inputs the user's facial expression and voice data into the emotion engine to recognize the user's emotions. Emotions such as joy, dissatisfaction, and surprise are recognized, and emotion recognition results are generated.

[1477] Step 7:

[1478] The server then adjusts the list of furniture and appliance candidates generated by the generative AI model based on the emotion recognition results. For example, if the user expresses dissatisfaction, furniture with a different design or functionality will be suggested.

[1479] Step 8:

[1480] Based on the adjusted candidate list, the server accesses the APIs of multiple sellers to retrieve product information, including product name, price, size, and availability, and stores it in a database.

[1481] Step 9:

[1482] The server generates a 3D simulation model based on the acquired product information. This simulation model virtually arranges furniture in a room and is created using a 3D graphics library (e.g., Unity, Three.js).

[1483] Step 10:

[1484] The server then transmits the generated 3D simulation model and product information to the terminal using a communication protocol. The data is then packetized and transmitted again using a secure communication protocol.

[1485] Step 11:

[1486] The device analyzes the received 3D simulation model and product information and displays it to the user. The user can view the 3D simulation via a smartphone or tablet. It is also possible to use the AR function to place virtual furniture in a real room.

[1487] Step 12:

[1488] The user checks the proposed furniture and appliance arrangement, and if satisfied, presses the "Purchase" button, which adds the desired items to the cart and proceeds with the purchase.

[1489] Step 13:

[1490] The terminal receives the purchase request and payment information from the user, encrypts it, and transmits it to the server using a security protocol.

[1491] Step 14:

[1492] The server accesses the seller's API to check inventory and initiate the shipping process. Once the order is confirmed, the server calculates the commission and secures the revenue.

[1493] Step 15:

[1494] The server completes the order confirmation and delivery procedures and notifies the terminal of the transaction status, allowing the user to check the transaction progress and delivery information in real time.

[1495] Step 16:

[1496] The terminal displays transaction status and delivery information to the user and notifies them that the purchase has been completed.

[1497] Example 2

[1498] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1499] Interior design has become an important element in modern lifestyles. However, it is not easy for users to select and appropriately arrange furniture and appliances themselves. In particular, it is difficult to obtain proposals that reflect one's own feelings and desires. This results in users spending a great deal of time and effort to create a satisfying space. A system that can solve this problem and provide optimal interior proposals more quickly and efficiently is needed.

[1500] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1501] In this invention, the server includes: means for a user to acquire an image of a room and input their requirements; means for transmitting the image data and the requirements to the server; means for analyzing the image data and recognizing the size of the space and the existing furniture layout; means for generating optimal furniture and home appliance candidates based on the recognition results and the requirements using a generative AI model; means for analyzing the user's facial expressions and voice data and recognizing emotions; means for adjusting the generated furniture and home appliance candidates based on the recognized emotions; means for acquiring product information from multiple retailers; means for generating a layout based on the acquired product information and displaying a 3D simulation to the user; and means for the user to check the 3D simulation and then proceed with purchasing the products. This allows the user to quickly and effectively obtain interior design suggestions that reflect their emotions and requirements.

[1502] "User" refers to the entity that uses the system to receive interior design proposals.

[1503] "Image" refers to visual data captured of the room situation.

[1504] "Requests" refer to the requirements for interior design desired by the user.

[1505] A "terminal" is an electronic device that allows a user to take pictures and input data.

[1506] "Server" means a central processing unit for analyzing received data, generating and managing offers.

[1507] "Image data" refers to digital information obtained from images of a room taken by a user.

[1508] "Analysis" refers to the process of extracting and understanding specific information from image data, audio data, etc.

[1509] "Space" refers to the physical size or area of ​​a room.

[1510] "Existing furniture arrangement" refers to the position and layout of furniture already placed in the room.

[1511] A "generative AI model" is an algorithm that uses artificial intelligence to generate suggestions based on user requests.

[1512] "Facial expression data" refers to digital information that captures a user's facial movements and expressions.

[1513] "Voice data" refers to digital information that records a user's speech or voice.

[1514] "Emotion" refers to information that indicates the user's psychological state.

[1515] "Adjustment" is the process of modifying the proposal to better suit the user's feelings and desires.

[1516] "Retailer" refers to a company or store that offers furniture or home appliances.

[1517] "Product information" refers to detailed product information provided by a retailer.

[1518] "Layout" refers to the arrangement plan of furniture and appliances.

[1519] "3D simulation" is a technology that visualizes the placement of furniture and home appliances in a virtual space.

[1520] "Augmented reality technology" is a technology that displays virtual elements superimposed on real scenery.

[1521] The "purchase procedure" refers to the procedure for actually purchasing the proposed product.

[1522] MODE FOR CARRYING OUT THE INVENTION

[1523] The "AI Interior Coordinator" system of this invention not only proposes optimal furniture and home appliances based on user input data, but also recognizes the user's emotions and adjusts the proposals accordingly. This system is primarily composed of a terminal operated by the user, a server that performs data analysis and emotion recognition, and a product information database that utilizes the APIs of multiple retailers.

[1524] User Input and Data Submission

[1525] The user takes three photos of the room using the device's camera and inputs their preferences, such as "I want the room to look spacious" or "I want a modern design." The photos should be taken from different angles to capture the entire room. The user also collects their facial expressions and voice data through the interface. This data is encrypted and sent to the server.

[1526] Data analysis and proposal generation

[1527] The server receives the photo data, requests, facial expressions, and voice data sent from the device. The photo data is analyzed using image analysis algorithms (e.g., OpenCV, TensorFlow). As a result of the analysis, the size of the room, the layout of existing furniture, the color of the walls, etc. are identified.

[1528] Next, the server inputs the analysis results and the user's request into a generative AI model (e.g., GPT-4, DALL-E). The prompt is in the form of "Please suggest furniture arrangements that will make a 30-square-meter room appear larger," and the AI ​​model generates a list of optimal furniture and appliance candidates.

[1529] Emotion recognition and suggestion adjustment

[1530] The server analyzes the user's facial expressions and voice data using an emotion engine (e.g., Microsoft Azure Emotion API, Google Cloud Vision API). For example, it uses face-api.js to analyze facial movements and Google Speech-to-Text API to analyze voice data. The emotion engine identifies emotions such as joy, surprise, and dissatisfaction and adjusts the generated list of furniture and appliance candidates.

[1531] Obtaining product information and generating layouts

[1532] Based on the adjusted candidate list, the server accesses APIs of multiple sellers (e.g., Amazon Product Advertising API, Rakuten API) to obtain product information. The obtained product information includes product name, size, price, and stock status. A 3D simulation model is generated based on this information. This simulation is created using a 3D graphics library (e.g., Unity, Three.js).

[1533] View and review suggestions

[1534] The device displays the 3D simulation model and product information received from the server to the user. Users can check the 3D simulation through the interface of their smartphone or tablet. Furthermore, it is possible to place virtual furniture in a real room using AR technology (e.g., ARKit, ARCore) and check how it will look.

[1535] Purchase Process and Transaction Notifications

[1536] The user checks the proposed furniture and appliance arrangement, and if they like it, they press the "Purchase" button to add it to their cart. The device receives the user's purchase request and payment information, encrypts it, and sends it to the server. The server checks inventory and initiates delivery procedures via the seller's API, and once the order is confirmed, notifies the user's device of the transaction status. This allows the user to check the transaction progress and delivery information in real time.

[1537] Examples of concrete examples and prompts

[1538] Specific examples

[1539] The user takes three photos of a 30-square-meter apartment for a single person and inputs the request to "make it look larger." The emotion engine then collects the user's facial expressions and voice data to recognize emotions. The server analyzes the photo data and the request to identify the room's dimensions and the location of existing furniture. Based on the request, the generative AI model suggests a glass table or a small sofa, and the suggestions are adjusted based on the emotion recognition results. The adjusted product information is obtained from multiple sellers, and a 3D simulation is generated and sent to the device. The user reviews the 3D simulation and, if they like it, purchases the suggested product. The server then sends the order to the seller and notifies the device of the transaction status and delivery information.

[1540] Prompt Sentence Examples

[1541] "Please suggest the best furniture layout for a 30 square meter room for a single person."

[1542] "Please tell me some interior design ideas to make a room look larger."

[1543] "Please suggest some furniture with a modern design."

[1544] This system allows users to quickly and effectively obtain interior design suggestions that reflect their own feelings and desires.

[1545] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1546] Step 1:

[1547] The user takes three photos of the room using the device's camera and enters their requirements.

[1548] Input: 3 photos of the room taken by the user, request text

[1549] How it works: The user inputs their preferences, such as "I want the room to look larger" or "I want a modern design," into the device's interface. Once the photo is taken, this data is collected and encrypted by the device.

[1550] Output: Encrypted photo data, request text

[1551] Step 2:

[1552] The terminal transmits the collected photo data and the request text to the server.

[1553] Input: Encrypted photo data, request text

[1554] How it works: The device sends this data to a server over the Internet, where it is encrypted and sent using a secure protocol.

[1555] Output: Photo data received by the server, request text

[1556] Step 3:

[1557] The server then uses image analysis algorithms to analyze the received photo data and determine the size of the room and the existing furniture layout.

[1558] Input: Photo data

[1559] How it works: The server uses image analysis software such as OpenCV and TensorFlow to process the data and determine the size of the room, the color of the walls, and the layout of the existing furniture.

[1560] Output: Data about the analyzed room size, existing furniture placement, and wall color

[1561] Step 4:

[1562] The server inputs the analyzed data and the desired text into a generative AI model to generate a list of suitable furniture and home appliances.

[1563] Input: Parsed data, requested text

[1564] Specific operation: The server uses a generative AI model (e.g., GPT-4, DALL-E) to create a prompt and input it into the model. Specifically, a prompt such as "Please suggest a furniture arrangement that will make a 30-square-meter room appear larger" is used.

[1565] Output: A list of the best furniture and appliances

[1566] Step 5:

[1567] The server uses an emotion engine to analyze the user's facial expressions and voice data and recognize the user's emotions.

[1568] Input: User's facial expression data, voice data

[1569] Specific operation: The server uses an emotion engine (e.g., Microsoft Azure Emotion API, Google Cloud Vision API) to analyze the user's facial expressions and voice to identify emotions such as joy, surprise, dissatisfaction, etc. For example, facial expression analysis is performed using face-api.js, and voice analysis is performed using Google Speech-to-Text API.

[1570] Output: Recognized emotion data

[1571] Step 6:

[1572] The server adjusts the list of furniture and home appliance candidates generated based on the emotion data.

[1573] Input: List of suitable furniture and appliances, recognized emotion data

[1574] Specific behavior: When a user shows a facial expression of joy, the candidate list is adjusted to prioritize suggestions that increase that joy.

[1575] Output: A tailored list of furniture and appliances

[1576] Step 7:

[1577] The server accesses the APIs of multiple sellers to obtain product information based on the adjusted candidate list.

[1578] Input: A tailored list of furniture and appliances

[1579] Specific operation: The server uses the API to obtain the necessary product information from multiple sellers, such as product name, size, price, and availability.

[1580] Output: Retrieved product information

[1581] Step 8:

[1582] The server generates a 3D simulation model based on the acquired product information.

[1583] Input: Retrieved product information

[1584] What it does: The server uses a 3D graphics library (e.g. Unity, Three.js) to generate a simulation of placing furniture in a virtual model of the room.

[1585] Output: 3D simulation model

[1586] Step 9:

[1587] The terminal displays the 3D simulation model and product information received from the server to the user.

[1588] Input: 3D simulation model, product information

[1589] How it works: The device displays this 3D simulation to the user through the interface of a smartphone or tablet. It is also possible to place virtual furniture in a real room using AR technology (e.g., ARKit, ARCore) and see how it will look.

[1590] Output: 3D simulation and product information displayed to the user

[1591] Step 10:

[1592] The user checks the proposed furniture and appliance arrangement, and if they like it, they press the "Purchase" button to add it to their cart. The terminal encrypts the purchase request and payment information and sends it to the server.

[1593] Input: User purchase request, payment information

[1594] Specific operation: When the user presses the purchase button, the terminal encrypts the entered payment information and sends it to the server.

[1595] Output: Purchase request sent to server, payment information

[1596] Step 11:

[1597] The server will check inventory and initiate delivery procedures via the seller's API, and once the order is confirmed, will notify the user's device of the transaction status.

[1598] Input: Purchase request, payment information

[1599] Specific operation: The server checks inventory and processes delivery, and once the order is confirmed, it notifies the user's terminal of the transaction status in real time.

[1600] Output: Transaction status reported to the user's terminal

[1601] This allows users to quickly and effectively receive interior design suggestions that reflect their feelings and desires, ensuring a consistent and smooth process leading up to the purchase.

[1602] (Application example 2)

[1603] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1604] Conventional interior design suggestion systems have had the problem of making it difficult for users to find the perfect interior design that matches their own emotions and preferences. In particular, when receiving suggestions in a physical store, it is not possible to adjust the suggestions in real time, making it difficult to provide suggestions that will satisfy the user.

[1605] The specification processing by the specification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes: means for a user to take a photo of a room and input their requirements; means for transmitting the photo data and the requirements to the server; means for analyzing the photo data and recognizing the size of the room and the layout of existing furniture; means for generating optimal furniture and home appliance candidates based on the recognition results and the requirements; means for analyzing the user's facial expressions and voice data using an emotion recognition engine and recognizing emotions; means for adjusting the proposal content based on the recognized emotions; means for acquiring product information from multiple retailers; means for generating a layout based on the acquired product information and displaying a 3D simulation to the user; and means for the user to check the 3D simulation and then proceed with the purchase of the product. This enables optimal interior design proposals to be made in real time according to the user's emotions and requirements.

[1606] "Means for users to take photos of a room and input their requirements" refers to equipment and an interface that allows users to take multiple photos of a room from different angles and input their desired interior style, furniture requirements, etc.

[1607] The "means for transmitting the photo data and request to the server" refers to a function and mechanism for transmitting the photograph taken and the request entered from the user terminal to the server via a communication line.

[1608] "Means for analyzing the photographic data and recognizing the size of the room and the existing furniture layout" refers to a device and program that uses an image analysis algorithm to identify the dimensions of a room and the current furniture layout from the photographic data taken.

[1609] The "means for generating optimal furniture and home appliance candidates based on the recognition results and user requests" refers to a generation algorithm and engine for selecting and listing appropriate furniture and home appliances based on the analyzed data and the user's requests.

[1610] "Means for recognizing emotions by analyzing a user's facial expressions and voice data using an emotion recognition engine" refers to a technology that uses emotion recognition software and machine learning models to analyze a user's facial expressions and voice to identify emotions such as joy, dissatisfaction, and surprise.

[1611] The "means for adjusting the suggestions based on the recognized emotion" refers to algorithms and programs for modifying and optimizing the furniture and appliance suggestions list depending on the user's emotional state.

[1612] "Means for obtaining product information from multiple retailers" refers to systems and programs that use APIs from various retailers to collect data such as product name, price, inventory, and size.

[1613] The "means for generating a layout based on the acquired product information and displaying a 3D simulation to the user" refers to a 3D graphics engine and user interface for creating a 3D layout using the acquired product information and visually displaying it to the user.

[1614] The "means for a user who has reviewed the 3D simulation to complete the purchase of a product" refers to an interface and process that allows a user to review the 3D simulation, select a product they like, and make a payment to purchase the product.

[1615] MODE FOR CARRYING OUT THE INVENTION

[1616] ...

Claims

1. A means for the user to take a photo of the room and input their requirements; means for transmitting the photo data and the request to a server; means for analyzing the photographic data and recognizing the size of the room and the existing furniture layout; A means for generating optimal furniture and home appliance candidates based on the recognition results and the customer's request; a means for obtaining product information from a plurality of sellers; a means for generating a layout based on the acquired product information and displaying a 3D simulation to a user; A means for a user who has checked the 3D simulation to purchase the product; A system including:

2. The system of claim 1 , wherein the 3D simulation is displayed using AR technology.

3. The system according to claim 1 , wherein the product information obtained from the plurality of retailers is not limited to a specific store.

Images