System

Abstract

A system is provided.SOLUTION: The system includes a generation model means for extracting a user's request in an interactive form, a generation model means for generating an interior proposal based on the extracted request, a means for transmitting the generated interior proposal to a user terminal, a means for displaying the generated interior proposal on the user terminal by using an AR function, and a cooperation means for on-line purchasing or selling the proposed interior item.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] When users purchase or move into a new home, it takes a lot of time and effort to coordinate the interior appropriately. It is particularly difficult to consistently do this, as many factors must be considered, including selecting furniture that suits the size and layout of the home, adjusting taste and color, purchasing within budget, and disposing of unnecessary furniture. Traditional methods require users to gather information from magazines and the Internet and make their own decisions, making it difficult to realize their ideal interior. [Means for solving the problem]

[0005] To address these challenges, the present invention provides a system that interactively extracts user requirements and uses a generative model to make interior design suggestions based on those requirements. This system includes a means for extracting detailed user requirements, such as floor plan, size, taste, color, and budget, using a generative AI model, and a means for generating interior design suggestions based on the extracted requirements. It also includes a means for sending the generated interior design suggestions to a user's device and displaying an image of the proposed items in an actual room using the device's AR function. The system also includes a linking means for purchasing the suggested interior items online and selling unwanted furniture, allowing users to consistently and efficiently coordinate their home.

[0006] "Generative model means" refers to an artificial intelligence (AI) model for generating specific proposals or designs based on user requests.

[0007] "Interior proposals" refer to furniture, decorations, and layout plans generated to reflect the user's requests.

[0008] "User terminal" refers to an electronic device such as a smartphone, tablet, or PC that a user uses to access the system.

[0009] "AR function" is a technology that stands for augmented reality, and refers to the function of overlaying virtual information on the real world through a camera.

[0010] "Online purchasing method" refers to the process or system for purchasing goods over the Internet.

[0011] "Online selling methods" refers to processes and systems for selling unwanted items over the Internet.

[0012] "User requests" refers to the specific wishes and needs that users have regarding interior design, such as layout, size, taste, color, and budget. [Brief explanation of the drawings]

[0013] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0014] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0015] First, the terms used in the following description will be explained.

[0016] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0017] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0018] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0019] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0020] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0021] [First embodiment]

[0022] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0023] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0024] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0025] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0026] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0027] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0028] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0029] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0030] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0031] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0032] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0033] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0034] This invention is a support system for interior coordination when users purchase or move into a new home. It combines generative AI models and AR technology to make optimal interior suggestions based on the user's requests, and also allows for easy online buying and selling.

[0035] The system begins with the user accessing the interface via a smartphone or computer. Once the user logs in to the system, an interactive questionnaire appears, and the user inputs the details of the room and the desired interior. For example, a specific request could be conveyed to the system, such as, "I want a living room that is 20 square meters in size, with a Scandinavian design, a budget of 100,000 yen, and a blue and white theme."

[0036] The server then uses the generative AI model to generate multiple interior design proposals based on the user's needs. The proposals are sent to the user's device, where the user can review them. For example, a blue and white Scandinavian-style sofa, a wooden low table, and a simple floor lamp are suggested.

[0037] The user can comment on the displayed suggestions, for example, by inputting a request for revision, such as "I'd like the sofa design to be more casual." The request for revision is then sent back to the server, and the generative AI model generates a new suggestion. This process is repeated to ensure the suggestion satisfies the user.

[0038] Once the proposal is finalized, the user's device activates the camera and uses AR to virtually place the proposed interior items in the actual room. Through the smartphone screen, the user can see how the proposed sofa or table will look in each position in the room.

[0039] After checking the specific layout image of the interior items, if the user wishes to purchase them, they select the purchase option on their device. The server receives the user's selection and works with the online shop to complete the purchase process. Similarly, if the user needs to dispose of unwanted furniture, they can select the sell option, and the server will work with a platform such as an auction site to complete the selling process.

[0040] As a concrete example, consider the case of coordinating the living room of a new home. The user inputs detailed requirements into the system, and the generative AI model based on these inputs suggests a blue and white Scandinavian-style sofa, a wooden low table, and a floor lamp with a simple design. The user then sends a correction request, such as "make the sofa more casual," and the generative AI model re-proposes a sofa with a casual design in response to the request. The user then uses their smartphone camera to check how the proposed furniture would look in their actual living room. If they like the furniture, they can complete the purchase process online and sell any unwanted furniture.

[0041] In this way, the system of the present invention allows the user to achieve an ideal interior coordination without much effort, and enables the process to be carried out simply and quickly.

[0042] The processing flow will be explained below.

[0043] Step 1:

[0044] The user accesses the coordination support system using a smartphone or computer. The user activates the dialogue mode and inputs, "Please coordinate my new living room."

[0045] Step 2:

[0046] The device receives the user's input and displays questions to input detailed requirements. The user inputs information such as layout, size, style, color, and budget. For example, the user might input "living room size 20 square meters, style Scandinavian style, budget 100,000 yen, color scheme blue and white."

[0047] Step 3:

[0048] The device temporarily stores the user's requests and sends them to the server, which then activates a generative AI model to generate multiple interior design suggestions based on the user's requests. For example, it might suggest a blue and white Scandinavian-style sofa, a wooden low table, and a simple floor lamp.

[0049] Step 4:

[0050] The server sends the generated interior design proposal to the terminal, which displays the proposal to the user. The user checks the proposal and inputs a request for modification, such as "make the sofa design more casual."

[0051] Step 5:

[0052] The device then sends the correction request back to the server, which then passes it on to the generative AI model, which then regenerates a new proposal—for example, a new interior design proposal that includes a sofa with a casual design.

[0053] Step 6:

[0054] The server sends the regenerated interior proposal to the device, which displays the re-proposed content to the user, who then reviews the proposal and makes a final decision.

[0055] Step 7:

[0056] The device will activate its camera and switch to AR mode. The user can then use the camera to view the room and virtually see how the proposed furniture will be arranged in the room. For example, they can see how a sofa or table will be arranged on their smartphone screen.

[0057] Step 8:

[0058] If the user likes the proposed furniture, they click the "Purchase" button on their device. The server connects to the online shop and completes the purchase process. Purchase confirmation and shipping information are sent to the user's device.

[0059] Step 9:

[0060] When a user wants to dispose of unwanted furniture, they click the "Sell" button on their device. The server connects to an online platform (e.g., an auction site) and completes the sale procedure. The sale information is then sent to the user's device.

[0061] Through this series of steps, users can efficiently coordinate their rooms and purchase or sell the most suitable interior items.

[0062] Example 1

[0063] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0064] Traditional interior coordination requires a lot of time and effort, and users have to go through a lot of trial and error to achieve their ideal interior. In particular, there has been a lack of a system for receiving efficient and optimal interior suggestions when purchasing a new home or moving. In addition, it is difficult to visualize how the suggested items will be arranged in the actual room, and the purchasing and selling procedures are complicated.

[0065] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0066] In this invention, the server includes generative model means for interactively extracting user requirements, generative model means for generating interior proposals based on the extracted requirements, means for transmitting the generated interior proposals to a user terminal, means for the user terminal to display the generated interior proposals using an augmented reality function, and linking means for purchasing or selling the proposed interior items online. This allows users to easily communicate their detailed requirements, receive optimal interior proposals in a short amount of time, and visually confirm the placement of the proposed items, allowing them to carry out purchasing or selling procedures in a consistent manner.

[0067] The "generative model means" is a function that uses a generative AI model to analyze user requests and generate interior design proposals based on those requests.

[0068] A "user terminal" is a device, such as a smartphone or PC, that a user uses to access the interface.

[0069] The "interactive format" is a method in which a user inputs a request into the system in natural language, and the system then presents a series of questions in response to the request, thereby extracting the user's specific requirements and conditions.

[0070] "Augmented reality" is a technology that overlays digital information on the real world, specifically a function that displays virtual interior items through the camera of a smartphone or tablet.

[0071] The "linking means" is a system that exchanges data with external online shops or auction platforms in order to purchase or sell the proposed interior items online.

[0072] The "requirement extraction means" is a function that analyzes the data input by the user and identifies detailed requirements such as floor plan, size, design style, preferred color, and budget.

[0073] "Regeneration" is the process of accepting user requests for modifications and again using the generative AI model to generate new interior proposals.

[0074] The interior coordination support system of the present invention provides a specific means for a user to receive interior design suggestions when purchasing or moving into a new home. The present invention can be implemented using the system configuration and operation procedure described below.

[0075] The system's program runs on a cloud server and combines generative AI models and augmented reality technology. The main hardware and software used are as follows:

[0076] User devices: Smartphones (iPhone (registered trademark), ANDROID (registered trademark)), PCs (Windows, macOS (registered trademark))

[0077] Server: Cloud server (AWS (registered trademark), Google (registered trademark) Cloud)

[0078] Generative AI models: GPT-4 (registered trademark), DALL-E

[0079] Augmented reality technology: ARKit (iOS), ARCore (Android)

[0080] Online shop integration: EC site API (Amazon, Rakuten)

[0081] Selling platform integration: Auction site API (eBay, Mercari)

[0082] First, the user accesses the system's web interface using a smartphone or PC. After logging in, an interactive interface for inputting requests appears, where the user inputs the room's conditions and desired interior details.

[0083] For example, a user might input a specific request such as, "I would like the living room to be 20 square meters in size, with a Scandinavian design, a budget of 100,000 yen, and a blue and white theme." This input data is sent to the server, which then generates and sends a prompt to the generative AI model.

[0084] An example of this prompt is:

[0085] A user wants to decorate their living room. Please provide suggestions based on the following criteria:

[0086] Room size: 20 square meters

[0087] Design Desired: Scandinavian style

[0088] Budget: 100,000 yen

[0089] Color: Blue and White

[0090] We are looking for suggestions for specific interior items (sofa, table, lamp, etc.).

[0091] Based on this prompt, a generative AI model (e.g., GPT-4) generates multiple interior design proposals that match the user's requirements. The generated proposals are sent from the server to the user's device, where the user can confirm the proposals. For example, a blue and white Scandinavian-style sofa, a wooden low table, and a simple floor lamp.

[0092] If the user is not satisfied with the proposal, they can input a request for revision, such as a more specific request like "I want the sofa design to be more casual." This request for revision is sent again to the server, and the generative AI model generates a new proposal, which is again sent to the user's device. This process is repeated until the user is satisfied.

[0093] Finally, when the user is satisfied with the proposed items, the camera on the user's device is activated and the proposed interior items are virtually placed in the actual room using augmented reality.The user can check on their smartphone screen how the proposed sofa or table will look in each position in the room.

[0094] After checking the specific layout image of the interior items, if the user wishes to purchase them, they select a purchase option on their device. The server receives the user's selection and works with the online shop to complete the purchase process. Similarly, if the user needs to dispose of unwanted furniture, they can select the sell option, and the server will work with a platform such as an auction site to complete the selling process.

[0095] In this way, the system of the present invention allows the user to achieve an ideal interior coordination without much effort, and enables the process to be carried out simply and quickly.

[0096] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0097] Step 1:

[0098] A user accesses the system interface using a browser on a smartphone or PC. The user's input is to enter the system's URL and access the interface through the browser. The output of this step is that the interface is successfully accessed and the login screen is displayed.

[0099] Specific behavior:

[0100] The user launches a browser and enters the specified URL.

[0101] The browser displays the system interface.

[0102] Step 2:

[0103] The user logs in to the system. After logging in, an interactive questionnaire is displayed, and the user inputs details of the room situation and desired interior. The input data includes requirements such as room size, design preference, budget, and color. The output of this step is that the user's requirements are sent to the server.

[0104] Specific behavior:

[0105] The user enters their login information and logs into the system.

[0106] Answer the interactive questions that appear after logging in.

[0107] For example, enter "living room 20 square meters, Scandinavian style, budget 100,000 yen, blue and white as the main colors."

[0108] Step 3:

[0109] The server receives the request data sent by the user and generates and sends a prompt to the generative AI model. The input data includes the user's request. The generative AI model generates an interior proposal based on this prompt. The output of this step is the generated interior proposal.

[0110] Specific behavior:

[0111] The server receives the user's request.

[0112] The server sends the prompt to the generative AI model.

[0113] Example prompt sentence:

[0114] A user wants to decorate their living room. Please provide suggestions based on the following criteria:

[0115] Room size: 20 square meters

[0116] Design Desired: Scandinavian style

[0117] Budget: 100,000 yen

[0118] Color: Blue and White

[0119] We are looking for suggestions for specific interior items (sofa, table, lamp, etc.).

[0120] Step 4:

[0121] A generative AI model (e.g., GPT-4) generates interior design suggestions based on a prompt. The input data includes the prompt, and the output is an interior design suggestion (e.g., "A blue and white Scandinavian-style sofa, a wooden low table, and a simple floor lamp"). The output of this step is the interior design suggestion.

[0122] Specific behavior:

[0123] The generative AI model receives the prompt and generates interior design suggestions that meet the user's needs.

[0124] The generated interior proposal is sent to the server.

[0125] Step 5:

[0126] The server receives the generated interior proposal and sends it to the user terminal. The input data is the generated interior proposal and the output is the transmission to the user terminal. The output of this step is the display of the interior proposal on the user terminal.

[0127] Specific behavior:

[0128] The server receives the generated interior proposals.

[0129] The server transmits the interior proposal to the user terminal.

[0130] Step 6:

[0131] The user reviews the submitted interior proposal and enters revision requests if necessary. The input data includes the user's revision requests. These requests are sent to the server, where the generative AI model generates a new proposal. The output of this step is a new proposal based on the revision requests.

[0132] Specific behavior:

[0133] The user confirms the proposal.

[0134] Enter any correction requests you may have for the proposal (e.g., "Make the sofa design more casual").

[0135] A modification request is sent to the server.

[0136] Step 7:

[0137] The server receives the user's revision request and re-prompts the generative AI model to generate a new interior proposal. The input data includes the user's revision request. The output of this step is the newly generated interior proposal.

[0138] Specific behavior:

[0139] The server sends a modification request to the generative AI model.

[0140] A generative AI model generates new interior proposals.

[0141] The new proposal is sent to the server.

[0142] Step 8:

[0143] Once the user is satisfied with the proposal, the user device activates the camera and uses augmented reality to virtually place the proposed interior items in the actual room. The input data includes the generated interior proposal. The output of this step is for the user to confirm the virtual placement results.

[0144] Specific behavior:

[0145] The user activates the smartphone camera.

[0146] Use augmented reality to virtually place suggested items in a room.

[0147] Check the placement on your smartphone screen.

[0148] Step 9:

[0149] If the user wishes to purchase the proposed item, he / she selects the purchase option on the user's device. The server receives this information and cooperates with the online shop to complete the purchase process. The input data includes the user's intention to purchase. The output of this step is the completion of the purchase process.

[0150] Specific behavior:

[0151] The user presses the purchase button.

[0152] The server connects with the online shop and completes the purchase process.

[0153] Step 10:

[0154] If the user needs to dispose of unwanted furniture, they select the selling option. The server receives this information and cooperates with platforms such as auction sites to complete the selling process. The input data includes the user's intention to sell. The output of this step is the completion of the selling process.

[0155] Specific behavior:

[0156] The user presses the sell button.

[0157] The server connects with the auction site and completes the sales process.

[0158] As described above, in the present invention, each process is performed sequentially, thereby supporting the user in achieving an ideal interior coordination without much effort.

[0159] (Application example 1)

[0160] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0161] Conventional interior coordination systems have had problems such as difficulty in efficiently providing proposals that reflect the user's needs and limited means for users to see how the proposals would actually look in a room, resulting in low user satisfaction. Another problem is that the procedures for purchasing or selling the proposed interior items online are cumbersome. Furthermore, there has been no system that allows users to view the generated interior proposals in a virtual space and then smoothly proceed to purchase them. The objective of the present invention is to solve these problems.

[0162] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0163] In this invention, the server includes a generative modeling unit for interactively extracting user requirements, a generative modeling unit for generating interior design proposals based on the extracted requirements, a unit for transmitting the generated interior design proposals to a user terminal, a unit for displaying the generated interior design proposals on the user terminal using an AR function, a linking unit for purchasing or selling the proposed interior design items online, and a unit for viewing the generated interior design proposals in a virtual space and proceeding directly to a purchase. This allows users to quickly and accurately receive interior design proposals based on their detailed requirements, virtually arrange and view the proposals in an actual room, and smoothly complete the online purchase or sale process. Additionally, the ability to view the generated interior design proposals in a virtual space and proceed directly to a purchase significantly improves user satisfaction and convenience.

[0164] The "generative model means for interactively extracting user needs" is a device that includes artificial intelligence for analyzing information input by a user interactively and extracting those needs.

[0165] The "generative model means for generating interior proposals" is a device that includes artificial intelligence for generating optimal interior proposals based on extracted user requests.

[0166] The "means for transmitting the generated interior proposal to the user terminal" is a communication means for transmitting the interior proposal generated by the server to the user's device.

[0167] "Means for displaying interior proposals generated by a user's device using AR functionality" refers to technology for displaying interior proposals generated on a user's device using augmented reality functionality.

[0168] The "connection means for purchasing or selling the proposed interior items online" refers to a means for connecting with an online platform to simplify the process of purchasing or selling interior items online.

[0169] "A means to check the generated interior design proposal in a virtual space and proceed directly to purchase" refers to technology that allows the generated interior design proposal to be checked in real time in a virtual environment and the purchase process to be completed online.

[0170] The system for realizing this invention provides a series of steps to extract user needs interactively, generate interior design proposals based on those needs, confirm those proposals in a virtual space, and purchase or sell the most suitable interior items online. Each element will be explained in detail below.

[0171] The server is equipped with a generative AI model and an interactive question-and-answer algorithm. In particular, the generative AI model analyzes and extracts the user's requests and generates interior proposals based on them. The generated interior proposals are sent to the user's device, where the user can review them. In addition, if the user requests revisions to the proposal, the generative AI model will generate a new proposal again.

[0172] The user device may be a smartphone or tablet. The user device receives the interior design suggestions sent from the server and uses AR (Augmented Reality) functionality to virtually display them in the actual room environment. This allows the user to see in real time how the proposed interior items will look. AR technology such as ARKit for iOS or ARCore for Android is used.

[0173] Users can purchase items they like online or sell unwanted items. The server works with online shops and auction sites to facilitate these procedures.

[0174] As a concrete example, consider a user who wants to decorate the living room of their new home. The user accesses the system and inputs detailed requirements through the following interface:

[0175] Room details:

[0176] Room type: Living room

[0177] Area: 20 square meters

[0178] Design Desire:

[0179] Style: Nordic style

[0180] Color: Blue and White

[0181] Budget: 100,000 yen

[0182] Based on this request, the server uses a generative AI model to generate suggestions such as a blue and white Scandinavian sofa, a wooden table, and a simple floor lamp, and sends them to the user's device. The user can then virtually place these suggestions in their actual room using the AR function to see how the items will look.

[0183] An example prompt might be used as follows:

[0184] Please suggest interior coordination based on the user's request: The living room is 20 square meters, and the design is Scandinavian-style, with a budget of 100,000 yen. The main colors are blue and white.

[0185] In this way, the system of the present invention allows the user to achieve ideal interior coordination without much effort, and the process can be carried out simply and quickly.

[0186] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0187] Step 1: User request input

[0188] Users access the interface via a smartphone or tablet and enter detailed requirements such as the type of room, layout, size, desired design, preferred colors, budget, etc. Specific input data might include "living room, 20 square meters, Scandinavian design, blue and white, 100,000 yen."

[0189] Step 2: Extracting desired data

[0190] The server receives the input data sent by the user and uses a generative AI model to analyze and extract the user's requirements interactively. During this process, the user's input data is analyzed and processed to extract specific keywords (room type, size, desired design, etc.). The extracted data is then passed on to the next step.

[0191] Step 3: Generate interior proposals

[0192] The server uses a generative AI model to generate interior design proposals based on the extracted request data. A prompt statement is input into the generative model, and multiple interior design proposals are generated based on that. For example, a "blue and white Scandinavian-style sofa, wooden table, and simple floor lamp" is generated. This proposal data is passed on to the next step.

[0193] Step 4: Submit your interior design proposal

[0194] The generated interior design proposals are sent from the server to the user's device. Here, the proposal data (e.g., "a blue and white Scandinavian-style sofa, a wooden table, and a simple floor lamp") is sent to the user's smartphone or tablet. The user can then check the proposals on the screen.

[0195] Step 5: User correction request

[0196] The user can input their opinions and requests for modifications to the displayed interior design proposal. For example, they can send a specific request for modifications such as "I want the sofa design to be more casual." The user's modification request data is passed to the next step.

[0197] Step 6: Regenerate correction suggestions

[0198] The server receives the modification request and uses the generative AI model to regenerate a new interior proposal. In this process, a new prompt sentence is generated, and a modified interior proposal is generated based on it. For example, "a sofa with a casual design." This modified proposal data is passed to the next step.

[0199] Step 7: Final confirmation using AR display

[0200] The user device receives the regenerated interior proposals and virtually places them in the actual room using the smartphone's camera and AR function. The user can check in real time on the device screen how the interior items will look in each position. AR technology (such as ARKit for iOS or ARCore for Android) is used here.

[0201] Step 8: Buy or sell online

[0202] Finally, if the user is satisfied with the suggestions, they can choose to purchase online or sell the unwanted items. The server automatically processes the purchase and sale procedures by connecting with online shops and auction sites. Specifically, when the user presses the purchase or sale button, information about the suggested interior items is sent to the connected platform, completing the process.

[0203] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0204] This invention is a system that recognizes users' emotions and makes optimal interior design suggestions based on that information. The system combines generative AI models, AR technology, and an emotion engine to provide sophisticated interior design suggestions and actual layout images based on user requests, and also supports online buying and selling.

[0205] The usage flow of this system is as follows: First, the user accesses the coordination support system using a smartphone or PC. The user starts the dialogue mode and enters, "I'd like you to coordinate my new living room."

[0206] Next, the user device displays an interface for entering detailed requirements. The user enters information such as floor plan, size, style, color, and budget. For example, a specific requirement might be entered as "living room size 20 square meters, style Scandinavian, budget 100,000 yen, color scheme blue and white."

[0207] In addition, the emotion engine analyzes the user's emotions from their voice and facial expressions. When a user inputs a request, facial and voice data is captured through the camera and microphone on the user's device, and the emotion engine analyzes this. After the emotion data is analyzed, it is sent to the server, and a request is generated that takes the user's emotions into account.

[0208] The server then receives this request data and uses a generative AI model to generate interior design suggestions, such as a blue and white Scandinavian-style sofa, a wooden low table, and a simple floor lamp.

[0209] The generated suggestions are sent to the device and displayed to the user. The user reviews the suggestions and may input requests for revisions, such as "make the sofa design more casual." If a revision is requested, the device sends it back to the server, and the generative AI model generates a new suggestion. The server then refers to the user's emotional data again to make the optimal suggestion.

[0210] The process is repeated to ensure the proposal satisfies the user. Once the proposal is finalized, the user's device activates the camera and switches to AR mode. The user then views the room through their smartphone and sees how the proposed furniture will actually look when arranged. The AR function displays a virtual image of the interior items, allowing the user to view the entire room.

[0211] If the user checks the final proposal and likes the item, they select the purchase option on their device. The server connects to the online shop and completes the purchase procedure for the selected item. Purchase confirmation and shipping information are sent to the user's device. Furthermore, if the user selects the sell option for unwanted furniture, the server will link with an online platform such as an auction site and complete the selling procedure.

[0212] As a concrete example, consider the case of coordinating the living room of a new home. The user inputs detailed requirements into the system, and the emotion engine detects that the user is relaxed. Based on this information, the generative AI model suggests a "Nordic-style blue and white sofa with an emphasis on relaxation, a wooden low table, and a floor lamp with a simple design." The user submits a request for revisions, and the emotion engine confirms that the user is satisfied with the changes. Once the proposal is finalized, the user can visually check the actual layout using AR functionality, and finally purchase the items online. Unwanted old furniture can also be sold in the same way.

[0213] In this way, the system of the present invention takes into consideration the user's feelings and efficiently realizes the ideal interior coordination, and also supports simple and quick purchase and sale procedures.

[0214] The processing flow will be explained below.

[0215] Step 1:

[0216] The user accesses the coordination support system using a smartphone or computer. The user activates the dialogue mode and inputs, "Please coordinate my new living room."

[0217] Step 2:

[0218] The device receives the user's input and displays questions for entering detailed requirements. The user enters information such as floor plan, size, style, color, and budget. For example, the user might enter, "Living room size: 20 square meters, style: Scandinavian, budget: 100,000 yen, color: blue and white."

[0219] Step 3:

[0220] The device's camera and microphone are used to input the user's facial expressions and voice into the emotion engine, which then analyzes this data to determine the user's emotions, such as whether they are relaxed or excited. The analysis results are temporarily stored on the device.

[0221] Step 4:

[0222] The device sends the user's desires and emotional data to the server, which then activates a generative AI model to generate multiple interior design suggestions based on the user's desires and emotions. For example, the server suggests a "blue and white Scandinavian-style sofa, a wooden low table, and a simple floor lamp, all of which emphasize relaxation."

[0223] Step 5:

[0224] The server sends the generated interior design proposal to the terminal, which displays the proposal to the user. The user checks the proposal and inputs a request for modification, such as "make the sofa design more casual."

[0225] Step 6:

[0226] The device then sends the correction request to the server again. The server then communicates the correction request to the generative AI model, which then regenerates a new proposal. The generative AI model then refers to the user's emotional data again to make the optimal proposal.

[0227] Step 7:

[0228] The server sends the regenerated interior proposal to the device, which displays the re-proposed content to the user, who then reviews the proposal and makes a final decision.

[0229] Step 8:

[0230] The device will activate its camera and switch to AR mode. The user can then use the camera to view the room and virtually see how the proposed furniture will be arranged in the room. For example, they can see how a sofa or table will be arranged on their smartphone screen.

[0231] Step 9:

[0232] If the user likes the proposed furniture, they click the "Purchase" button on their device. The server connects to the online shop and completes the purchase process. Purchase confirmation and shipping information are sent to the user's device.

[0233] Step 10:

[0234] When a user wants to dispose of unwanted furniture, they click the "Sell" button on their device. The server connects to an online platform (e.g., an auction site) and completes the sale procedure. The sale information is then sent to the user's device.

[0235] Through this series of steps, users can interactively communicate their needs to the system, taking their own feelings into consideration, and efficiently receive interior design suggestions while purchasing the most suitable items or selling unwanted items.

[0236] Example 2

[0237] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0238] Modern interior coordination requires a great deal of time and effort to create a space that matches the user's emotions and preferences. It is also difficult for users to accurately convey their emotionally-based latent needs when expressing specific requests. The process of visualizing and revising proposals, as well as online purchasing and selling, can be complex and difficult to understand. To solve these problems, a comprehensive system is needed that considers the user's emotions, efficiently proposes interiors, and ultimately creates a comfortable and ideal space.

[0239] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes a data generation means for interactively extracting user desires, a generative model means for generating interior design proposals based on the extracted desires, a sentiment analysis means for analyzing the user's emotions, a means for the generative model to generate proposals taking the sentiment analysis data into consideration, a means for transmitting the generated interior design proposals to a user terminal, a means for the user terminal to display the generated interior design proposals using an augmented reality function, and a linking means for purchasing or selling the proposed interior design items online. This enables highly accurate interior design proposals that take the user's emotions into consideration, and by visually confirming and correcting them, a highly satisfying coordination can be quickly and efficiently achieved.

[0240] The "data generating means" has a function for extracting user requests in an interactive format.

[0241] The "generative model means" refers to a model used to generate interior proposals based on the extracted needs.

[0242] The "emotion analysis means" has a function for analyzing the user's emotions from facial expressions, voice, and the like.

[0243] The "proposal generation means" has the function of creating optimal interior proposals using a generative model while taking into account emotion analysis data.

[0244] The "data transmission means" has a function for transmitting the generated interior proposal to the user terminal.

[0245] The "augmented reality function" is a function that allows interior design proposals generated on a user's device to be virtually displayed in real space.

[0246] "Linkage means" means a function that links with other systems or platforms to purchase or sell the proposed interior items online.

[0247] The present invention is a system that recognizes a user's emotions and makes optimal interior design suggestions based on that information. This system is realized by combining a generative AI model, AR technology, and an emotion analysis engine, and provides advanced interior design suggestions and actual layout images based on the user's requests, and also supports online purchasing and selling. Specific embodiments of the present invention are described in detail below.

[0248] First, the user accesses the coordination support system using a device such as a smartphone or PC. The user activates the dialogue mode and inputs, "I'd like you to coordinate my new living room." This causes the user device to display an interface for inputting detailed requests. The user inputs information such as floor plan, size, style, color, and budget. For example, the user might input specific requests such as, "Living room size: 20 square meters, style: Scandinavian, budget: 100,000 yen, color: blue and white."

[0249] Next, the user device collects the user's facial expressions and tone of voice through a camera and microphone, which are then analyzed by an emotion analysis engine. The emotion analysis engine determines the user's emotional state, such as whether they are relaxed or excited. For example, if the engine detects that the user is relaxed when entering their request, that information is collected.

[0250] The device then sends the user's detailed information and analyzed emotion data to a server. The server then uses a generative AI model (e.g., a general generative model) to generate optimal interior design suggestions based on the user's desires and emotions. Specifically, the server generates suggestions such as a blue and white Scandinavian-style sofa, a wooden low table, and a simple floor lamp.

[0251] The generated suggestions are sent to the device and displayed to the user. The user reviews the suggestions and may input requests for revisions, such as "make the sofa design more casual." If a revision is requested, the device sends it back to the server, and the generative AI model generates a new suggestion. The server then refers to the user's emotional data again to make the optimal suggestion.

[0252] The process is repeated to ensure the proposal is satisfactory to the user. Once the proposal is finalized, the user's device activates the camera and switches to AR mode. The user then uses the device to view the entire room and see the virtual arrangement of the proposed furniture. The AR function displays a virtual image of the interior items arranged, allowing the user to view the entire room.

[0253] If the user checks the final proposal and likes the item, they can select the purchase option on their device. The server connects to the online shop and completes the purchase process for the selected item. Purchase confirmation and shipping information are sent to the user's device. Additionally, if the user selects the sell option for unwanted furniture, the server can link with an online platform such as an auction site to complete the selling process.

[0254] Examples of prompts include the following:

[0255] "I'd like the living room to be 20 square meters in a Scandinavian style. The color scheme should be blue and white, and the budget should be within 100,000 yen. I'd like it to have a relaxing feel."

[0256] "Please make the sofa design a little more casual."

[0257] In this way, the system of the present invention efficiently provides ideal interior coordination while taking into consideration the user's feelings, and also allows for simple and quick purchase and sale procedures.

[0258] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0259] Step 1:

[0260] A user accesses the system using a terminal. The user activates the interactive mode and inputs, "I'd like you to coordinate a new living room." This causes the terminal to display an interface for inputting detailed requests.

[0261] Input: User access requests and command input (e.g., "Please help me coordinate my new living room").

[0262] Output: Display of the coordination support interface.

[0263] Specific operation: When a user opens the system's web page or app on a smartphone or PC, the device displays the login screen and the user enters the login information. The user activates the interactive mode, and the system displays the request input screen.

[0264] Step 2:

[0265] The user inputs details of their coordination needs, and the user's device receives and records information such as floor plan, size, taste, color, and budget through a form on the interface.

[0266] Input: User-entered data (e.g., "Living room is 20 sq m, style is Scandinavian, budget is 100,000 yen, colors are blue and white").

[0267] Output: Detailed coordination request data.

[0268] Specific operation: The user enters their request into the text box on the interface, and the terminal receives it in real time and displays a confirmation message.

[0269] Step 3:

[0270] The user's device collects the user's facial expressions and voice through a camera and microphone, which are then analyzed by an emotion analysis engine.

[0271] Input: User's facial expression and voice data.

[0272] Output: Parsed emotion data (e.g. "relaxed").

[0273] How it works: The device's camera captures the user's face and microphone records their voice. The emotion analysis engine analyzes this in real time to determine whether the user is relaxed, etc.

[0274] Step 4:

[0275] The device sends the user's detailed information and emotional data to the server, which then uses a generative AI model to generate interior design proposals based on the user's requests and emotional data.

[0276] Input: User details and parsed emotion data.

[0277] Output: Initial interior design proposal (e.g., "A blue and white Scandinavian-style sofa, a wooden low table, and a simple floor lamp").

[0278] How it works: When the device sends data to the server, the server uses a generative AI model to generate interior design suggestions, such as a "Scandinavian-style sofa, a wooden low table, and a simple floor lamp."

[0279] Step 5:

[0280] The server sends the generated interior design proposal to the user's device, which displays the proposal and accepts user requests for revisions.

[0281] Input: Interior design proposal sent from the server.

[0282] Output: Interior proposals and modification requests displayed on the user's screen.

[0283] Specific operation: The server sends the generated proposal to the device, and the device displays "a blue and white Scandinavian-style sofa, a wooden low table, and a simple floor lamp." The user then inputs a modification request, such as "Make the sofa design more casual."

[0284] Step 6:

[0285] The user device then sends a correction request to the server again, and the generative AI model generates a new proposal. The server also references the user's emotional data again to make the optimal proposal.

[0286] Input: User modification request data.

[0287] Output: Revised interior proposal.

[0288] Specific operation: When a revision request is sent to the server, the generative AI model again generates interior design suggestions and creates revision suggestions, such as a new sofa design, which are then sent back to the device.

[0289] Step 7:

[0290] Once the proposal is finalized, the user's device activates the camera and switches to augmented reality mode, allowing the user to view the entire room using their smartphone and see how the proposed furniture will be virtually arranged.

[0291] Input: Confirmed interior proposal.

[0292] Output: Virtual interior layout display in augmented reality mode.

[0293] Specific operation: The user's device activates the camera and uses the AR function to display a screen showing a virtual room with furniture arranged in it. The user can then view the image of the entire room, including a blue and white Scandinavian-style sofa, through their smartphone.

[0294] Step 8:

[0295] If the user likes the suggestions and selects a purchase option, the server connects to the online shop and completes the purchase process for the selected item. Additionally, if the user selects the option to sell unwanted furniture, the server will connect to an online platform such as an auction site and complete the sales process.

[0296] Input: User's buy and sell selections.

[0297] Output: Purchase confirmation, shipping information, sale process.

[0298] Specific operation: When the user clicks the purchase button on the device, the selected item is ordered in cooperation with the online shop. The server sends a purchase confirmation and shipping information to the device. If the user selects the sell option, the server cooperates with the auction site to assist in the sale process.

[0299] (Application example 2)

[0300] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0301] Conventional interior design suggestion systems are unable to consider user emotions, making it difficult to increase user satisfaction. Additionally, interior design suggestions in physical stores lack real-time feedback functions to improve the user experience.

[0302] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0303] In this invention, the server includes a generative model means for interactively extracting user desires, a generative model means for generating interior design proposals based on the extracted desires, a means for transmitting the generated interior design proposals to a user terminal, a means for the user terminal to display the generated interior design proposals using an AR function, a linking means for purchasing or selling the proposed interior design items online, an emotion analysis means for recognizing the emotions of customers, and a generative model means for generating interior design proposals based on the emotion analysis. This makes it possible to make interior design proposals that take user emotions into consideration, and to make proposals that meet user needs in real time even in physical stores.

[0304] A "generative model" is an algorithm or process for generating interior design suggestions based on user input data and emotion data.

[0305] "Emotion analysis means" refers to technology or a system that analyzes a user's facial expressions and voice and recognizes their emotional state.

[0306] A "user terminal" is a hardware device that a user uses to access the system and receive interior design suggestions, and specifically includes a smartphone, tablet, smart glasses, etc.

[0307] "AR function" refers to the ability to display virtual interior items in real space using augmented reality technology.

[0308] "Online purchasing or selling collaboration" refers to a system or service that supports the process of a user purchasing or selling suggested interior items online.

[0309] This invention is a system that recognizes a user's emotions and proposes optimal interior design based on that information. Specifically, it is realized by combining emotion analysis means, generative model means, AR functionality, and linking means for online purchases or sales.

[0310] First, when a user visits a physical store, the emotion analysis means recognizes and analyzes the user's facial expressions and voice in real time to obtain emotional data. Emotion analysis uses a general emotion recognition algorithm, such as facial recognition technology or voice analysis technology.

[0311] Next, the generative model means interactively extracts the user's requirements. Through the interactive interface, the user inputs detailed information such as the layout, size, taste, preferred color, and budget. In addition, emotion data obtained by the emotion analysis means is also passed to the generative model means.

[0312] The generative modeling method generates interior design suggestions based on the user's input data and emotional data. For example, it might suggest a "relaxing Scandinavian-style blue and white sofa, a wooden low table, and a simple floor lamp." The generative AI model is used to design an interior design that best suits the user's needs.

[0313] The generated interior design suggestions are sent to the user's device, such as a tablet or smart glasses. The user can then use the device's AR functionality to virtually place and visualize the generated interior design suggestions in their actual space. Through this visualization, the user can check how well the suggested items fit into their own space.

[0314] If the user is satisfied with the proposal, they can proceed to the online purchase procedure on the spot. The purchase request sent from the terminal is linked to the online shop, where the final purchase procedure is carried out. In addition, for unwanted interior items, the user can select the option to sell them, and the selling procedure will be carried out in conjunction with the online platform.

[0315] As a concrete example, consider a user requesting coordination of the living room of their new home. In this case, the user inputs detailed requests into the system, and the emotion analysis means detects that the user is relaxed. Based on this information, the generative AI model suggests a "Nordic-style blue and white sofa that emphasizes relaxation, a wooden low table, and a floor lamp with a simple design." The user then submits a request for revisions, and the emotion analysis means confirms that the user is satisfied with the changes. Once the proposal is finalized, the user can visually check the actual layout using the AR function, and finally purchase the items online. Unwanted old furniture can also be sold in the same way.

[0316] Example prompt sentence:

[0317] "Our customers want to relax. Please suggest a Scandinavian-style blue and white sofa for the living room, which will create a relaxing atmosphere."

[0318] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0319] Step 1:

[0320] When a user visits a store, the device (smart glasses or tablet) captures their facial expressions and voice. The input data is collected in real time through a camera and microphone and sent to the emotion analysis means. The emotion analysis means uses facial expression recognition algorithms and voice analysis algorithms to analyze and identify the user's emotions as data. The output is the user's emotional state (e.g., "relaxed").

[0321] Step 2:

[0322] Next, the user inputs detailed information about the room for which they would like interior suggestions (layout, size, taste, preferred colors, budget, etc.) through their device. The data the user inputs into the dialogue interface on the tablet or smart glasses is sent to the generative model means. The input data is passed to the generative AI model, which analyzes the user's request and constructs a prompt sentence. The output is a specific prompt sentence that reflects the user's wishes.

[0323] Step 3:

[0324] The generative model means combines the emotion data obtained in step 1 with the user desire data obtained in step 2. The generative AI model generates interior design proposals based on this data. Using data mining and pattern recognition techniques, it designs an interior design plan that best suits the user's desires and emotions. The output is a specific interior design proposal (for example, "A relaxing Scandinavian-style blue and white sofa, a wooden low table, and a simply designed floor lamp").

[0325] Step 4:

[0326] The generated interior proposal is sent from the server to the device. The user can check the proposal on the device's display. At this point, the user can request revisions to the proposal. If revisions are requested, the proposal is sent again from the device to the server, and a new proposal is generated by the generative model means.

[0327] Step 5:

[0328] After the final proposal is finalized, the user's device uses its AR function to display a visual image of the interior items virtually arranged in the real store. The user uses the device's camera to capture the real store space and overlay the virtual interior on it. The output is an image of the virtual interior arrangement in the real space.

[0329] Step 6:

[0330] If the user is satisfied with the proposed interior items, the terminal supports the online purchase process. When the user presses the purchase button, a purchase request is sent to the online shop via the server. After the purchase process is completed, shipping information is sent to the user terminal. The output is a purchase confirmation and shipping information.

[0331] Step 7:

[0332] If a user wants to sell unwanted interior items, the user device supports the selling procedure. When the user selects the selling option, the server will connect to an online platform such as an auction site to complete the selling procedure. The output is a confirmation of the sale and detailed information.

[0333] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0334] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0335] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0336] [Second embodiment]

[0337] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0338] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0339] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0340] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0341] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0342] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0343] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0344] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0345] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0346] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0347] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0348] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0349] This invention is a support system for interior coordination when users purchase or move into a new home. It combines generative AI models and AR technology to make optimal interior suggestions based on the user's requests, and also allows for easy online buying and selling.

[0350] The system begins with the user accessing the interface via a smartphone or computer. Once the user logs in to the system, an interactive questionnaire appears, and the user inputs the details of the room and the desired interior. For example, a specific request could be conveyed to the system, such as, "I want a living room that is 20 square meters in size, with a Scandinavian design, a budget of 100,000 yen, and a blue and white theme."

[0351] The server then uses the generative AI model to generate multiple interior design proposals based on the user's needs. The proposals are sent to the user's device, where the user can review them. For example, a blue and white Scandinavian-style sofa, a wooden low table, and a simple floor lamp are suggested.

[0352] The user can comment on the displayed suggestions, for example, by inputting a request for revision, such as "I'd like the sofa design to be more casual." The request for revision is then sent back to the server, and the generative AI model generates a new suggestion. This process is repeated to ensure the suggestion satisfies the user.

[0353] Once the proposal is finalized, the user's device activates the camera and uses AR to virtually place the proposed interior items in the actual room. Through the smartphone screen, the user can see how the proposed sofa or table will look in each position in the room.

[0354] After checking the specific layout image of the interior items, if the user wishes to purchase them, they select the purchase option on their device. The server receives the user's selection and works with the online shop to complete the purchase process. Similarly, if the user needs to dispose of unwanted furniture, they can select the sell option, and the server will work with a platform such as an auction site to complete the selling process.

[0355] As a concrete example, consider the case of coordinating the living room of a new home. The user inputs detailed requirements into the system, and the generative AI model based on these inputs suggests a blue and white Scandinavian-style sofa, a wooden low table, and a floor lamp with a simple design. The user then sends a correction request, such as "make the sofa more casual," and the generative AI model re-proposes a sofa with a casual design in response to the request. The user then uses their smartphone camera to check how the proposed furniture would look in their actual living room. If they like the furniture, they can complete the purchase process online and sell any unwanted furniture.

[0356] In this way, the system of the present invention allows the user to achieve an ideal interior coordination without much effort, and enables the process to be carried out simply and quickly.

[0357] The processing flow will be explained below.

[0358] Step 1:

[0359] The user accesses the coordination support system using a smartphone or computer. The user activates the dialogue mode and inputs, "Please coordinate my new living room."

[0360] Step 2:

[0361] The device receives the user's input and displays questions to input detailed requirements. The user inputs information such as layout, size, style, color, and budget. For example, the user might input "living room size 20 square meters, style Scandinavian style, budget 100,000 yen, color scheme blue and white."

[0362] Step 3:

[0363] The device temporarily stores the user's requests and sends them to the server, which then activates a generative AI model to generate multiple interior design suggestions based on the user's requests. For example, it might suggest a blue and white Scandinavian-style sofa, a wooden low table, and a simple floor lamp.

[0364] Step 4:

[0365] The server sends the generated interior design proposal to the terminal, which displays the proposal to the user. The user checks the proposal and inputs a request for modification, such as "make the sofa design more casual."

[0366] Step 5:

[0367] The device then sends the correction request back to the server, which then passes it on to the generative AI model, which then regenerates a new proposal—for example, a new interior design proposal that includes a sofa with a casual design.

[0368] Step 6:

[0369] The server sends the regenerated interior proposal to the device, which displays the re-proposed content to the user, who then reviews the proposal and makes a final decision.

[0370] Step 7:

[0371] The device will activate its camera and switch to AR mode. The user can then use the camera to view the room and virtually see how the proposed furniture will be arranged in the room. For example, they can see how a sofa or table will be arranged on their smartphone screen.

[0372] Step 8:

[0373] If the user likes the proposed furniture, they click the "Purchase" button on their device. The server connects to the online shop and completes the purchase process. Purchase confirmation and shipping information are sent to the user's device.

[0374] Step 9:

[0375] When a user wants to dispose of unwanted furniture, they click the "Sell" button on their device. The server connects to an online platform (e.g., an auction site) and completes the sale procedure. The sale information is then sent to the user's device.

[0376] Through this series of steps, users can efficiently coordinate their rooms and purchase or sell the most suitable interior items.

[0377] Example 1

[0378] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0379] Traditional interior coordination requires a lot of time and effort, and users have to go through a lot of trial and error to achieve their ideal interior. In particular, there has been a lack of a system for receiving efficient and optimal interior suggestions when purchasing a new home or moving. In addition, it is difficult to visualize how the suggested items will be arranged in the actual room, and the purchasing and selling procedures are complicated.

[0380] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0381] In this invention, the server includes generative model means for interactively extracting user requirements, generative model means for generating interior proposals based on the extracted requirements, means for transmitting the generated interior proposals to a user terminal, means for the user terminal to display the generated interior proposals using an augmented reality function, and linking means for purchasing or selling the proposed interior items online. This allows users to easily communicate their detailed requirements, receive optimal interior proposals in a short amount of time, and visually confirm the placement of the proposed items, allowing them to carry out purchasing or selling procedures in a consistent manner.

[0382] The "generative model means" is a function that uses a generative AI model to analyze user requests and generate interior design proposals based on those requests.

[0383] A "user terminal" is a device, such as a smartphone or PC, that a user uses to access the interface.

[0384] The "interactive format" is a method in which a user inputs a request into the system in natural language, and the system then presents a series of questions in response to the request, thereby extracting the user's specific requirements and conditions.

[0385] "Augmented reality" is a technology that overlays digital information on the real world, specifically a function that displays virtual interior items through the camera of a smartphone or tablet.

[0386] The "linking means" is a system that exchanges data with external online shops or auction platforms in order to purchase or sell the proposed interior items online.

[0387] The "requirement extraction means" is a function that analyzes the data input by the user and identifies detailed requirements such as floor plan, size, design style, preferred color, and budget.

[0388] "Regeneration" is the process of accepting user requests for modifications and again using the generative AI model to generate new interior proposals.

[0389] The interior coordination support system of the present invention provides a specific means for a user to receive interior design suggestions when purchasing or moving into a new home. The present invention can be implemented using the system configuration and operation procedure described below.

[0390] The system's program runs on a cloud server and combines generative AI models and augmented reality technology. The main hardware and software used are as follows:

[0391] User devices: Smartphones (iPhone, Android), PCs (Windows, macOS)

[0392] Server: Cloud server (AWS, Google Cloud)

[0393] Generative AI models: GPT-4, DALL-E

[0394] Augmented reality technology: ARKit (iOS), ARCore (Android)

[0395] Online shop integration: EC site API (Amazon, Rakuten)

[0396] Selling platform integration: Auction site API (eBay, Mercari)

[0397] First, the user accesses the system's web interface using a smartphone or PC. After logging in, an interactive interface for inputting requests appears, where the user inputs the room's conditions and desired interior details.

[0398] For example, a user might input a specific request such as, "I would like the living room to be 20 square meters in size, with a Scandinavian design, a budget of 100,000 yen, and a blue and white theme." This input data is sent to the server, which then generates and sends a prompt to the generative AI model.

[0399] An example of this prompt is:

[0400] A user wants to decorate their living room. Please provide suggestions based on the following criteria:

[0401] Room size: 20 square meters

[0402] Design Desired: Scandinavian style

[0403] Budget: 100,000 yen

[0404] Color: Blue and White

[0405] We are looking for suggestions for specific interior items (sofa, table, lamp, etc.).

[0406] Based on this prompt, a generative AI model (e.g., GPT-4) generates multiple interior design proposals that match the user's requirements. The generated proposals are sent from the server to the user's device, where the user can confirm the proposals. For example, a blue and white Scandinavian-style sofa, a wooden low table, and a simple floor lamp.

[0407] If the user is not satisfied with the proposal, they can input a request for revision, such as a more specific request like "I want the sofa design to be more casual." This request for revision is sent again to the server, and the generative AI model generates a new proposal, which is again sent to the user's device. This process is repeated until the user is satisfied.

[0408] Finally, when the user is satisfied with the proposed items, the camera on the user's device is activated and the proposed interior items are virtually placed in the actual room using augmented reality.The user can check on their smartphone screen how the proposed sofa or table will look in each position in the room.

[0409] After checking the specific layout image of the interior items, if the user wishes to purchase them, they select a purchase option on their device. The server receives the user's selection and works with the online shop to complete the purchase process. Similarly, if the user needs to dispose of unwanted furniture, they can select the sell option, and the server will work with a platform such as an auction site to complete the selling process.

[0410] In this way, the system of the present invention allows the user to achieve an ideal interior coordination without much effort, and enables the process to be carried out simply and quickly.

[0411] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0412] Step 1:

[0413] A user accesses the system interface using a browser on a smartphone or PC. The user's input is to enter the system's URL and access the interface through the browser. The output of this step is that the interface is successfully accessed and the login screen is displayed.

[0414] Specific behavior:

[0415] The user launches a browser and enters the specified URL.

[0416] The browser displays the system interface.

[0417] Step 2:

[0418] The user logs in to the system. After logging in, an interactive questionnaire is displayed, and the user inputs details of the room situation and desired interior. The input data includes requirements such as room size, design preference, budget, and color. The output of this step is that the user's requirements are sent to the server.

[0419] Specific behavior:

[0420] The user enters their login information and logs into the system.

[0421] Answer the interactive questions that appear after logging in.

[0422] For example, enter "living room 20 square meters, Scandinavian style, budget 100,000 yen, blue and white as the main colors."

[0423] Step 3:

[0424] The server receives the request data sent by the user and generates and sends a prompt to the generative AI model. The input data includes the user's request. The generative AI model generates an interior proposal based on this prompt. The output of this step is the generated interior proposal.

[0425] Specific behavior:

[0426] The server receives the user's request.

[0427] The server sends the prompt to the generative AI model.

[0428] Example prompt sentence:

[0429] A user wants to decorate their living room. Please provide suggestions based on the following criteria:

[0430] Room size: 20 square meters

[0431] Design Desired: Scandinavian style

[0432] Budget: 100,000 yen

[0433] Color: Blue and White

[0434] We are looking for suggestions for specific interior items (sofa, table, lamp, etc.).

[0435] Step 4:

[0436] A generative AI model (e.g., GPT-4) generates interior design suggestions based on a prompt. The input data includes the prompt, and the output is an interior design suggestion (e.g., "A blue and white Scandinavian-style sofa, a wooden low table, and a simple floor lamp"). The output of this step is the interior design suggestion.

[0437] Specific behavior:

[0438] The generative AI model receives the prompt and generates interior design suggestions that meet the user's needs.

[0439] The generated interior proposal is sent to the server.

[0440] Step 5:

[0441] The server receives the generated interior proposal and sends it to the user terminal. The input data is the generated interior proposal and the output is the transmission to the user terminal. The output of this step is the display of the interior proposal on the user terminal.

[0442] Specific behavior:

[0443] The server receives the generated interior proposals.

[0444] The server transmits the interior proposal to the user terminal.

[0445] Step 6:

[0446] The user reviews the submitted interior proposal and enters revision requests if necessary. The input data includes the user's revision requests. These requests are sent to the server, where the generative AI model generates a new proposal. The output of this step is a new proposal based on the revision requests.

[0447] Specific behavior:

[0448] The user confirms the proposal.

[0449] Enter any correction requests you may have for the proposal (e.g., "Make the sofa design more casual").

[0450] A modification request is sent to the server.

[0451] Step 7:

[0452] The server receives the user's revision request and re-prompts the generative AI model to generate a new interior proposal. The input data includes the user's revision request. The output of this step is the newly generated interior proposal.

[0453] Specific behavior:

[0454] The server sends a modification request to the generative AI model.

[0455] A generative AI model generates new interior proposals.

[0456] The new proposal is sent to the server.

[0457] Step 8:

[0458] Once the user is satisfied with the proposal, the user device activates the camera and uses augmented reality to virtually place the proposed interior items in the actual room. The input data includes the generated interior proposal. The output of this step is for the user to confirm the virtual placement results.

[0459] Specific behavior:

[0460] The user activates the smartphone camera.

[0461] Use augmented reality to virtually place suggested items in a room.

[0462] Check the placement on your smartphone screen.

[0463] Step 9:

[0464] If the user wishes to purchase the proposed item, he / she selects the purchase option on the user's device. The server receives this information and cooperates with the online shop to complete the purchase process. The input data includes the user's intention to purchase. The output of this step is the completion of the purchase process.

[0465] Specific behavior:

[0466] The user presses the purchase button.

[0467] The server connects with the online shop and completes the purchase process.

[0468] Step 10:

[0469] If the user needs to dispose of unwanted furniture, they select the selling option. The server receives this information and cooperates with platforms such as auction sites to complete the selling process. The input data includes the user's intention to sell. The output of this step is the completion of the selling process.

[0470] Specific behavior:

[0471] The user presses the sell button.

[0472] The server connects with the auction site and completes the sales process.

[0473] As described above, in the present invention, each process is performed sequentially, thereby supporting the user in achieving an ideal interior coordination without much effort.

[0474] (Application example 1)

[0475] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0476] Conventional interior coordination systems have had problems such as difficulty in efficiently providing proposals that reflect the user's needs and limited means for users to see how the proposals would actually look in a room, resulting in low user satisfaction. Another problem is that the procedures for purchasing or selling the proposed interior items online are cumbersome. Furthermore, there has been no system that allows users to view the generated interior proposals in a virtual space and then smoothly proceed to purchase them. The objective of the present invention is to solve these problems.

[0477] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0478] In this invention, the server includes a generative modeling unit for interactively extracting user requirements, a generative modeling unit for generating interior design proposals based on the extracted requirements, a unit for transmitting the generated interior design proposals to a user terminal, a unit for displaying the generated interior design proposals on the user terminal using an AR function, a linking unit for purchasing or selling the proposed interior design items online, and a unit for viewing the generated interior design proposals in a virtual space and proceeding directly to a purchase. This allows users to quickly and accurately receive interior design proposals based on their detailed requirements, virtually arrange and view the proposals in an actual room, and smoothly complete the online purchase or sale process. Additionally, the ability to view the generated interior design proposals in a virtual space and proceed directly to a purchase significantly improves user satisfaction and convenience.

[0479] The "generative model means for interactively extracting user needs" is a device that includes artificial intelligence for analyzing information input by a user interactively and extracting those needs.

[0480] The "generative model means for generating interior proposals" is a device that includes artificial intelligence for generating optimal interior proposals based on extracted user requests.

[0481] The "means for transmitting the generated interior proposal to the user terminal" is a communication means for transmitting the interior proposal generated by the server to the user's device.

[0482] "Means for displaying interior proposals generated by a user's device using AR functionality" refers to technology for displaying interior proposals generated on a user's device using augmented reality functionality.

[0483] The "connection means for purchasing or selling the proposed interior items online" refers to a means for connecting with an online platform to simplify the process of purchasing or selling interior items online.

[0484] "A means to check the generated interior design proposal in a virtual space and proceed directly to purchase" refers to technology that allows the generated interior design proposal to be checked in real time in a virtual environment and the purchase process to be completed online.

[0485] The system for realizing this invention provides a series of steps to extract user needs interactively, generate interior design proposals based on those needs, confirm those proposals in a virtual space, and purchase or sell the most suitable interior items online. Each element will be explained in detail below.

[0486] The server is equipped with a generative AI model and an interactive question-and-answer algorithm. In particular, the generative AI model analyzes and extracts the user's requests and generates interior proposals based on them. The generated interior proposals are sent to the user's device, where the user can review them. In addition, if the user requests revisions to the proposal, the generative AI model will generate a new proposal again.

[0487] The user device may be a smartphone or tablet. The user device receives the interior design suggestions sent from the server and uses AR (Augmented Reality) functionality to virtually display them in the actual room environment. This allows the user to see in real time how the proposed interior items will look. AR technology such as ARKit for iOS or ARCore for Android is used.

[0488] Users can purchase items they like online or sell unwanted items. The server works with online shops and auction sites to facilitate these procedures.

[0489] As a concrete example, consider a user who wants to decorate the living room of their new home. The user accesses the system and inputs detailed requirements through the following interface:

[0490] Room details:

[0491] Room type: Living room

[0492] Area: 20 square meters

[0493] Design Desire:

[0494] Style: Nordic style

[0495] Color: Blue and White

[0496] Budget: 100,000 yen

[0497] Based on this request, the server uses a generative AI model to generate suggestions such as a blue and white Scandinavian sofa, a wooden table, and a simple floor lamp, and sends them to the user's device. The user can then virtually place these suggestions in their actual room using the AR function to see how the items will look.

[0498] An example prompt might be used as follows:

[0499] Please suggest interior coordination based on the user's request: The living room is 20 square meters, and the design is Scandinavian-style, with a budget of 100,000 yen. The main colors are blue and white.

[0500] In this way, the system of the present invention allows the user to achieve ideal interior coordination without much effort, and the process can be carried out simply and quickly.

[0501] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0502] Step 1: User request input

[0503] Users access the interface via a smartphone or tablet and enter detailed requirements such as the type of room, layout, size, desired design, preferred colors, budget, etc. Specific input data might include "living room, 20 square meters, Scandinavian design, blue and white, 100,000 yen."

[0504] Step 2: Extracting desired data

[0505] The server receives the input data sent by the user and uses a generative AI model to analyze and extract the user's requirements interactively. During this process, the user's input data is analyzed and processed to extract specific keywords (room type, size, desired design, etc.). The extracted data is then passed on to the next step.

[0506] Step 3: Generate interior proposals

[0507] The server uses a generative AI model to generate interior design proposals based on the extracted request data. A prompt statement is input into the generative model, and multiple interior design proposals are generated based on that. For example, a "blue and white Scandinavian-style sofa, wooden table, and simple floor lamp" is generated. This proposal data is passed on to the next step.

[0508] Step 4: Submit your interior design proposal

[0509] The generated interior design proposals are sent from the server to the user's device. Here, the proposal data (e.g., "a blue and white Scandinavian-style sofa, a wooden table, and a simple floor lamp") is sent to the user's smartphone or tablet. The user can then check the proposals on the screen.

[0510] Step 5: User correction request

[0511] The user can input their opinions and requests for modifications to the displayed interior design proposal. For example, they can send a specific request for modifications such as "I want the sofa design to be more casual." The user's modification request data is passed to the next step.

[0512] Step 6: Regenerate correction suggestions

[0513] The server receives the modification request and uses the generative AI model to regenerate a new interior proposal. In this process, a new prompt sentence is generated, and a modified interior proposal is generated based on it. For example, "a sofa with a casual design." This modified proposal data is passed to the next step.

[0514] Step 7: Final confirmation using AR display

[0515] The user device receives the regenerated interior proposals and virtually places them in the actual room using the smartphone's camera and AR function. The user can check in real time on the device screen how the interior items will look in each position. AR technology (such as ARKit for iOS or ARCore for Android) is used here.

[0516] Step 8: Buy or sell online

[0517] Finally, if the user is satisfied with the suggestions, they can choose to purchase online or sell the unwanted items. The server automatically processes the purchase and sale procedures by connecting with online shops and auction sites. Specifically, when the user presses the purchase or sale button, information about the suggested interior items is sent to the connected platform, completing the process.

[0518] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0519] This invention is a system that recognizes users' emotions and makes optimal interior design suggestions based on that information. The system combines generative AI models, AR technology, and an emotion engine to provide sophisticated interior design suggestions and actual layout images based on user requests, and also supports online buying and selling.

[0520] The usage flow of this system is as follows: First, the user accesses the coordination support system using a smartphone or PC. The user starts the dialogue mode and enters, "I'd like you to coordinate my new living room."

[0521] Next, the user device displays an interface for entering detailed requirements. The user enters information such as floor plan, size, style, color, and budget. For example, a specific requirement might be entered as "living room size 20 square meters, style Scandinavian, budget 100,000 yen, color scheme blue and white."

[0522] In addition, the emotion engine analyzes the user's emotions from their voice and facial expressions. When a user inputs a request, facial and voice data is captured through the camera and microphone on the user's device, and the emotion engine analyzes this. After the emotion data is analyzed, it is sent to the server, and a request is generated that takes the user's emotions into account.

[0523] The server then receives this request data and uses a generative AI model to generate interior design suggestions, such as a blue and white Scandinavian-style sofa, a wooden low table, and a simple floor lamp.

[0524] The generated suggestions are sent to the device and displayed to the user. The user reviews the suggestions and may input requests for revisions, such as "make the sofa design more casual." If a revision is requested, the device sends it back to the server, and the generative AI model generates a new suggestion. The server then refers to the user's emotional data again to make the optimal suggestion.

[0525] The process is repeated to ensure the proposal satisfies the user. Once the proposal is finalized, the user's device activates the camera and switches to AR mode. The user then views the room through their smartphone and sees how the proposed furniture will actually look when arranged. The AR function displays a virtual image of the interior items, allowing the user to view the entire room.

[0526] If the user checks the final proposal and likes the item, they select the purchase option on their device. The server connects to the online shop and completes the purchase procedure for the selected item. Purchase confirmation and shipping information are sent to the user's device. Furthermore, if the user selects the sell option for unwanted furniture, the server will link with an online platform such as an auction site and complete the selling procedure.

[0527] As a concrete example, consider the case of coordinating the living room of a new home. The user inputs detailed requirements into the system, and the emotion engine detects that the user is relaxed. Based on this information, the generative AI model suggests a "Nordic-style blue and white sofa with an emphasis on relaxation, a wooden low table, and a floor lamp with a simple design." The user submits a request for revisions, and the emotion engine confirms that the user is satisfied with the changes. Once the proposal is finalized, the user can visually check the actual layout using AR functionality, and finally purchase the items online. Unwanted old furniture can also be sold in the same way.

[0528] In this way, the system of the present invention takes into consideration the user's feelings and efficiently realizes the ideal interior coordination, and also supports simple and quick purchase and sale procedures.

[0529] The processing flow will be explained below.

[0530] Step 1:

[0531] The user accesses the coordination support system using a smartphone or computer. The user activates the dialogue mode and inputs, "Please coordinate my new living room."

[0532] Step 2:

[0533] The device receives the user's input and displays questions for entering detailed requirements. The user enters information such as floor plan, size, style, color, and budget. For example, the user might enter, "Living room size: 20 square meters, style: Scandinavian, budget: 100,000 yen, color: blue and white."

[0534] Step 3:

[0535] The device's camera and microphone are used to input the user's facial expressions and voice into the emotion engine, which then analyzes this data to determine the user's emotions, such as whether they are relaxed or excited. The analysis results are temporarily stored on the device.

[0536] Step 4:

[0537] The device sends the user's desires and emotional data to the server, which then activates a generative AI model to generate multiple interior design suggestions based on the user's desires and emotions. For example, the server suggests a "blue and white Scandinavian-style sofa, a wooden low table, and a simple floor lamp, all of which emphasize relaxation."

[0538] Step 5:

[0539] The server sends the generated interior design proposal to the terminal, which displays the proposal to the user. The user checks the proposal and inputs a request for modification, such as "make the sofa design more casual."

[0540] Step 6:

[0541] The device then sends the correction request to the server again. The server then communicates the correction request to the generative AI model, which then regenerates a new proposal. The generative AI model then refers to the user's emotional data again to make the optimal proposal.

[0542] Step 7:

[0543] The server sends the regenerated interior proposal to the device, which displays the re-proposed content to the user, who then reviews the proposal and makes a final decision.

[0544] Step 8:

[0545] The device will activate its camera and switch to AR mode. The user can then use the camera to view the room and virtually see how the proposed furniture will be arranged in the room. For example, they can see how a sofa or table will be arranged on their smartphone screen.

[0546] Step 9:

[0547] If the user likes the proposed furniture, they click the "Purchase" button on their device. The server connects to the online shop and completes the purchase process. Purchase confirmation and shipping information are sent to the user's device.

[0548] Step 10:

[0549] When a user wants to dispose of unwanted furniture, they click the "Sell" button on their device. The server connects to an online platform (e.g., an auction site) and completes the sale procedure. The sale information is then sent to the user's device.

[0550] Through this series of steps, users can interactively communicate their needs to the system, taking their own feelings into consideration, and efficiently receive interior design suggestions while purchasing the most suitable items or selling unwanted items.

[0551] Example 2

[0552] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0553] Modern interior coordination requires a great deal of time and effort to create a space that matches the user's emotions and preferences. It is also difficult for users to accurately convey their emotionally-based latent needs when expressing specific requests. The process of visualizing and revising proposals, as well as online purchasing and selling, can be complex and difficult to understand. To solve these problems, a comprehensive system is needed that considers the user's emotions, efficiently proposes interiors, and ultimately creates a comfortable and ideal space.

[0554] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes a data generation means for interactively extracting user desires, a generative model means for generating interior design proposals based on the extracted desires, a sentiment analysis means for analyzing the user's emotions, a means for the generative model to generate proposals taking the sentiment analysis data into consideration, a means for transmitting the generated interior design proposals to a user terminal, a means for the user terminal to display the generated interior design proposals using an augmented reality function, and a linking means for purchasing or selling the proposed interior design items online. This enables highly accurate interior design proposals that take the user's emotions into consideration, and by visually confirming and correcting them, a highly satisfying coordination can be quickly and efficiently achieved.

[0555] The "data generating means" has a function for extracting user requests in an interactive format.

[0556] The "generative model means" refers to a model used to generate interior proposals based on the extracted needs.

[0557] The "emotion analysis means" has a function for analyzing the user's emotions from facial expressions, voice, and the like.

[0558] The "proposal generation means" has the function of creating optimal interior proposals using a generative model while taking into account emotion analysis data.

[0559] The "data transmission means" has a function for transmitting the generated interior proposal to the user terminal.

[0560] The "augmented reality function" is a function that allows interior design proposals generated on a user's device to be virtually displayed in real space.

[0561] "Linkage means" means a function that links with other systems or platforms to purchase or sell the proposed interior items online.

[0562] The present invention is a system that recognizes a user's emotions and makes optimal interior design suggestions based on that information. This system is realized by combining a generative AI model, AR technology, and an emotion analysis engine, and provides advanced interior design suggestions and actual layout images based on the user's requests, and also supports online purchasing and selling. Specific embodiments of the present invention are described in detail below.

[0563] First, the user accesses the coordination support system using a device such as a smartphone or PC. The user activates the dialogue mode and inputs, "I'd like you to coordinate my new living room." This causes the user device to display an interface for inputting detailed requests. The user inputs information such as floor plan, size, style, color, and budget. For example, the user might input specific requests such as, "Living room size: 20 square meters, style: Scandinavian, budget: 100,000 yen, color: blue and white."

[0564] Next, the user device collects the user's facial expressions and tone of voice through a camera and microphone, which are then analyzed by an emotion analysis engine. The emotion analysis engine determines the user's emotional state, such as whether they are relaxed or excited. For example, if the engine detects that the user is relaxed when entering their request, that information is collected.

[0565] The device then sends the user's detailed information and analyzed emotion data to a server. The server then uses a generative AI model (e.g., a general generative model) to generate optimal interior design suggestions based on the user's desires and emotions. Specifically, the server generates suggestions such as a blue and white Scandinavian-style sofa, a wooden low table, and a simple floor lamp.

[0566] The generated suggestions are sent to the device and displayed to the user. The user reviews the suggestions and may input requests for revisions, such as "make the sofa design more casual." If a revision is requested, the device sends it back to the server, and the generative AI model generates a new suggestion. The server then refers to the user's emotional data again to make the optimal suggestion.

[0567] The process is repeated to ensure the proposal is satisfactory to the user. Once the proposal is finalized, the user's device activates the camera and switches to AR mode. The user then uses the device to view the entire room and see the virtual arrangement of the proposed furniture. The AR function displays a virtual image of the interior items arranged, allowing the user to view the entire room.

[0568] If the user checks the final proposal and likes the item, they can select the purchase option on their device. The server connects to the online shop and completes the purchase process for the selected item. Purchase confirmation and shipping information are sent to the user's device. Additionally, if the user selects the sell option for unwanted furniture, the server can link with an online platform such as an auction site to complete the selling process.

[0569] Examples of prompts include the following:

[0570] "I'd like the living room to be 20 square meters in a Scandinavian style. The color scheme should be blue and white, and the budget should be within 100,000 yen. I'd like it to have a relaxing feel."

[0571] "Please make the sofa design a little more casual."

[0572] In this way, the system of the present invention efficiently provides ideal interior coordination while taking into consideration the user's feelings, and also allows for simple and quick purchase and sale procedures.

[0573] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0574] Step 1:

[0575] A user accesses the system using a terminal. The user activates the interactive mode and inputs, "I'd like you to coordinate a new living room." This causes the terminal to display an interface for inputting detailed requests.

[0576] Input: User access requests and command input (e.g., "Please help me coordinate my new living room").

[0577] Output: Display of the coordination support interface.

[0578] Specific operation: When a user opens the system's web page or app on a smartphone or PC, the device displays the login screen and the user enters the login information. The user activates the interactive mode, and the system displays the request input screen.

[0579] Step 2:

[0580] The user inputs details of their coordination needs, and the user's device receives and records information such as floor plan, size, taste, color, and budget through a form on the interface.

[0581] Input: User-entered data (e.g., "Living room is 20 sq m, style is Scandinavian, budget is 100,000 yen, colors are blue and white").

[0582] Output: Detailed coordination request data.

[0583] Specific operation: The user enters their request into the text box on the interface, and the terminal receives it in real time and displays a confirmation message.

[0584] Step 3:

[0585] The user's device collects the user's facial expressions and voice through a camera and microphone, which are then analyzed by an emotion analysis engine.

[0586] Input: User's facial expression and voice data.

[0587] Output: Parsed emotion data (e.g. "relaxed").

[0588] How it works: The device's camera captures the user's face and microphone records their voice. The emotion analysis engine analyzes this in real time to determine whether the user is relaxed, etc.

[0589] Step 4:

[0590] The device sends the user's detailed information and emotional data to the server, which then uses a generative AI model to generate interior design proposals based on the user's requests and emotional data.

[0591] Input: User details and parsed emotion data.

[0592] Output: Initial interior design proposal (e.g., "A blue and white Scandinavian-style sofa, a wooden low table, and a simple floor lamp").

[0593] How it works: When the device sends data to the server, the server uses a generative AI model to generate interior design suggestions, such as a "Scandinavian-style sofa, a wooden low table, and a simple floor lamp."

[0594] Step 5:

[0595] The server sends the generated interior design proposal to the user's device, which displays the proposal and accepts user requests for revisions.

[0596] Input: Interior design proposal sent from the server.

[0597] Output: Interior proposals and modification requests displayed on the user's screen.

[0598] Specific operation: The server sends the generated proposal to the device, and the device displays "a blue and white Scandinavian-style sofa, a wooden low table, and a simple floor lamp." The user then inputs a modification request, such as "Make the sofa design more casual."

[0599] Step 6:

[0600] The user device then sends a correction request to the server again, and the generative AI model generates a new proposal. The server also references the user's emotional data again to make the optimal proposal.

[0601] Input: User modification request data.

[0602] Output: Revised interior proposal.

[0603] Specific operation: When a revision request is sent to the server, the generative AI model again generates interior design suggestions and creates revision suggestions, such as a new sofa design, which are then sent back to the device.

[0604] Step 7:

[0605] Once the proposal is finalized, the user's device activates the camera and switches to augmented reality mode, allowing the user to view the entire room using their smartphone and see how the proposed furniture will be virtually arranged.

[0606] Input: Confirmed interior proposal.

[0607] Output: Virtual interior layout display in augmented reality mode.

[0608] Specific operation: The user's device activates the camera and uses the AR function to display a screen showing a virtual room with furniture arranged in it. The user can then view the image of the entire room, including a blue and white Scandinavian-style sofa, through their smartphone.

[0609] Step 8:

[0610] If the user likes the suggestions and selects a purchase option, the server connects to the online shop and completes the purchase process for the selected item. Additionally, if the user selects the option to sell unwanted furniture, the server will connect to an online platform such as an auction site and complete the sales process.

[0611] Input: User's buy and sell selections.

[0612] Output: Purchase confirmation, shipping information, sale process.

[0613] Specific operation: When the user clicks the purchase button on the device, the selected item is ordered in cooperation with the online shop. The server sends a purchase confirmation and shipping information to the device. If the user selects the sell option, the server cooperates with the auction site to assist in the sale process.

[0614] (Application example 2)

[0615] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0616] Conventional interior design suggestion systems are unable to consider user emotions, making it difficult to increase user satisfaction. Additionally, interior design suggestions in physical stores lack real-time feedback functions to improve the user experience.

[0617] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0618] In this invention, the server includes a generative model means for interactively extracting user desires, a generative model means for generating interior design proposals based on the extracted desires, a means for transmitting the generated interior design proposals to a user terminal, a means for the user terminal to display the generated interior design proposals using an AR function, a linking means for purchasing or selling the proposed interior design items online, an emotion analysis means for recognizing the emotions of customers, and a generative model means for generating interior design proposals based on the emotion analysis. This makes it possible to make interior design proposals that take user emotions into consideration, and to make proposals that meet user needs in real time even in physical stores.

[0619] A "generative model" is an algorithm or process for generating interior design suggestions based on user input data and emotion data.

[0620] "Emotion analysis means" refers to technology or a system that analyzes a user's facial expressions and voice and recognizes their emotional state.

[0621] A "user terminal" is a hardware device that a user uses to access the system and receive interior design suggestions, and specifically includes a smartphone, tablet, smart glasses, etc.

[0622] "AR function" refers to the ability to display virtual interior items in real space using augmented reality technology.

[0623] "Online purchasing or selling collaboration" refers to a system or service that supports the process of a user purchasing or selling suggested interior items online.

[0624] This invention is a system that recognizes a user's emotions and proposes optimal interior design based on that information. Specifically, it is realized by combining emotion analysis means, generative model means, AR functionality, and linking means for online purchases or sales.

[0625] First, when a user visits a physical store, the emotion analysis means recognizes and analyzes the user's facial expressions and voice in real time to obtain emotional data. Emotion analysis uses a general emotion recognition algorithm, such as facial recognition technology or voice analysis technology.

[0626] Next, the generative model means interactively extracts the user's requirements. Through the interactive interface, the user inputs detailed information such as the layout, size, taste, preferred color, and budget. In addition, emotion data obtained by the emotion analysis means is also passed to the generative model means.

[0627] The generative modeling method generates interior design suggestions based on the user's input data and emotional data. For example, it might suggest a "relaxing Scandinavian-style blue and white sofa, a wooden low table, and a simple floor lamp." The generative AI model is used to design an interior design that best suits the user's needs.

[0628] The generated interior design suggestions are sent to the user's device, such as a tablet or smart glasses. The user can then use the device's AR functionality to virtually place and visualize the generated interior design suggestions in their actual space. Through this visualization, the user can check how well the suggested items fit into their own space.

[0629] If the user is satisfied with the proposal, they can proceed to the online purchase procedure on the spot. The purchase request sent from the terminal is linked to the online shop, where the final purchase procedure is carried out. In addition, for unwanted interior items, the user can select the option to sell them, and the selling procedure will be carried out in conjunction with the online platform.

[0630] As a concrete example, consider a user requesting coordination of the living room of their new home. In this case, the user inputs detailed requests into the system, and the emotion analysis means detects that the user is relaxed. Based on this information, the generative AI model suggests a "Nordic-style blue and white sofa that emphasizes relaxation, a wooden low table, and a floor lamp with a simple design." The user then submits a request for revisions, and the emotion analysis means confirms that the user is satisfied with the changes. Once the proposal is finalized, the user can visually check the actual layout using the AR function, and finally purchase the items online. Unwanted old furniture can also be sold in the same way.

[0631] Example prompt sentence:

[0632] "Our customers want to relax. Please suggest a Scandinavian-style blue and white sofa for the living room, which will create a relaxing atmosphere."

[0633] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0634] Step 1:

[0635] When a user visits a store, the device (smart glasses or tablet) captures their facial expressions and voice. The input data is collected in real time through a camera and microphone and sent to the emotion analysis means. The emotion analysis means uses facial expression recognition algorithms and voice analysis algorithms to analyze and identify the user's emotions as data. The output is the user's emotional state (e.g., "relaxed").

[0636] Step 2:

[0637] Next, the user inputs detailed information about the room for which they would like interior suggestions (layout, size, taste, preferred colors, budget, etc.) through their device. The data the user inputs into the dialogue interface on the tablet or smart glasses is sent to the generative model means. The input data is passed to the generative AI model, which analyzes the user's request and constructs a prompt sentence. The output is a specific prompt sentence that reflects the user's wishes.

[0638] Step 3:

[0639] The generative model means combines the emotion data obtained in step 1 with the user desire data obtained in step 2. The generative AI model generates interior design proposals based on this data. Using data mining and pattern recognition techniques, it designs an interior design plan that best suits the user's desires and emotions. The output is a specific interior design proposal (for example, "A relaxing Scandinavian-style blue and white sofa, a wooden low table, and a simply designed floor lamp").

[0640] Step 4:

[0641] The generated interior proposal is sent from the server to the device. The user can check the proposal on the device's display. At this point, the user can request revisions to the proposal. If revisions are requested, the proposal is sent again from the device to the server, and a new proposal is generated by the generative model means.

[0642] Step 5:

[0643] After the final proposal is finalized, the user's device uses its AR function to display a visual image of the interior items virtually arranged in the real store. The user uses the device's camera to capture the real store space and overlay the virtual interior on it. The output is an image of the virtual interior arrangement in the real space.

[0644] Step 6:

[0645] If the user is satisfied with the proposed interior items, the terminal supports the online purchase process. When the user presses the purchase button, a purchase request is sent to the online shop via the server. After the purchase process is completed, shipping information is sent to the user terminal. The output is a purchase confirmation and shipping information.

[0646] Step 7:

[0647] If a user wants to sell unwanted interior items, the user device supports the selling procedure. When the user selects the selling option, the server will connect to an online platform such as an auction site to complete the selling procedure. The output is a confirmation of the sale and detailed information.

[0648] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0649] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0650] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0651] [Third embodiment]

[0652] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0653] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0654] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0655] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0656] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0657] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0658] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0659] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0660] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0661] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0662] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0663] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0664] This invention is a support system for interior coordination when users purchase or move into a new home. It combines generative AI models and AR technology to make optimal interior suggestions based on the user's requests, and also allows for easy online buying and selling.

[0665] The system begins with the user accessing the interface via a smartphone or computer. Once the user logs in to the system, an interactive questionnaire appears, and the user inputs the details of the room and the desired interior. For example, a specific request could be conveyed to the system, such as, "I want a living room that is 20 square meters in size, with a Scandinavian design, a budget of 100,000 yen, and a blue and white theme."

[0666] The server then uses the generative AI model to generate multiple interior design proposals based on the user's needs. The proposals are sent to the user's device, where the user can review them. For example, a blue and white Scandinavian-style sofa, a wooden low table, and a simple floor lamp are suggested.

[0667] The user can comment on the displayed suggestions, for example, by inputting a request for revision, such as "I'd like the sofa design to be more casual." The request for revision is then sent back to the server, and the generative AI model generates a new suggestion. This process is repeated to ensure the suggestion satisfies the user.

[0668] Once the proposal is finalized, the user's device activates the camera and uses AR to virtually place the proposed interior items in the actual room. Through the smartphone screen, the user can see how the proposed sofa or table will look in each position in the room.

[0669] After checking the specific layout image of the interior items, if the user wishes to purchase them, they select the purchase option on their device. The server receives the user's selection and works with the online shop to complete the purchase process. Similarly, if the user needs to dispose of unwanted furniture, they can select the sell option, and the server will work with a platform such as an auction site to complete the selling process.

[0670] As a concrete example, consider the case of coordinating the living room of a new home. The user inputs detailed requirements into the system, and the generative AI model based on these inputs suggests a blue and white Scandinavian-style sofa, a wooden low table, and a floor lamp with a simple design. The user then sends a correction request, such as "make the sofa more casual," and the generative AI model re-proposes a sofa with a casual design in response to the request. The user then uses their smartphone camera to check how the proposed furniture would look in their actual living room. If they like the furniture, they can complete the purchase process online and sell any unwanted furniture.

[0671] In this way, the system of the present invention allows the user to achieve an ideal interior coordination without much effort, and enables the process to be carried out simply and quickly.

[0672] The processing flow will be explained below.

[0673] Step 1:

[0674] The user accesses the coordination support system using a smartphone or computer. The user activates the dialogue mode and inputs, "Please coordinate my new living room."

[0675] Step 2:

[0676] The device receives the user's input and displays questions to input detailed requirements. The user inputs information such as layout, size, style, color, and budget. For example, the user might input "living room size 20 square meters, style Scandinavian style, budget 100,000 yen, color scheme blue and white."

[0677] Step 3:

[0678] The device temporarily stores the user's requests and sends them to the server, which then activates a generative AI model to generate multiple interior design suggestions based on the user's requests. For example, it might suggest a blue and white Scandinavian-style sofa, a wooden low table, and a simple floor lamp.

[0679] Step 4:

[0680] The server sends the generated interior design proposal to the terminal, which displays the proposal to the user. The user checks the proposal and inputs a request for modification, such as "make the sofa design more casual."

[0681] Step 5:

[0682] The device then sends the correction request back to the server, which then passes it on to the generative AI model, which then regenerates a new proposal—for example, a new interior design proposal that includes a sofa with a casual design.

[0683] Step 6:

[0684] The server sends the regenerated interior proposal to the device, which displays the re-proposed content to the user, who then reviews the proposal and makes a final decision.

[0685] Step 7:

[0686] The device will activate its camera and switch to AR mode. The user can then use the camera to view the room and virtually see how the proposed furniture will be arranged in the room. For example, they can see how a sofa or table will be arranged on their smartphone screen.

[0687] Step 8:

[0688] If the user likes the proposed furniture, they click the "Purchase" button on their device. The server connects to the online shop and completes the purchase process. Purchase confirmation and shipping information are sent to the user's device.

[0689] Step 9:

[0690] When a user wants to dispose of unwanted furniture, they click the "Sell" button on their device. The server connects to an online platform (e.g., an auction site) and completes the sale procedure. The sale information is then sent to the user's device.

[0691] Through this series of steps, users can efficiently coordinate their rooms and purchase or sell the most suitable interior items.

[0692] Example 1

[0693] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0694] Traditional interior coordination requires a lot of time and effort, and users have to go through a lot of trial and error to achieve their ideal interior. In particular, there has been a lack of a system for receiving efficient and optimal interior suggestions when purchasing a new home or moving. In addition, it is difficult to visualize how the suggested items will be arranged in the actual room, and the purchasing and selling procedures are complicated.

[0695] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0696] In this invention, the server includes generative model means for interactively extracting user requirements, generative model means for generating interior proposals based on the extracted requirements, means for transmitting the generated interior proposals to a user terminal, means for the user terminal to display the generated interior proposals using an augmented reality function, and linking means for purchasing or selling the proposed interior items online. This allows users to easily communicate their detailed requirements, receive optimal interior proposals in a short amount of time, and visually confirm the placement of the proposed items, allowing them to carry out purchasing or selling procedures in a consistent manner.

[0697] The "generative model means" is a function that uses a generative AI model to analyze user requests and generate interior design proposals based on those requests.

[0698] A "user terminal" is a device, such as a smartphone or PC, that a user uses to access the interface.

[0699] The "interactive format" is a method in which a user inputs a request into the system in natural language, and the system then presents a series of questions in response to the request, thereby extracting the user's specific requirements and conditions.

[0700] "Augmented reality" is a technology that overlays digital information on the real world, specifically a function that displays virtual interior items through the camera of a smartphone or tablet.

[0701] The "linking means" is a system that exchanges data with external online shops or auction platforms in order to purchase or sell the proposed interior items online.

[0702] The "requirement extraction means" is a function that analyzes the data input by the user and identifies detailed requirements such as floor plan, size, design style, preferred color, and budget.

[0703] "Regeneration" is the process of accepting user requests for modifications and again using the generative AI model to generate new interior proposals.

[0704] The interior coordination support system of the present invention provides a specific means for a user to receive interior design suggestions when purchasing or moving into a new home. The present invention can be implemented using the system configuration and operation procedure described below.

[0705] The system's program runs on a cloud server and combines generative AI models and augmented reality technology. The main hardware and software used are as follows:

[0706] User devices: Smartphones (iPhone, Android), PCs (Windows, macOS)

[0707] Server: Cloud server (AWS, Google Cloud)

[0708] Generative AI models: GPT-4, DALL-E

[0709] Augmented reality technology: ARKit (iOS), ARCore (Android)

[0710] Online shop integration: EC site API (Amazon, Rakuten)

[0711] Selling platform integration: Auction site API (eBay, Mercari)

[0712] First, the user accesses the system's web interface using a smartphone or PC. After logging in, an interactive interface for inputting requests appears, where the user inputs the room's conditions and desired interior details.

[0713] For example, a user might input a specific request such as, "I would like the living room to be 20 square meters in size, with a Scandinavian design, a budget of 100,000 yen, and a blue and white theme." This input data is sent to the server, which then generates and sends a prompt to the generative AI model.

[0714] An example of this prompt is:

[0715] A user wants to decorate their living room. Please provide suggestions based on the following criteria:

[0716] Room size: 20 square meters

[0717] Design Desired: Scandinavian style

[0718] Budget: 100,000 yen

[0719] Color: Blue and White

[0720] We are looking for suggestions for specific interior items (sofa, table, lamp, etc.).

[0721] Based on this prompt, a generative AI model (e.g., GPT-4) generates multiple interior design proposals that match the user's requirements. The generated proposals are sent from the server to the user's device, where the user can confirm the proposals. For example, a blue and white Scandinavian-style sofa, a wooden low table, and a simple floor lamp.

[0722] If the user is not satisfied with the proposal, they can input a request for revision, such as a more specific request like "I want the sofa design to be more casual." This request for revision is sent again to the server, and the generative AI model generates a new proposal, which is again sent to the user's device. This process is repeated until the user is satisfied.

[0723] Finally, when the user is satisfied with the proposed items, the camera on the user's device is activated and the proposed interior items are virtually placed in the actual room using augmented reality.The user can check on their smartphone screen how the proposed sofa or table will look in each position in the room.

[0724] After checking the specific layout image of the interior items, if the user wishes to purchase them, they select a purchase option on their device. The server receives the user's selection and works with the online shop to complete the purchase process. Similarly, if the user needs to dispose of unwanted furniture, they can select the sell option, and the server will work with a platform such as an auction site to complete the selling process.

[0725] In this way, the system of the present invention allows the user to achieve an ideal interior coordination without much effort, and enables the process to be carried out simply and quickly.

[0726] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0727] Step 1:

[0728] A user accesses the system interface using a browser on a smartphone or PC. The user's input is to enter the system's URL and access the interface through the browser. The output of this step is that the interface is successfully accessed and the login screen is displayed.

[0729] Specific behavior:

[0730] The user launches a browser and enters the specified URL.

[0731] The browser displays the system interface.

[0732] Step 2:

[0733] The user logs in to the system. After logging in, an interactive questionnaire is displayed, and the user inputs details of the room situation and desired interior. The input data includes requirements such as room size, design preference, budget, and color. The output of this step is that the user's requirements are sent to the server.

[0734] Specific behavior:

[0735] The user enters their login information and logs into the system.

[0736] Answer the interactive questions that appear after logging in.

[0737] For example, enter "living room 20 square meters, Scandinavian style, budget 100,000 yen, blue and white as the main colors."

[0738] Step 3:

[0739] The server receives the request data sent by the user and generates and sends a prompt to the generative AI model. The input data includes the user's request. The generative AI model generates an interior proposal based on this prompt. The output of this step is the generated interior proposal.

[0740] Specific behavior:

[0741] The server receives the user's request.

[0742] The server sends the prompt to the generative AI model.

[0743] Example prompt sentence:

[0744] A user wants to decorate their living room. Please provide suggestions based on the following criteria:

[0745] Room size: 20 square meters

[0746] Design Desired: Scandinavian style

[0747] Budget: 100,000 yen

[0748] Color: Blue and White

[0749] We are looking for suggestions for specific interior items (sofa, table, lamp, etc.).

[0750] Step 4:

[0751] A generative AI model (e.g., GPT-4) generates interior design suggestions based on a prompt. The input data includes the prompt, and the output is an interior design suggestion (e.g., "A blue and white Scandinavian-style sofa, a wooden low table, and a simple floor lamp"). The output of this step is the interior design suggestion.

[0752] Specific behavior:

[0753] The generative AI model receives the prompt and generates interior design suggestions that meet the user's needs.

[0754] The generated interior proposal is sent to the server.

[0755] Step 5:

[0756] The server receives the generated interior proposal and sends it to the user terminal. The input data is the generated interior proposal and the output is the transmission to the user terminal. The output of this step is the display of the interior proposal on the user terminal.

[0757] Specific behavior:

[0758] The server receives the generated interior proposals.

[0759] The server transmits the interior proposal to the user terminal.

[0760] Step 6:

[0761] The user reviews the submitted interior proposal and enters revision requests if necessary. The input data includes the user's revision requests. These requests are sent to the server, where the generative AI model generates a new proposal. The output of this step is a new proposal based on the revision requests.

[0762] Specific behavior:

[0763] The user confirms the proposal.

[0764] Enter any correction requests you may have for the proposal (e.g., "Make the sofa design more casual").

[0765] A modification request is sent to the server.

[0766] Step 7:

[0767] The server receives the user's revision request and re-prompts the generative AI model to generate a new interior proposal. The input data includes the user's revision request. The output of this step is the newly generated interior proposal.

[0768] Specific behavior:

[0769] The server sends a modification request to the generative AI model.

[0770] A generative AI model generates new interior proposals.

[0771] The new proposal is sent to the server.

[0772] Step 8:

[0773] Once the user is satisfied with the proposal, the user device activates the camera and uses augmented reality to virtually place the proposed interior items in the actual room. The input data includes the generated interior proposal. The output of this step is for the user to confirm the virtual placement results.

[0774] Specific behavior:

[0775] The user activates the smartphone camera.

[0776] Use augmented reality to virtually place suggested items in a room.

[0777] Check the placement on your smartphone screen.

[0778] Step 9:

[0779] If the user wishes to purchase the proposed item, he / she selects the purchase option on the user's device. The server receives this information and cooperates with the online shop to complete the purchase process. The input data includes the user's intention to purchase. The output of this step is the completion of the purchase process.

[0780] Specific behavior:

[0781] The user presses the purchase button.

[0782] The server connects with the online shop and completes the purchase process.

[0783] Step 10:

[0784] If the user needs to dispose of unwanted furniture, they select the selling option. The server receives this information and cooperates with platforms such as auction sites to complete the selling process. The input data includes the user's intention to sell. The output of this step is the completion of the selling process.

[0785] Specific behavior:

[0786] The user presses the sell button.

[0787] The server connects with the auction site and completes the sales process.

[0788] As described above, in the present invention, each process is performed sequentially, thereby supporting the user in achieving an ideal interior coordination without much effort.

[0789] (Application example 1)

[0790] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0791] Conventional interior coordination systems have had problems such as difficulty in efficiently providing proposals that reflect the user's needs and limited means for users to see how the proposals would actually look in a room, resulting in low user satisfaction. Another problem is that the procedures for purchasing or selling the proposed interior items online are cumbersome. Furthermore, there has been no system that allows users to view the generated interior proposals in a virtual space and then smoothly proceed to purchase them. The objective of the present invention is to solve these problems.

[0792] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0793] In this invention, the server includes a generative modeling unit for interactively extracting user requirements, a generative modeling unit for generating interior design proposals based on the extracted requirements, a unit for transmitting the generated interior design proposals to a user terminal, a unit for displaying the generated interior design proposals on the user terminal using an AR function, a linking unit for purchasing or selling the proposed interior design items online, and a unit for viewing the generated interior design proposals in a virtual space and proceeding directly to a purchase. This allows users to quickly and accurately receive interior design proposals based on their detailed requirements, virtually arrange and view the proposals in an actual room, and smoothly complete the online purchase or sale process. Additionally, the ability to view the generated interior design proposals in a virtual space and proceed directly to a purchase significantly improves user satisfaction and convenience.

[0794] The "generative model means for interactively extracting user needs" is a device that includes artificial intelligence for analyzing information input by a user interactively and extracting those needs.

[0795] The "generative model means for generating interior proposals" is a device that includes artificial intelligence for generating optimal interior proposals based on extracted user requests.

[0796] The "means for transmitting the generated interior proposal to the user terminal" is a communication means for transmitting the interior proposal generated by the server to the user's device.

[0797] "Means for displaying interior proposals generated by a user's device using AR functionality" refers to technology for displaying interior proposals generated on a user's device using augmented reality functionality.

[0798] The "connection means for purchasing or selling the proposed interior items online" refers to a means for connecting with an online platform to simplify the process of purchasing or selling interior items online.

[0799] "A means to check the generated interior design proposal in a virtual space and proceed directly to purchase" refers to technology that allows the generated interior design proposal to be checked in real time in a virtual environment and the purchase process to be completed online.

[0800] The system for realizing this invention provides a series of steps to extract user needs interactively, generate interior design proposals based on those needs, confirm those proposals in a virtual space, and purchase or sell the most suitable interior items online. Each element will be explained in detail below.

[0801] The server is equipped with a generative AI model and an interactive question-and-answer algorithm. In particular, the generative AI model analyzes and extracts the user's requests and generates interior proposals based on them. The generated interior proposals are sent to the user's device, where the user can review them. In addition, if the user requests revisions to the proposal, the generative AI model will generate a new proposal again.

[0802] The user device may be a smartphone or tablet. The user device receives the interior design suggestions sent from the server and uses AR (Augmented Reality) functionality to virtually display them in the actual room environment. This allows the user to see in real time how the proposed interior items will look. AR technology such as ARKit for iOS or ARCore for Android is used.

[0803] Users can purchase items they like online or sell unwanted items. The server works with online shops and auction sites to facilitate these procedures.

[0804] As a concrete example, consider a user who wants to decorate the living room of their new home. The user accesses the system and inputs detailed requirements through the following interface:

[0805] Room details:

[0806] Room type: Living room

[0807] Area: 20 square meters

[0808] Design Desire:

[0809] Style: Nordic style

[0810] Color: Blue and White

[0811] Budget: 100,000 yen

[0812] Based on this request, the server uses a generative AI model to generate suggestions such as a blue and white Scandinavian sofa, a wooden table, and a simple floor lamp, and sends them to the user's device. The user can then virtually place these suggestions in their actual room using the AR function to see how the items will look.

[0813] An example prompt might be used as follows:

[0814] Please suggest interior coordination based on the user's request: The living room is 20 square meters, and the design is Scandinavian-style, with a budget of 100,000 yen. The main colors are blue and white.

[0815] In this way, the system of the present invention allows the user to achieve ideal interior coordination without much effort, and the process can be carried out simply and quickly.

[0816] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0817] Step 1: User request input

[0818] Users access the interface via a smartphone or tablet and enter detailed requirements such as the type of room, layout, size, desired design, preferred colors, budget, etc. Specific input data might include "living room, 20 square meters, Scandinavian design, blue and white, 100,000 yen."

[0819] Step 2: Extracting desired data

[0820] The server receives the input data sent by the user and uses a generative AI model to analyze and extract the user's requirements interactively. During this process, the user's input data is analyzed and processed to extract specific keywords (room type, size, desired design, etc.). The extracted data is then passed on to the next step.

[0821] Step 3: Generate interior proposals

[0822] The server uses a generative AI model to generate interior design proposals based on the extracted request data. A prompt statement is input into the generative model, and multiple interior design proposals are generated based on that. For example, a "blue and white Scandinavian-style sofa, wooden table, and simple floor lamp" is generated. This proposal data is passed on to the next step.

[0823] Step 4: Submit your interior design proposal

[0824] The generated interior design proposals are sent from the server to the user's device. Here, the proposal data (e.g., "a blue and white Scandinavian-style sofa, a wooden table, and a simple floor lamp") is sent to the user's smartphone or tablet. The user can then check the proposals on the screen.

[0825] Step 5: User correction request

[0826] The user can input their opinions and requests for modifications to the displayed interior design proposal. For example, they can send a specific request for modifications such as "I want the sofa design to be more casual." The user's modification request data is passed to the next step.

[0827] Step 6: Regenerate correction suggestions

[0828] The server receives the modification request and uses the generative AI model to regenerate a new interior proposal. In this process, a new prompt sentence is generated, and a modified interior proposal is generated based on it. For example, "a sofa with a casual design." This modified proposal data is passed to the next step.

[0829] Step 7: Final confirmation using AR display

[0830] The user device receives the regenerated interior proposals and virtually places them in the actual room using the smartphone's camera and AR function. The user can check in real time on the device screen how the interior items will look in each position. AR technology (such as ARKit for iOS or ARCore for Android) is used here.

[0831] Step 8: Buy or sell online

[0832] Finally, if the user is satisfied with the suggestions, they can choose to purchase online or sell the unwanted items. The server automatically processes the purchase and sale procedures by connecting with online shops and auction sites. Specifically, when the user presses the purchase or sale button, information about the suggested interior items is sent to the connected platform, completing the process.

[0833] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0834] This invention is a system that recognizes users' emotions and makes optimal interior design suggestions based on that information. The system combines generative AI models, AR technology, and an emotion engine to provide sophisticated interior design suggestions and actual layout images based on user requests, and also supports online buying and selling.

[0835] The usage flow of this system is as follows: First, the user accesses the coordination support system using a smartphone or PC. The user starts the dialogue mode and enters, "I'd like you to coordinate my new living room."

[0836] Next, the user device displays an interface for entering detailed requirements. The user enters information such as floor plan, size, style, color, and budget. For example, a specific requirement might be entered as "living room size 20 square meters, style Scandinavian, budget 100,000 yen, color scheme blue and white."

[0837] In addition, the emotion engine analyzes the user's emotions from their voice and facial expressions. When a user inputs a request, facial and voice data is captured through the camera and microphone on the user's device, and the emotion engine analyzes this. After the emotion data is analyzed, it is sent to the server, and a request is generated that takes the user's emotions into account.

[0838] The server then receives this request data and uses a generative AI model to generate interior design suggestions, such as a blue and white Scandinavian-style sofa, a wooden low table, and a simple floor lamp.

[0839] The generated suggestions are sent to the device and displayed to the user. The user reviews the suggestions and may input requests for revisions, such as "make the sofa design more casual." If a revision is requested, the device sends it back to the server, and the generative AI model generates a new suggestion. The server then refers to the user's emotional data again to make the optimal suggestion.

[0840] The process is repeated to ensure the proposal satisfies the user. Once the proposal is finalized, the user's device activates the camera and switches to AR mode. The user then views the room through their smartphone and sees how the proposed furniture will actually look when arranged. The AR function displays a virtual image of the interior items, allowing the user to view the entire room.

[0841] If the user checks the final proposal and likes the item, they select the purchase option on their device. The server connects to the online shop and completes the purchase procedure for the selected item. Purchase confirmation and shipping information are sent to the user's device. Furthermore, if the user selects the sell option for unwanted furniture, the server will link with an online platform such as an auction site and complete the selling procedure.

[0842] As a concrete example, consider the case of coordinating the living room of a new home. The user inputs detailed requirements into the system, and the emotion engine detects that the user is relaxed. Based on this information, the generative AI model suggests a "Nordic-style blue and white sofa with an emphasis on relaxation, a wooden low table, and a floor lamp with a simple design." The user submits a request for revisions, and the emotion engine confirms that the user is satisfied with the changes. Once the proposal is finalized, the user can visually check the actual layout using AR functionality, and finally purchase the items online. Unwanted old furniture can also be sold in the same way.

[0843] In this way, the system of the present invention takes into consideration the user's feelings and efficiently realizes the ideal interior coordination, and also supports simple and quick purchase and sale procedures.

[0844] The processing flow will be explained below.

[0845] Step 1:

[0846] The user accesses the coordination support system using a smartphone or computer. The user activates the dialogue mode and inputs, "Please coordinate my new living room."

[0847] Step 2:

[0848] The device receives the user's input and displays questions for entering detailed requirements. The user enters information such as floor plan, size, style, color, and budget. For example, the user might enter, "Living room size: 20 square meters, style: Scandinavian, budget: 100,000 yen, color: blue and white."

[0849] Step 3:

[0850] The device's camera and microphone are used to input the user's facial expressions and voice into the emotion engine, which then analyzes this data to determine the user's emotions, such as whether they are relaxed or excited. The analysis results are temporarily stored on the device.

[0851] Step 4:

[0852] The device sends the user's desires and emotional data to the server, which then activates a generative AI model to generate multiple interior design suggestions based on the user's desires and emotions. For example, the server suggests a "blue and white Scandinavian-style sofa, a wooden low table, and a simple floor lamp, all of which emphasize relaxation."

[0853] Step 5:

[0854] The server sends the generated interior design proposal to the terminal, which displays the proposal to the user. The user checks the proposal and inputs a request for modification, such as "make the sofa design more casual."

[0855] Step 6:

[0856] The device then sends the correction request to the server again. The server then communicates the correction request to the generative AI model, which then regenerates a new proposal. The generative AI model then refers to the user's emotional data again to make the optimal proposal.

[0857] Step 7:

[0858] The server sends the regenerated interior proposal to the device, which displays the re-proposed content to the user, who then reviews the proposal and makes a final decision.

[0859] Step 8:

[0860] The device will activate its camera and switch to AR mode. The user can then use the camera to view the room and virtually see how the proposed furniture will be arranged in the room. For example, they can see how a sofa or table will be arranged on their smartphone screen.

[0861] Step 9:

[0862] If the user likes the proposed furniture, they click the "Purchase" button on their device. The server connects to the online shop and completes the purchase process. Purchase confirmation and shipping information are sent to the user's device.

[0863] Step 10:

[0864] When a user wants to dispose of unwanted furniture, they click the "Sell" button on their device. The server connects to an online platform (e.g., an auction site) and completes the sale procedure. The sale information is then sent to the user's device.

[0865] Through this series of steps, users can interactively communicate their needs to the system, taking their own feelings into consideration, and efficiently receive interior design suggestions while purchasing the most suitable items or selling unwanted items.

[0866] Example 2

[0867] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0868] Modern interior coordination requires a great deal of time and effort to create a space that matches the user's emotions and preferences. It is also difficult for users to accurately convey their emotionally-based latent needs when expressing specific requests. The process of visualizing and revising proposals, as well as online purchasing and selling, can be complex and difficult to understand. To solve these problems, a comprehensive system is needed that considers the user's emotions, efficiently proposes interiors, and ultimately creates a comfortable and ideal space.

[0869] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes a data generation means for interactively extracting user desires, a generative model means for generating interior design proposals based on the extracted desires, a sentiment analysis means for analyzing the user's emotions, a means for the generative model to generate proposals taking the sentiment analysis data into consideration, a means for transmitting the generated interior design proposals to a user terminal, a means for the user terminal to display the generated interior design proposals using an augmented reality function, and a linking means for purchasing or selling the proposed interior design items online. This enables highly accurate interior design proposals that take the user's emotions into consideration, and by visually confirming and correcting them, a highly satisfying coordination can be quickly and efficiently achieved.

[0870] The "data generating means" has a function for extracting user requests in an interactive format.

[0871] The "generative model means" refers to a model used to generate interior proposals based on the extracted needs.

[0872] The "emotion analysis means" has a function for analyzing the user's emotions from facial expressions, voice, and the like.

[0873] The "proposal generation means" has the function of creating optimal interior proposals using a generative model while taking into account emotion analysis data.

[0874] The "data transmission means" has a function for transmitting the generated interior proposal to the user terminal.

[0875] The "augmented reality function" is a function that allows interior design proposals generated on a user's device to be virtually displayed in real space.

[0876] "Linkage means" means a function that links with other systems or platforms to purchase or sell the proposed interior items online.

[0877] The present invention is a system that recognizes a user's emotions and makes optimal interior design suggestions based on that information. This system is realized by combining a generative AI model, AR technology, and an emotion analysis engine, and provides advanced interior design suggestions and actual layout images based on the user's requests, and also supports online purchasing and selling. Specific embodiments of the present invention are described in detail below.

[0878] First, the user accesses the coordination support system using a device such as a smartphone or PC. The user activates the dialogue mode and inputs, "I'd like you to coordinate my new living room." This causes the user device to display an interface for inputting detailed requests. The user inputs information such as floor plan, size, style, color, and budget. For example, the user might input specific requests such as, "Living room size: 20 square meters, style: Scandinavian, budget: 100,000 yen, color: blue and white."

[0879] Next, the user device collects the user's facial expressions and tone of voice through a camera and microphone, which are then analyzed by an emotion analysis engine. The emotion analysis engine determines the user's emotional state, such as whether they are relaxed or excited. For example, if the engine detects that the user is relaxed when entering their request, that information is collected.

[0880] The device then sends the user's detailed information and analyzed emotion data to a server. The server then uses a generative AI model (e.g., a general generative model) to generate optimal interior design suggestions based on the user's desires and emotions. Specifically, the server generates suggestions such as a blue and white Scandinavian-style sofa, a wooden low table, and a simple floor lamp.

[0881] The generated suggestions are sent to the device and displayed to the user. The user reviews the suggestions and may input requests for revisions, such as "make the sofa design more casual." If a revision is requested, the device sends it back to the server, and the generative AI model generates a new suggestion. The server then refers to the user's emotional data again to make the optimal suggestion.

[0882] The process is repeated to ensure the proposal is satisfactory to the user. Once the proposal is finalized, the user's device activates the camera and switches to AR mode. The user then uses the device to view the entire room and see the virtual arrangement of the proposed furniture. The AR function displays a virtual image of the interior items arranged, allowing the user to view the entire room.

[0883] If the user checks the final proposal and likes the item, they can select the purchase option on their device. The server connects to the online shop and completes the purchase process for the selected item. Purchase confirmation and shipping information are sent to the user's device. Additionally, if the user selects the sell option for unwanted furniture, the server can link with an online platform such as an auction site to complete the selling process.

[0884] Examples of prompts include the following:

[0885] "I'd like the living room to be 20 square meters in a Scandinavian style. The color scheme should be blue and white, and the budget should be within 100,000 yen. I'd like it to have a relaxing feel."

[0886] "Please make the sofa design a little more casual."

[0887] In this way, the system of the present invention efficiently provides ideal interior coordination while taking into consideration the user's feelings, and also allows for simple and quick purchase and sale procedures.

[0888] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0889] Step 1:

[0890] A user accesses the system using a terminal. The user activates the interactive mode and inputs, "I'd like you to coordinate a new living room." This causes the terminal to display an interface for inputting detailed requests.

[0891] Input: User access requests and command input (e.g., "Please help me coordinate my new living room").

[0892] Output: Display of the coordination support interface.

[0893] Specific operation: When a user opens the system's web page or app on a smartphone or PC, the device displays the login screen and the user enters the login information. The user activates the interactive mode, and the system displays the request input screen.

[0894] Step 2:

[0895] The user inputs details of their coordination needs, and the user's device receives and records information such as floor plan, size, taste, color, and budget through a form on the interface.

[0896] Input: User-entered data (e.g., "Living room is 20 sq m, style is Scandinavian, budget is 100,000 yen, colors are blue and white").

[0897] Output: Detailed coordination request data.

[0898] Specific operation: The user enters their request into the text box on the interface, and the terminal receives it in real time and displays a confirmation message.

[0899] Step 3:

[0900] The user's device collects the user's facial expressions and voice through a camera and microphone, which are then analyzed by an emotion analysis engine.

[0901] Input: User's facial expression and voice data.

[0902] Output: Parsed emotion data (e.g. "relaxed").

[0903] How it works: The device's camera captures the user's face and microphone records their voice. The emotion analysis engine analyzes this in real time to determine whether the user is relaxed, etc.

[0904] Step 4:

[0905] The device sends the user's detailed information and emotional data to the server, which then uses a generative AI model to generate interior design proposals based on the user's requests and emotional data.

[0906] Input: User details and parsed emotion data.

[0907] Output: Initial interior design proposal (e.g., "A blue and white Scandinavian-style sofa, a wooden low table, and a simple floor lamp").

[0908] How it works: When the device sends data to the server, the server uses a generative AI model to generate interior design suggestions, such as a "Scandinavian-style sofa, a wooden low table, and a simple floor lamp."

[0909] Step 5:

[0910] The server sends the generated interior design proposal to the user's device, which displays the proposal and accepts user requests for revisions.

[0911] Input: Interior design proposal sent from the server.

[0912] Output: Interior proposals and modification requests displayed on the user's screen.

[0913] Specific operation: The server sends the generated proposal to the device, and the device displays "a blue and white Scandinavian-style sofa, a wooden low table, and a simple floor lamp." The user then inputs a modification request, such as "Make the sofa design more casual."

[0914] Step 6:

[0915] The user device then sends a correction request to the server again, and the generative AI model generates a new proposal. The server also references the user's emotional data again to make the optimal proposal.

[0916] Input: User modification request data.

[0917] Output: Revised interior proposal.

[0918] Specific operation: When a revision request is sent to the server, the generative AI model again generates interior design suggestions and creates revision suggestions, such as a new sofa design, which are then sent back to the device.

[0919] Step 7:

[0920] Once the proposal is finalized, the user's device activates the camera and switches to augmented reality mode, allowing the user to view the entire room using their smartphone and see how the proposed furniture will be virtually arranged.

[0921] Input: Confirmed interior proposal.

[0922] Output: Virtual interior layout display in augmented reality mode.

[0923] Specific operation: The user's device activates the camera and uses the AR function to display a screen showing a virtual room with furniture arranged in it. The user can then view the image of the entire room, including a blue and white Scandinavian-style sofa, through their smartphone.

[0924] Step 8:

[0925] If the user likes the suggestions and selects a purchase option, the server connects to the online shop and completes the purchase process for the selected item. Additionally, if the user selects the option to sell unwanted furniture, the server will connect to an online platform such as an auction site and complete the sales process.

[0926] Input: User's buy and sell selections.

[0927] Output: Purchase confirmation, shipping information, sale process.

[0928] Specific operation: When the user clicks the purchase button on the device, the selected item is ordered in cooperation with the online shop. The server sends a purchase confirmation and shipping information to the device. If the user selects the sell option, the server cooperates with the auction site to assist in the sale process.

[0929] (Application example 2)

[0930] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0931] Conventional interior design suggestion systems are unable to consider user emotions, making it difficult to increase user satisfaction. Additionally, interior design suggestions in physical stores lack real-time feedback functions to improve the user experience.

[0932] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0933] In this invention, the server includes a generative model means for interactively extracting user desires, a generative model means for generating interior design proposals based on the extracted desires, a means for transmitting the generated interior design proposals to a user terminal, a means for the user terminal to display the generated interior design proposals using an AR function, a linking means for purchasing or selling the proposed interior design items online, an emotion analysis means for recognizing the emotions of customers, and a generative model means for generating interior design proposals based on the emotion analysis. This makes it possible to make interior design proposals that take user emotions into consideration, and to make proposals that meet user needs in real time even in physical stores.

[0934] A "generative model" is an algorithm or process for generating interior design suggestions based on user input data and emotion data.

[0935] "Emotion analysis means" refers to technology or a system that analyzes a user's facial expressions and voice and recognizes their emotional state.

[0936] A "user terminal" is a hardware device that a user uses to access the system and receive interior design suggestions, and specifically includes a smartphone, tablet, smart glasses, etc.

[0937] "AR function" refers to the ability to display virtual interior items in real space using augmented reality technology.

[0938] "Online purchasing or selling collaboration" refers to a system or service that supports the process of a user purchasing or selling suggested interior items online.

[0939] This invention is a system that recognizes a user's emotions and proposes optimal interior design based on that information. Specifically, it is realized by combining emotion analysis means, generative model means, AR functionality, and linking means for online purchases or sales.

[0940] First, when a user visits a physical store, the emotion analysis means recognizes and analyzes the user's facial expressions and voice in real time to obtain emotional data. Emotion analysis uses a general emotion recognition algorithm, such as facial recognition technology or voice analysis technology.

[0941] Next, the generative model means interactively extracts the user's requirements. Through the interactive interface, the user inputs detailed information such as the layout, size, taste, preferred color, and budget. In addition, emotion data obtained by the emotion analysis means is also passed to the generative model means.

[0942] The generative modeling method generates interior design suggestions based on the user's input data and emotional data. For example, it might suggest a "relaxing Scandinavian-style blue and white sofa, a wooden low table, and a simple floor lamp." The generative AI model is used to design an interior design that best suits the user's needs.

[0943] The generated interior design suggestions are sent to the user's device, such as a tablet or smart glasses. The user can then use the device's AR functionality to virtually place and visualize the generated interior design suggestions in their actual space. Through this visualization, the user can check how well the suggested items fit into their own space.

[0944] If the user is satisfied with the proposal, they can proceed to the online purchase procedure on the spot. The purchase request sent from the terminal is linked to the online shop, where the final purchase procedure is carried out. In addition, for unwanted interior items, the user can select the option to sell them, and the selling procedure will be carried out in conjunction with the online platform.

[0945] As a concrete example, consider a user requesting coordination of the living room of their new home. In this case, the user inputs detailed requests into the system, and the emotion analysis means detects that the user is relaxed. Based on this information, the generative AI model suggests a "Nordic-style blue and white sofa that emphasizes relaxation, a wooden low table, and a floor lamp with a simple design." The user then submits a request for revisions, and the emotion analysis means confirms that the user is satisfied with the changes. Once the proposal is finalized, the user can visually check the actual layout using the AR function, and finally purchase the items online. Unwanted old furniture can also be sold in the same way.

[0946] Example prompt sentence:

[0947] "Our customers want to relax. Please suggest a Scandinavian-style blue and white sofa for the living room, which will create a relaxing atmosphere."

[0948] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0949] Step 1:

[0950] When a user visits a store, the device (smart glasses or tablet) captures their facial expressions and voice. The input data is collected in real time through a camera and microphone and sent to the emotion analysis means. The emotion analysis means uses facial expression recognition algorithms and voice analysis algorithms to analyze and identify the user's emotions as data. The output is the user's emotional state (e.g., "relaxed").

[0951] Step 2:

[0952] Next, the user inputs detailed information about the room for which they would like interior suggestions (layout, size, taste, preferred colors, budget, etc.) through their device. The data the user inputs into the dialogue interface on the tablet or smart glasses is sent to the generative model means. The input data is passed to the generative AI model, which analyzes the user's request and constructs a prompt sentence. The output is a specific prompt sentence that reflects the user's wishes.

[0953] Step 3:

[0954] The generative model means combines the emotion data obtained in step 1 with the user desire data obtained in step 2. The generative AI model generates interior design proposals based on this data. Using data mining and pattern recognition techniques, it designs an interior design plan that best suits the user's desires and emotions. The output is a specific interior design proposal (for example, "A relaxing Scandinavian-style blue and white sofa, a wooden low table, and a simply designed floor lamp").

[0955] Step 4:

[0956] The generated interior proposal is sent from the server to the device. The user can check the proposal on the device's display. At this point, the user can request revisions to the proposal. If revisions are requested, the proposal is sent again from the device to the server, and a new proposal is generated by the generative model means.

[0957] Step 5:

[0958] After the final proposal is finalized, the user's device uses its AR function to display a visual image of the interior items virtually arranged in the real store. The user uses the device's camera to capture the real store space and overlay the virtual interior on it. The output is an image of the virtual interior arrangement in the real space.

[0959] Step 6:

[0960] If the user is satisfied with the proposed interior items, the terminal supports the online purchase process. When the user presses the purchase button, a purchase request is sent to the online shop via the server. After the purchase process is completed, shipping information is sent to the user terminal. The output is a purchase confirmation and shipping information.

[0961] Step 7:

[0962] If a user wants to sell unwanted interior items, the user device supports the selling procedure. When the user selects the selling option, the server will connect to an online platform such as an auction site to complete the selling procedure. The output is a confirmation of the sale and detailed information.

[0963] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0964] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0965] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[0966] [Fourth embodiment]

[0967] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[0968] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0969] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0970] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[0971] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0972] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0973] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0974] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[0975] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0976] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0977] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0978] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0979] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[0980] This invention is a support system for interior coordination when users purchase or move into a new home. It combines generative AI models and AR technology to make optimal interior suggestions based on the user's requests, and also allows for easy online buying and selling.

[0981] The system begins with the user accessing the interface via a smartphone or computer. Once the user logs in to the system, an interactive questionnaire appears, and the user inputs the details of the room and the desired interior. For example, a specific request could be conveyed to the system, such as, "I want a living room that is 20 square meters in size, with a Scandinavian design, a budget of 100,000 yen, and a blue and white theme."

[0982] The server then uses the generative AI model to generate multiple interior design proposals based on the user's needs. The proposals are sent to the user's device, where the user can review them. For example, a blue and white Scandinavian-style sofa, a wooden low table, and a simple floor lamp are suggested.

[0983] The user can comment on the displayed suggestions, for example, by inputting a request for revision, such as "I'd like the sofa design to be more casual." The request for revision is then sent back to the server, and the generative AI model generates a new suggestion. This process is repeated to ensure the suggestion satisfies the user.

[0984] Once the proposal is finalized, the user's device activates the camera and uses AR to virtually place the proposed interior items in the actual room. Through the smartphone screen, the user can see how the proposed sofa or table will look in each position in the room.

[0985] After checking the specific layout image of the interior items, if the user wishes to purchase them, they select the purchase option on their device. The server receives the user's selection and works with the online shop to complete the purchase process. Similarly, if the user needs to dispose of unwanted furniture, they can select the sell option, and the server will work with a platform such as an auction site to complete the selling process.

[0986] As a concrete example, consider the case of coordinating the living room of a new home. The user inputs detailed requirements into the system, and the generative AI model based on these inputs suggests a blue and white Scandinavian-style sofa, a wooden low table, and a floor lamp with a simple design. The user then sends a correction request, such as "make the sofa more casual," and the generative AI model re-proposes a sofa with a casual design in response to the request. The user then uses their smartphone camera to check how the proposed furniture would look in their actual living room. If they like the furniture, they can complete the purchase process online and sell any unwanted furniture.

[0987] In this way, the system of the present invention allows the user to achieve an ideal interior coordination without much effort, and enables the process to be carried out simply and quickly.

[0988] The processing flow will be explained below.

[0989] Step 1:

[0990] The user accesses the coordination support system using a smartphone or computer. The user activates the dialogue mode and inputs, "Please coordinate my new living room."

[0991] Step 2:

[0992] The device receives the user's input and displays questions to input detailed requirements. The user inputs information such as layout, size, style, color, and budget. For example, the user might input "living room size 20 square meters, style Scandinavian style, budget 100,000 yen, color scheme blue and white."

[0993] Step 3:

[0994] The device temporarily stores the user's requests and sends them to the server, which then activates a generative AI model to generate multiple interior design suggestions based on the user's requests. For example, it might suggest a blue and white Scandinavian-style sofa, a wooden low table, and a simple floor lamp.

[0995] Step 4:

[0996] The server sends the generated interior design proposal to the terminal, which displays the proposal to the user. The user checks the proposal and inputs a request for modification, such as "make the sofa design more casual."

[0997] Step 5:

[0998] The device then sends the correction request back to the server, which then passes it on to the generative AI model, which then regenerates a new proposal—for example, a new interior design proposal that includes a sofa with a casual design.

[0999] Step 6:

[1000] The server sends the regenerated interior proposal to the device, which displays the re-proposed content to the user, who then reviews the proposal and makes a final decision.

[1001] Step 7:

[1002] The device will activate its camera and switch to AR mode. The user can then use the camera to view the room and virtually see how the proposed furniture will be arranged in the room. For example, they can see how a sofa or table will be arranged on their smartphone screen.

[1003] Step 8:

[1004] If the user likes the proposed furniture, they click the "Purchase" button on their device. The server connects to the online shop and completes the purchase process. Purchase confirmation and shipping information are sent to the user's device.

[1005] Step 9:

[1006] When a user wants to dispose of unwanted furniture, they click the "Sell" button on their device. The server connects to an online platform (e.g., an auction site) and completes the sale procedure. The sale information is then sent to the user's device.

[1007] Through this series of steps, users can efficiently coordinate their rooms and purchase or sell the most suitable interior items.

[1008] Example 1

[1009] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1010] Traditional interior coordination requires a lot of time and effort, and users have to go through a lot of trial and error to achieve their ideal interior. In particular, there has been a lack of a system for receiving efficient and optimal interior suggestions when purchasing a new home or moving. In addition, it is difficult to visualize how the suggested items will be arranged in the actual room, and the purchasing and selling procedures are complicated.

[1011] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1012] In this invention, the server includes generative model means for interactively extracting user requirements, generative model means for generating interior proposals based on the extracted requirements, means for transmitting the generated interior proposals to a user terminal, means for the user terminal to display the generated interior proposals using an augmented reality function, and linking means for purchasing or selling the proposed interior items online. This allows users to easily communicate their detailed requirements, receive optimal interior proposals in a short amount of time, and visually confirm the placement of the proposed items, allowing them to carry out purchasing or selling procedures in a consistent manner.

[1013] The "generative model means" is a function that uses a generative AI model to analyze user requests and generate interior design proposals based on those requests.

[1014] A "user terminal" is a device, such as a smartphone or PC, that a user uses to access the interface.

[1015] The "interactive format" is a method in which a user inputs a request into the system in natural language, and the system then presents a series of questions in response to the request, thereby extracting the user's specific requirements and conditions.

[1016] "Augmented reality" is a technology that overlays digital information on the real world, specifically a function that displays virtual interior items through the camera of a smartphone or tablet.

[1017] The "linking means" is a system that exchanges data with external online shops or auction platforms in order to purchase or sell the proposed interior items online.

[1018] The "requirement extraction means" is a function that analyzes the data input by the user and identifies detailed requirements such as floor plan, size, design style, preferred color, and budget.

[1019] "Regeneration" is the process of accepting user requests for modifications and again using the generative AI model to generate new interior proposals.

[1020] The interior coordination support system of the present invention provides a specific means for a user to receive interior design suggestions when purchasing or moving into a new home. The present invention can be implemented using the system configuration and operation procedure described below.

[1021] The system's program runs on a cloud server and combines generative AI models and augmented reality technology. The main hardware and software used are as follows:

[1022] User devices: Smartphones (iPhone, Android), PCs (Windows, macOS)

[1023] Server: Cloud server (AWS, Google Cloud)

[1024] Generative AI models: GPT-4, DALL-E

[1025] Augmented reality technology: ARKit (iOS), ARCore (Android)

[1026] Online shop integration: EC site API (Amazon, Rakuten)

[1027] Selling platform integration: Auction site API (eBay, Mercari)

[1028] First, the user accesses the system's web interface using a smartphone or PC. After logging in, an interactive interface for inputting requests appears, where the user inputs the room's conditions and desired interior details.

[1029] For example, a user might input a specific request such as, "I would like the living room to be 20 square meters in size, with a Scandinavian design, a budget of 100,000 yen, and a blue and white theme." This input data is sent to the server, which then generates and sends a prompt to the generative AI model.

[1030] An example of this prompt is:

[1031] A user wants to decorate their living room. Please provide suggestions based on the following criteria:

[1032] Room size: 20 square meters

[1033] Design Desired: Scandinavian style

[1034] Budget: 100,000 yen

[1035] Color: Blue and White

[1036] We are looking for suggestions for specific interior items (sofa, table, lamp, etc.).

[1037] Based on this prompt, a generative AI model (e.g., GPT-4) generates multiple interior design proposals that match the user's requirements. The generated proposals are sent from the server to the user's device, where the user can confirm the proposals. For example, a blue and white Scandinavian-style sofa, a wooden low table, and a simple floor lamp.

[1038] If the user is not satisfied with the proposal, they can input a request for revision, such as a more specific request like "I want the sofa design to be more casual." This request for revision is sent again to the server, and the generative AI model generates a new proposal, which is again sent to the user's device. This process is repeated until the user is satisfied.

[1039] Finally, when the user is satisfied with the proposed items, the camera on the user's device is activated and the proposed interior items are virtually placed in the actual room using augmented reality.The user can check on their smartphone screen how the proposed sofa or table will look in each position in the room.

[1040] After checking the specific layout image of the interior items, if the user wishes to purchase them, they select a purchase option on their device. The server receives the user's selection and works with the online shop to complete the purchase process. Similarly, if the user needs to dispose of unwanted furniture, they can select the sell option, and the server will work with a platform such as an auction site to complete the selling process.

[1041] In this way, the system of the present invention allows the user to achieve an ideal interior coordination without much effort, and enables the process to be carried out simply and quickly.

[1042] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1043] Step 1:

[1044] A user accesses the system interface using a browser on a smartphone or PC. The user's input is to enter the system's URL and access the interface through the browser. The output of this step is that the interface is successfully accessed and the login screen is displayed.

[1045] Specific behavior:

[1046] The user launches a browser and enters the specified URL.

[1047] The browser displays the system interface.

[1048] Step 2:

[1049] The user logs in to the system. After logging in, an interactive questionnaire is displayed, and the user inputs details of the room situation and desired interior. The input data includes requirements such as room size, design preference, budget, and color. The output of this step is that the user's requirements are sent to the server.

[1050] Specific behavior:

[1051] The user enters their login information and logs into the system.

[1052] Answer the interactive questions that appear after logging in.

[1053] For example, enter "living room 20 square meters, Scandinavian style, budget 100,000 yen, blue and white as the main colors."

[1054] Step 3:

[1055] The server receives the request data sent by the user and generates and sends a prompt to the generative AI model. The input data includes the user's request. The generative AI model generates an interior proposal based on this prompt. The output of this step is the generated interior proposal.

[1056] Specific behavior:

[1057] The server receives the user's request.

[1058] The server sends the prompt to the generative AI model.

[1059] Example prompt sentence:

[1060] A user wants to decorate their living room. Please provide suggestions based on the following criteria:

[1061] Room size: 20 square meters

[1062] Design Desired: Scandinavian style

[1063] Budget: 100,000 yen

[1064] Color: Blue and White

[1065] We are looking for suggestions for specific interior items (sofa, table, lamp, etc.).

[1066] Step 4:

[1067] A generative AI model (e.g., GPT-4) generates interior design suggestions based on a prompt. The input data includes the prompt, and the output is an interior design suggestion (e.g., "A blue and white Scandinavian-style sofa, a wooden low table, and a simple floor lamp"). The output of this step is the interior design suggestion.

[1068] Specific behavior:

[1069] The generative AI model receives the prompt and generates interior design suggestions that meet the user's needs.

[1070] The generated interior proposal is sent to the server.

[1071] Step 5:

[1072] The server receives the generated interior proposal and sends it to the user terminal. The input data is the generated interior proposal and the output is the transmission to the user terminal. The output of this step is the display of the interior proposal on the user terminal.

[1073] Specific behavior:

[1074] The server receives the generated interior proposals.

[1075] The server transmits the interior proposal to the user terminal.

[1076] Step 6:

[1077] The user reviews the submitted interior proposal and enters revision requests if necessary. The input data includes the user's revision requests. These requests are sent to the server, where the generative AI model generates a new proposal. The output of this step is a new proposal based on the revision requests.

[1078] Specific behavior:

[1079] The user confirms the proposal.

[1080] Enter any correction requests you may have for the proposal (e.g., "Make the sofa design more casual").

[1081] A modification request is sent to the server.

[1082] Step 7:

[1083] The server receives the user's revision request and re-prompts the generative AI model to generate a new interior proposal. The input data includes the user's revision request. The output of this step is the newly generated interior proposal.

[1084] Specific behavior:

[1085] The server sends a modification request to the generative AI model.

[1086] A generative AI model generates new interior proposals.

[1087] The new proposal is sent to the server.

[1088] Step 8:

[1089] Once the user is satisfied with the proposal, the user device activates the camera and uses augmented reality to virtually place the proposed interior items in the actual room. The input data includes the generated interior proposal. The output of this step is for the user to confirm the virtual placement results.

[1090] Specific behavior:

[1091] The user activates the smartphone camera.

[1092] Use augmented reality to virtually place suggested items in a room.

[1093] Check the placement on your smartphone screen.

[1094] Step 9:

[1095] If the user wishes to purchase the proposed item, he / she selects the purchase option on the user's device. The server receives this information and cooperates with the online shop to complete the purchase process. The input data includes the user's intention to purchase. The output of this step is the completion of the purchase process.

[1096] Specific behavior:

[1097] The user presses the purchase button.

[1098] The server connects with the online shop and completes the purchase process.

[1099] Step 10:

[1100] If the user needs to dispose of unwanted furniture, they select the selling option. The server receives this information and cooperates with platforms such as auction sites to complete the selling process. The input data includes the user's intention to sell. The output of this step is the completion of the selling process.

[1101] Specific behavior:

[1102] The user presses the sell button.

[1103] The server connects with the auction site and completes the sales process.

[1104] As described above, in the present invention, each process is performed sequentially, thereby supporting the user in achieving an ideal interior coordination without much effort.

[1105] (Application example 1)

[1106] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1107] Conventional interior coordination systems have had problems such as difficulty in efficiently providing proposals that reflect the user's needs and limited means for users to see how the proposals would actually look in a room, resulting in low user satisfaction. Another problem is that the procedures for purchasing or selling the proposed interior items online are cumbersome. Furthermore, there has been no system that allows users to view the generated interior proposals in a virtual space and then smoothly proceed to purchase them. The objective of the present invention is to solve these problems.

[1108] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1109] In this invention, the server includes a generative modeling unit for interactively extracting user requirements, a generative modeling unit for generating interior design proposals based on the extracted requirements, a unit for transmitting the generated interior design proposals to a user terminal, a unit for displaying the generated interior design proposals on the user terminal using an AR function, a linking unit for purchasing or selling the proposed interior design items online, and a unit for viewing the generated interior design proposals in a virtual space and proceeding directly to a purchase. This allows users to quickly and accurately receive interior design proposals based on their detailed requirements, virtually arrange and view the proposals in an actual room, and smoothly complete the online purchase or sale process. Additionally, the ability to view the generated interior design proposals in a virtual space and proceed directly to a purchase significantly improves user satisfaction and convenience.

[1110] The "generative model means for interactively extracting user needs" is a device that includes artificial intelligence for analyzing information input by a user interactively and extracting those needs.

[1111] The "generative model means for generating interior proposals" is a device that includes artificial intelligence for generating optimal interior proposals based on extracted user requests.

[1112] The "means for transmitting the generated interior proposal to the user terminal" is a communication means for transmitting the interior proposal generated by the server to the user's device.

[1113] "Means for displaying interior proposals generated by a user's device using AR functionality" refers to technology for displaying interior proposals generated on a user's device using augmented reality functionality.

[1114] The "connection means for purchasing or selling the proposed interior items online" refers to a means for connecting with an online platform to simplify the process of purchasing or selling interior items online.

[1115] "A means to check the generated interior design proposal in a virtual space and proceed directly to purchase" refers to technology that allows the generated interior design proposal to be checked in real time in a virtual environment and the purchase process to be completed online.

[1116] The system for realizing this invention provides a series of steps to extract user needs interactively, generate interior design proposals based on those needs, confirm those proposals in a virtual space, and purchase or sell the most suitable interior items online. Each element will be explained in detail below.

[1117] The server is equipped with a generative AI model and an interactive question-and-answer algorithm. In particular, the generative AI model analyzes and extracts the user's requests and generates interior proposals based on them. The generated interior proposals are sent to the user's device, where the user can review them. In addition, if the user requests revisions to the proposal, the generative AI model will generate a new proposal again.

[1118] The user device may be a smartphone or tablet. The user device receives the interior design suggestions sent from the server and uses AR (Augmented Reality) functionality to virtually display them in the actual room environment. This allows the user to see in real time how the proposed interior items will look. AR technology such as ARKit for iOS or ARCore for Android is used.

[1119] Users can purchase items they like online or sell unwanted items. The server works with online shops and auction sites to facilitate these procedures.

[1120] As a concrete example, consider a user who wants to decorate the living room of their new home. The user accesses the system and inputs detailed requirements through the following interface:

[1121] Room details:

[1122] Room type: Living room

[1123] Area: 20 square meters

[1124] Design Desire:

[1125] Style: Nordic style

[1126] Color: Blue and White

[1127] Budget: 100,000 yen

[1128] Based on this request, the server uses a generative AI model to generate suggestions such as a blue and white Scandinavian sofa, a wooden table, and a simple floor lamp, and sends them to the user's device. The user can then virtually place these suggestions in their actual room using the AR function to see how the items will look.

[1129] An example prompt might be used as follows:

[1130] Please suggest interior coordination based on the user's request: The living room is 20 square meters, and the design is Scandinavian-style, with a budget of 100,000 yen. The main colors are blue and white.

[1131] In this way, the system of the present invention allows the user to achieve ideal interior coordination without much effort, and the process can be carried out simply and quickly.

[1132] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1133] Step 1: User request input

[1134] Users access the interface via a smartphone or tablet and enter detailed requirements such as the type of room, layout, size, desired design, preferred colors, budget, etc. Specific input data might include "living room, 20 square meters, Scandinavian design, blue and white, 100,000 yen."

[1135] Step 2: Extracting desired data

[1136] The server receives the input data sent by the user and uses a generative AI model to analyze and extract the user's requirements interactively. During this process, the user's input data is analyzed and processed to extract specific keywords (room type, size, desired design, etc.). The extracted data is then passed on to the next step.

[1137] Step 3: Generate interior proposals

[1138] The server uses a generative AI model to generate interior design proposals based on the extracted request data. A prompt statement is input into the generative model, and multiple interior design proposals are generated based on that. For example, a "blue and white Scandinavian-style sofa, wooden table, and simple floor lamp" is generated. This proposal data is passed on to the next step.

[1139] Step 4: Submit your interior design proposal

[1140] The generated interior design proposals are sent from the server to the user's device. Here, the proposal data (e.g., "a blue and white Scandinavian-style sofa, a wooden table, and a simple floor lamp") is sent to the user's smartphone or tablet. The user can then check the proposals on the screen.

[1141] Step 5: User correction request

[1142] The user can input their opinions and requests for modifications to the displayed interior design proposal. For example, they can send a specific request for modifications such as "I want the sofa design to be more casual." The user's modification request data is passed to the next step.

[1143] Step 6: Regenerate correction suggestions

[1144] The server receives the modification request and uses the generative AI model to regenerate a new interior proposal. In this process, a new prompt sentence is generated, and a modified interior proposal is generated based on it. For example, "a sofa with a casual design." This modified proposal data is passed to the next step.

[1145] Step 7: Final confirmation using AR display

[1146] The user device receives the regenerated interior proposals and virtually places them in the actual room using the smartphone's camera and AR function. The user can check in real time on the device screen how the interior items will look in each position. AR technology (such as ARKit for iOS or ARCore for Android) is used here.

[1147] Step 8: Buy or sell online

[1148] Finally, if the user is satisfied with the suggestions, they can choose to purchase online or sell the unwanted items. The server automatically processes the purchase and sale procedures by connecting with online shops and auction sites. Specifically, when the user presses the purchase or sale button, information about the suggested interior items is sent to the connected platform, completing the process.

[1149] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1150] This invention is a system that recognizes users' emotions and makes optimal interior design suggestions based on that information. The system combines generative AI models, AR technology, and an emotion engine to provide sophisticated interior design suggestions and actual layout images based on user requests, and also supports online buying and selling.

[1151] The usage flow of this system is as follows: First, the user accesses the coordination support system using a smartphone or PC. The user starts the dialogue mode and enters, "I'd like you to coordinate my new living room."

[1152] Next, the user device displays an interface for entering detailed requirements. The user enters information such as floor plan, size, style, color, and budget. For example, a specific requirement might be entered as "living room size 20 square meters, style Scandinavian, budget 100,000 yen, color scheme blue and white."

[1153] In addition, the emotion engine analyzes the user's emotions from their voice and facial expressions. When a user inputs a request, facial and voice data is captured through the camera and microphone on the user's device, and the emotion engine analyzes this. After the emotion data is analyzed, it is sent to the server, and a request is generated that takes the user's emotions into account.

[1154] The server then receives this request data and uses a generative AI model to generate interior design suggestions, such as a blue and white Scandinavian-style sofa, a wooden low table, and a simple floor lamp.

[1155] The generated suggestions are sent to the device and displayed to the user. The user reviews the suggestions and may input requests for revisions, such as "make the sofa design more casual." If a revision is requested, the device sends it back to the server, and the generative AI model generates a new suggestion. The server then refers to the user's emotional data again to make the optimal suggestion.

[1156] The process is repeated to ensure the proposal satisfies the user. Once the proposal is finalized, the user's device activates the camera and switches to AR mode. The user then views the room through their smartphone and sees how the proposed furniture will actually look when arranged. The AR function displays a virtual image of the interior items, allowing the user to view the entire room.

[1157] If the user checks the final proposal and likes the item, they select the purchase option on their device. The server connects to the online shop and completes the purchase procedure for the selected item. Purchase confirmation and shipping information are sent to the user's device. Furthermore, if the user selects the sell option for unwanted furniture, the server will link with an online platform such as an auction site and complete the selling procedure.

[1158] As a concrete example, consider the case of coordinating the living room of a new home. The user inputs detailed requirements into the system, and the emotion engine detects that the user is relaxed. Based on this information, the generative AI model suggests a "Nordic-style blue and white sofa with an emphasis on relaxation, a wooden low table, and a floor lamp with a simple design." The user submits a request for revisions, and the emotion engine confirms that the user is satisfied with the changes. Once the proposal is finalized, the user can visually check the actual layout using AR functionality, and finally purchase the items online. Unwanted old furniture can also be sold in the same way.

[1159] In this way, the system of the present invention takes into consideration the user's feelings and efficiently realizes the ideal interior coordination, and also supports simple and quick purchase and sale procedures.

[1160] The processing flow will be explained below.

[1161] Step 1:

[1162] The user accesses the coordination support system using a smartphone or computer. The user activates the dialogue mode and inputs, "Please coordinate my new living room."

[1163] Step 2:

[1164] The device receives the user's input and displays questions for entering detailed requirements. The user enters information such as floor plan, size, style, color, and budget. For example, the user might enter, "Living room size: 20 square meters, style: Scandinavian, budget: 100,000 yen, color: blue and white."

[1165] Step 3:

[1166] The device's camera and microphone are used to input the user's facial expressions and voice into the emotion engine, which then analyzes this data to determine the user's emotions, such as whether they are relaxed or excited. The analysis results are temporarily stored on the device.

[1167] Step 4:

[1168] The device sends the user's desires and emotional data to the server, which then activates a generative AI model to generate multiple interior design suggestions based on the user's desires and emotions. For example, the server suggests a "blue and white Scandinavian-style sofa, a wooden low table, and a simple floor lamp, all of which emphasize relaxation."

[1169] Step 5:

[1170] The server sends the generated interior design proposal to the terminal, which displays the proposal to the user. The user checks the proposal and inputs a request for modification, such as "make the sofa design more casual."

[1171] Step 6:

[1172] The device then sends the correction request to the server again. The server then communicates the correction request to the generative AI model, which then regenerates a new proposal. The generative AI model then refers to the user's emotional data again to make the optimal proposal.

[1173] Step 7:

[1174] The server sends the regenerated interior proposal to the device, which displays the re-proposed content to the user, who then reviews the proposal and makes a final decision.

[1175] Step 8:

[1176] The device will activate its camera and switch to AR mode. The user can then use the camera to view the room and virtually see how the proposed furniture will be arranged in the room. For example, they can see how a sofa or table will be arranged on their smartphone screen.

[1177] Step 9:

[1178] If the user likes the proposed furniture, they click the "Purchase" button on their device. The server connects to the online shop and completes the purchase process. Purchase confirmation and shipping information are sent to the user's device.

[1179] Step 10:

[1180] When a user wants to dispose of unwanted furniture, they click the "Sell" button on their device. The server connects to an online platform (e.g., an auction site) and completes the sale procedure. The sale information is then sent to the user's device.

[1181] Through this series of steps, users can interactively communicate their needs to the system, taking their own feelings into consideration, and efficiently receive interior design suggestions while purchasing the most suitable items or selling unwanted items.

[1182] Example 2

[1183] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1184] Modern interior coordination requires a great deal of time and effort to create a space that matches the user's emotions and preferences. It is also difficult for users to accurately convey their emotionally-based latent needs when expressing specific requests. The process of visualizing and revising proposals, as well as online purchasing and selling, can be complex and difficult to understand. To solve these problems, a comprehensive system is needed that considers the user's emotions, efficiently proposes interiors, and ultimately creates a comfortable and ideal space.

[1185] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes a data generation means for interactively extracting user desires, a generative model means for generating interior design proposals based on the extracted desires, a sentiment analysis means for analyzing the user's emotions, a means for the generative model to generate proposals taking the sentiment analysis data into consideration, a means for transmitting the generated interior design proposals to a user terminal, a means for the user terminal to display the generated interior design proposals using an augmented reality function, and a linking means for purchasing or selling the proposed interior design items online. This enables highly accurate interior design proposals that take the user's emotions into consideration, and by visually confirming and correcting them, a highly satisfying coordination can be quickly and efficiently achieved.

[1186] The "data generating means" has a function for extracting user requests in an interactive format.

[1187] The "generative model means" refers to a model used to generate interior proposals based on the extracted needs.

[1188] The "emotion analysis means" has a function for analyzing the user's emotions from facial expressions, voice, and the like.

[1189] The "proposal generation means" has the function of creating optimal interior proposals using a generative model while taking into account emotion analysis data.

[1190] The "data transmission means" has a function for transmitting the generated interior proposal to the user terminal.

[1191] The "augmented reality function" is a function that allows interior design proposals generated on a user's device to be virtually displayed in real space.

[1192] "Linkage means" means a function that links with other systems or platforms to purchase or sell the proposed interior items online.

[1193] The present invention is a system that recognizes a user's emotions and makes optimal interior design suggestions based on that information. This system is realized by combining a generative AI model, AR technology, and an emotion analysis engine, and provides advanced interior design suggestions and actual layout images based on the user's requests, and also supports online purchasing and selling. Specific embodiments of the present invention are described in detail below.

[1194] First, the user accesses the coordination support system using a device such as a smartphone or PC. The user activates the dialogue mode and inputs, "I'd like you to coordinate my new living room." This causes the user device to display an interface for inputting detailed requests. The user inputs information such as floor plan, size, style, color, and budget. For example, the user might input specific requests such as, "Living room size: 20 square meters, style: Scandinavian, budget: 100,000 yen, color: blue and white."

[1195] Next, the user device collects the user's facial expressions and tone of voice through a camera and microphone, which are then analyzed by an emotion analysis engine. The emotion analysis engine determines the user's emotional state, such as whether they are relaxed or excited. For example, if the engine detects that the user is relaxed when entering their request, that information is collected.

[1196] The device then sends the user's detailed information and analyzed emotion data to a server. The server then uses a generative AI model (e.g., a general generative model) to generate optimal interior design suggestions based on the user's desires and emotions. Specifically, the server generates suggestions such as a blue and white Scandinavian-style sofa, a wooden low table, and a simple floor lamp.

[1197] The generated suggestions are sent to the device and displayed to the user. The user reviews the suggestions and may input requests for revisions, such as "make the sofa design more casual." If a revision is requested, the device sends it back to the server, and the generative AI model generates a new suggestion. The server then refers to the user's emotional data again to make the optimal suggestion.

[1198] The process is repeated to ensure the proposal is satisfactory to the user. Once the proposal is finalized, the user's device activates the camera and switches to AR mode. The user then uses the device to view the entire room and see the virtual arrangement of the proposed furniture. The AR function displays a virtual image of the interior items arranged, allowing the user to view the entire room.

[1199] If the user checks the final proposal and likes the item, they can select the purchase option on their device. The server connects to the online shop and completes the purchase process for the selected item. Purchase confirmation and shipping information are sent to the user's device. Additionally, if the user selects the sell option for unwanted furniture, the server can link with an online platform such as an auction site to complete the selling process.

[1200] Examples of prompts include the following:

[1201] "I'd like the living room to be 20 square meters in a Scandinavian style. The color scheme should be blue and white, and the budget should be within 100,000 yen. I'd like it to have a relaxing feel."

[1202] "Please make the sofa design a little more casual."

[1203] In this way, the system of the present invention efficiently provides ideal interior coordination while taking into consideration the user's feelings, and also allows for simple and quick purchase and sale procedures.

[1204] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1205] Step 1:

[1206] A user accesses the system using a terminal. The user activates the interactive mode and inputs, "I'd like you to coordinate a new living room." This causes the terminal to display an interface for inputting detailed requests.

[1207] Input: User access requests and command input (e.g., "Please help me coordinate my new living room").

[1208] Output: Display of the coordination support interface.

[1209] Specific operation: When a user opens the system's web page or app on a smartphone or PC, the device displays the login screen and the user enters the login information. The user activates the interactive mode, and the system displays the request input screen.

[1210] Step 2:

[1211] The user inputs details of their coordination needs, and the user's device receives and records information such as floor plan, size, taste, color, and budget through a form on the interface.

[1212] Input: User-entered data (e.g., "Living room is 20 sq m, style is Scandinavian, budget is 100,000 yen, colors are blue and white").

[1213] Output: Detailed coordination request data.

[1214] Specific operation: The user enters their request into the text box on the interface, and the terminal receives it in real time and displays a confirmation message.

[1215] Step 3:

[1216] The user's device collects the user's facial expressions and voice through a camera and microphone, which are then analyzed by an emotion analysis engine.

[1217] Input: User's facial expression and voice data.

[1218] Output: Parsed emotion data (e.g. "relaxed").

[1219] How it works: The device's camera captures the user's face and microphone records their voice. The emotion analysis engine analyzes this in real time to determine whether the user is relaxed, etc.

[1220] Step 4:

[1221] The device sends the user's detailed information and emotional data to the server, which then uses a generative AI model to generate interior design proposals based on the user's requests and emotional data.

[1222] Input: User details and parsed emotion data.

[1223] Output: Initial interior design proposal (e.g., "A blue and white Scandinavian-style sofa, a wooden low table, and a simple floor lamp").

[1224] How it works: When the device sends data to the server, the server uses a generative AI model to generate interior design suggestions, such as a "Scandinavian-style sofa, a wooden low table, and a simple floor lamp."

[1225] Step 5:

[1226] The server sends the generated interior design proposal to the user's device, which displays the proposal and accepts user requests for revisions.

[1227] Input: Interior design proposal sent from the server.

[1228] Output: Interior proposals and modification requests displayed on the user's screen.

[1229] Specific operation: The server sends the generated proposal to the device, and the device displays "a blue and white Scandinavian-style sofa, a wooden low table, and a simple floor lamp." The user then inputs a modification request, such as "Make the sofa design more casual."

[1230] Step 6:

[1231] The user device then sends a correction request to the server again, and the generative AI model generates a new proposal. The server also references the user's emotional data again to make the optimal proposal.

[1232] Input: User modification request data.

[1233] Output: Revised interior proposal.

[1234] Specific operation: When a revision request is sent to the server, the generative AI model again generates interior design suggestions and creates revision suggestions, such as a new sofa design, which are then sent back to the device.

[1235] Step 7:

[1236] Once the proposal is finalized, the user's device activates the camera and switches to augmented reality mode, allowing the user to view the entire room using their smartphone and see how the proposed furniture will be virtually arranged.

[1237] Input: Confirmed interior proposal.

[1238] Output: Virtual interior layout display in augmented reality mode.

[1239] Specific operation: The user's device activates the camera and uses the AR function to display a screen showing a virtual room with furniture arranged in it. The user can then view the image of the entire room, including a blue and white Scandinavian-style sofa, through their smartphone.

[1240] Step 8:

[1241] If the user likes the suggestions and selects a purchase option, the server connects to the online shop and completes the purchase process for the selected item. Additionally, if the user selects the option to sell unwanted furniture, the server will connect to an online platform such as an auction site and complete the sales process.

[1242] Input: User's buy and sell selections.

[1243] Output: Purchase confirmation, shipping information, sale process.

[1244] Specific operation: When the user clicks the purchase button on the device, the selected item is ordered in cooperation with the online shop. The server sends a purchase confirmation and shipping information to the device. If the user selects the sell option, the server cooperates with the auction site to assist in the sale process.

[1245] (Application example 2)

[1246] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1247] Conventional interior design suggestion systems are unable to consider user emotions, making it difficult to increase user satisfaction. Additionally, interior design suggestions in physical stores lack real-time feedback functions to improve the user experience.

[1248] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1249] In this invention, the server includes a generative model means for interactively extracting user desires, a generative model means for generating interior design proposals based on the extracted desires, a means for transmitting the generated interior design proposals to a user terminal, a means for the user terminal to display the generated interior design proposals using an AR function, a linking means for purchasing or selling the proposed interior design items online, an emotion analysis means for recognizing the emotions of customers, and a generative model means for generating interior design proposals based on the emotion analysis. This makes it possible to make interior design proposals that take user emotions into consideration, and to make proposals that meet user needs in real time even in physical stores.

[1250] A "generative model" is an algorithm or process for generating interior design suggestions based on user input data and emotion data.

[1251] "Emotion analysis means" refers to technology or a system that analyzes a user's facial expressions and voice and recognizes their emotional state.

[1252] A "user terminal" is a hardware device that a user uses to access the system and receive interior design suggestions, and specifically includes a smartphone, tablet, smart glasses, etc.

[1253] "AR function" refers to the ability to display virtual interior items in real space using augmented reality technology.

[1254] "Online purchasing or selling collaboration" refers to a system or service that supports the process of a user purchasing or selling suggested interior items online.

[1255] This invention is a system that recognizes a user's emotions and proposes optimal interior design based on that information. Specifically, it is realized by combining emotion analysis means, generative model means, AR functionality, and linking means for online purchases or sales.

[1256] First, when a user visits a physical store, the emotion analysis means recognizes and analyzes the user's facial expressions and voice in real time to obtain emotional data. Emotion analysis uses a general emotion recognition algorithm, such as facial recognition technology or voice analysis technology.

[1257] Next, the generative model means interactively extracts the user's requirements. Through the interactive interface, the user inputs detailed information such as the layout, size, taste, preferred color, and budget. In addition, emotion data obtained by the emotion analysis means is also passed to the generative model means.

[1258] The generative modeling method generates interior design suggestions based on the user's input data and emotional data. For example, it might suggest a "relaxing Scandinavian-style blue and white sofa, a wooden low table, and a simple floor lamp." The generative AI model is used to design an interior design that best suits the user's needs.

[1259] The generated interior design suggestions are sent to the user's device, such as a tablet or smart glasses. The user can then use the device's AR functionality to virtually place and visualize the generated interior design suggestions in their actual space. Through this visualization, the user can check how well the suggested items fit into their own space.

[1260] If the user is satisfied with the proposal, they can proceed to the online purchase procedure on the spot. The purchase request sent from the terminal is linked to the online shop, where the final purchase procedure is carried out. In addition, for unwanted interior items, the user can select the option to sell them, and the selling procedure will be carried out in conjunction with the online platform.

[1261] As a concrete example, consider a user requesting coordination of the living room of their new home. In this case, the user inputs detailed requests into the system, and the emotion analysis means detects that the user is relaxed. Based on this information, the generative AI model suggests a "Nordic-style blue and white sofa that emphasizes relaxation, a wooden low table, and a floor lamp with a simple design." The user then submits a request for revisions, and the emotion analysis means confirms that the user is satisfied with the changes. Once the proposal is finalized, the user can visually check the actual layout using the AR function, and finally purchase the items online. Unwanted old furniture can also be sold in the same way.

[1262] Example prompt sentence:

[1263] "Our customers want to relax. Please suggest a Scandinavian-style blue and white sofa for the living room, which will create a relaxing atmosphere."

[1264] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1265] Step 1:

[1266] When a user visits a store, the device (smart glasses or tablet) captures their facial expressions and voice. The input data is collected in real time through a camera and microphone and sent to the emotion analysis means. The emotion analysis means uses facial expression recognition algorithms and voice analysis algorithms to analyze and identify the user's emotions as data. The output is the user's emotional state (e.g., "relaxed").

[1267] Step 2:

[1268] Next, the user inputs detailed information about the room for which they would like interior suggestions (layout, size, taste, preferred colors, budget, etc.) through their device. The data the user inputs into the dialogue interface on the tablet or smart glasses is sent to the generative model means. The input data is passed to the generative AI model, which analyzes the user's request and constructs a prompt sentence. The output is a specific prompt sentence that reflects the user's wishes.

[1269] Step 3:

[1270] The generative model means combines the emotion data obtained in step 1 with the user desire data obtained in step 2. The generative AI model generates interior design proposals based on this data. Using data mining and pattern recognition techniques, it designs an interior design plan that best suits the user's desires and emotions. The output is a specific interior design proposal (for example, "A relaxing Scandinavian-style blue and white sofa, a wooden low table, and a simply designed floor lamp").

[1271] Step 4:

[1272] The generated interior proposal is sent from the server to the device. The user can check the proposal on the device's display. At this point, the user can request revisions to the proposal. If revisions are requested, the proposal is sent again from the device to the server, and a new proposal is generated by the generative model means.

[1273] Step 5:

[1274] After the final proposal is finalized, the user's device uses its AR function to display a visual image of the interior items virtually arranged in the real store. The user uses the device's camera to capture the real store space and overlay the virtual interior on it. The output is an image of the virtual interior arrangement in the real space.

[1275] Step 6:

[1276] If the user is satisfied with the proposed interior items, the terminal supports the online purchase process. When the user presses the purchase button, a purchase request is sent to the online shop via the server. After the purchase process is completed, shipping information is sent to the user terminal. The output is a purchase confirmation and shipping information.

[1277] Step 7:

[1278] If a user wants to sell unwanted interior items, the user device supports the selling procedure. When the user selects the selling option, the server will connect to an online platform such as an auction site to complete the selling procedure. The output is a confirmation of the sale and detailed information.

[1279] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1280] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1281] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1282] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1283] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1284] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1285] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1286] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1287] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1288] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1289] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1290] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1291] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1292] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1293] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1294] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1295] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1296] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1297] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1298] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1299] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1300] The following is further disclosed regarding the above embodiment.

[1301] (Claim 1)

[1302] a generative model means for interactively extracting user requests;

[1303] a generative model means for generating interior proposals based on the extracted needs;

[1304] means for transmitting the generated interior proposal to a user terminal;

[1305] a means for displaying the generated interior proposal by using an AR function on the user terminal;

[1306] A means of collaboration for online purchase or sale of the proposed interior items;

[1307] A system including:

[1308] (Claim 2)

[1309] 2. The system according to claim 1, wherein the interactive requirement extraction means analyzes data input by the user and specifies the layout, size, taste, preferred color and budget.

[1310] (Claim 3)

[1311] 2. The system according to claim 1, further comprising generative model means for accepting a user's request for modification of the generated interior proposal and regenerating the proposal.

[1312] (Claim 4)

[1313] 2. The system according to claim 1, wherein the user terminal includes an AR display means for displaying an image of the proposed interior layout superimposed on the actual room using a camera.

[1314] (Claim 5)

[1315] 2. The system according to claim 1, further comprising an online purchasing means for carrying out a purchasing procedure for the interior item selected by the user in cooperation with an online shop.

[1316] (Claim 6)

[1317] 10. The system of claim 1, further comprising an online selling means for selling unwanted interior items via an online platform.

[1318] "Example 1"

[1319] (Claim 1)

[1320] a generative model means for interactively extracting user requests;

[1321] a generative model means for generating interior proposals based on the extracted needs;

[1322] means for transmitting the generated interior proposal to a user terminal;

[1323] a means for displaying the generated interior proposal by using an augmented reality function in a user terminal;

[1324] A means of collaboration for online purchase or sale of the proposed interior items;

[1325] A system including:

[1326] (Claim 2)

[1327] 2. The system according to claim 1, wherein the interactive requirement extraction means analyzes data input by the user and specifies the layout, size, design style, preferred color, and budget.

[1328] (Claim 3)

[1329] 2. The system according to claim 1, further comprising generative model means for accepting a user's request for modification of the generated interior proposal and regenerating the proposal.

[1330] "Application Example 1"

[1331] (Claim 1)

[1332] a generative model means for interactively extracting user requests;

[1333] a generative model means for generating interior proposals based on the extracted needs;

[1334] means for transmitting the generated interior proposal to a user terminal;

[1335] a means for displaying the generated interior proposal by using an AR function on the user terminal;

[1336] A means of collaboration for online purchase or sale of the proposed interior items;

[1337] A way to check the generated interior design proposals in a virtual space and proceed to purchase them.

[1338] A system including:

[1339] (Claim 2)

[1340] 2. The system according to claim 1, wherein the interactive requirement extraction means analyzes data input by the user and specifies the layout, size, taste, preferred color and budget.

[1341] (Claim 3)

[1342] 2. The system according to claim 1, further comprising generative model means for accepting a user's request for modification of the generated interior proposal and regenerating the proposal.

[1343] "Example 2: Combining Emotion Engines"

[1344] (Claim 1)

[1345] data generation means for interactively extracting user requests;

[1346] a generative model means for generating interior proposals based on the extracted needs;

[1347] emotion analysis means for analyzing the emotions of a user;

[1348] a means for the generative model to generate suggestions taking into account the sentiment analysis data;

[1349] means for transmitting the generated interior proposal to a user terminal;

[1350] a means for displaying the generated interior proposal by using an augmented reality function in a user terminal;

[1351] A means of collaboration for online purchase or sale of the proposed interior items;

[1352] A system including:

[1353] (Claim 2)

[1354] 2. The system according to claim 1, wherein the interactive requirement extraction means analyzes data input by the user and specifies the layout, size, taste, preferred color and budget.

[1355] (Claim 3)

[1356] 2. The system according to claim 1, further comprising generative model means for accepting a user's request for modification of the generated interior proposal and regenerating the proposal.

[1357] "Application example 2 when combining emotion engines"

[1358] (Claim 1)

[1359] a generative model means for interactively extracting user requests;

[1360] a generative model means for generating interior proposals based on the extracted needs;

[1361] means for transmitting the generated interior proposal to a user terminal;

[1362] a means for displaying the generated interior proposal by using an AR function on the user terminal;

[1363] A means of collaboration for online purchase or sale of the proposed interior items;

[1364] An emotion analysis means for recognizing the emotions of customers;

[1365] a generative model means for generating interior design proposals based on emotion analysis;

[1366] A system including:

[1367] (Claim 2)

[1368] 2. The system according to claim 1, wherein the interactive requirement extraction means analyzes data input by the user and specifies the layout, size, taste, preferred color and budget.

[1369] (Claim 3)

[1370] 2. The system according to claim 1, further comprising generative model means for accepting a user's request for modification of the generated interior proposal and regenerating the proposal. [Explanation of symbols]

[1371] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. a generative model means for interactively extracting user requests; a generative model means for generating interior proposals based on the extracted needs; means for transmitting the generated interior proposal to a user terminal; a means for displaying the generated interior proposal by using an AR function on the user terminal; A means of collaboration for online purchase or sale of the proposed interior items; A system including:

2. 2. The system according to claim 1, wherein the interactive requirement extraction means analyzes data input by the user and specifies the layout, size, taste, preferred color and budget.

3. The system according to claim 1, further comprising generative model means for accepting a user's request for modification of the generated interior proposal and regenerating the proposal.

4. The system according to claim 1, wherein the user terminal includes an AR display means for displaying an image of the proposed interior layout superimposed on an actual room using a camera.

5. 2. The system according to claim 1, further comprising online purchasing means for carrying out a purchasing procedure for the interior items selected by the user in cooperation with an online shop.

6. The system according to claim 1, further comprising an online selling means for selling unwanted interior items via an online platform.

Images