system

Abstract

We provide the system. [Solution] A means for the user to take a photo of the interior using a mobile device and send the photo data to a server, The aforementioned server analyzes the received photo data and provides a means for suggesting similar interior design ideas. The aforementioned server has a means of searching for and displaying in a list appropriate furniture and interior items from multiple online shopping sites based on the proposed coordination. A means by which the aforementioned user can purchase online the product they selected from the displayed list of items, A system that includes this.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor, the method including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a character of the chatbot, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance as a response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] Modern consumers want to easily achieve interior coordination according to their preferences and budgets. However, to obtain inspiration, they need to search through multiple magazines and websites, and then spend time searching for similar products individually on each shopping site. Therefore, consumers will spend a lot of time and effort. The purpose of the present invention is to solve this problem by providing a system that allows users to easily achieve interior coordination and even proceed to purchase immediately.

Means for Solving the Problems

[0005] This invention provides a system that includes means for a user to take photos of an interior using a mobile device and send the photo data to a server, and means for the server to analyze the received photo data and suggest similar interior coordination ideas. Furthermore, the server has means for searching for appropriate furniture and interior items from multiple online shopping sites based on the suggested coordination ideas and displaying them in a list. The system also provides means for the user to purchase selected items online, enabling the user to quickly and easily realize interior coordination and make purchases on the spot. The server also provides means for calculating and monetizing a coordination fee based on the purchase amount. This reduces the burden on the user and makes it possible to provide an efficient shopping experience.

[0006] A "mobile device" is an electronic device that a user can carry with them, and includes, but is not limited to, smartphones and tablets that have communication and photography functions.

[0007] "Photo data" refers to image information of interiors that users take with their mobile devices and upload to the server.

[0008] A "server" is a central processing system that receives data transmitted from terminals via a network, processes and stores it, and sends the results back to the terminals as needed.

[0009] "Interior coordination" refers to the design and planning of a room's aesthetically pleasing and functional arrangement, combining furniture, decorative items, lighting, and other elements.

[0010] An "online shopping site" is a website that sells products over the internet and serves as a platform where users can complete the purchase process.

[0011] "Furniture" refers to items such as tables, chairs, sofas, and beds that are used to make living spaces more comfortable.

[0012] "Interior items" refer to decorative items and accessories such as art, carpets, cushions, and lighting used to decorate or improve the functionality of a room.

[0013] "List view" refers to a format where the server displays multiple products or items together on the screen to make it easier for the user to select them.

[0014] "Purchase process" refers to the process by which a user selects a product, completes the payment process through an online shopping site, and confirms the purchase.

[0015] "Coordination fee" refers to a portion of the revenue that the server collects for providing coordination suggestions and services when a user purchases a product. [Brief explanation of the drawing]

[0016] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] Shows an emotion map to which a plurality of emotions are mapped. [Figure 10] Shows an emotion map to which a plurality of emotions are mapped. [Figure 11] It is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] It is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] It is a sequence diagram showing the processing flow of the data processing system in Example 2 when an emotion engine is combined. [Figure 14] It is a sequence diagram showing the processing flow of the data processing system in Application Example 2 when an emotion engine is combined.

Mode for Carrying Out the Invention

[0017] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0018] First, the language used in the following description will be explained.

[0019] In the following embodiments, the numbered processor (hereinafter simply referred to as "processor") may be one arithmetic unit or a combination of a plurality of arithmetic units. Also, the processor may be one type of arithmetic unit or a combination of a plurality of types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0020] In the following embodiments, signed RAM (Random Access Memory) is a memory that temporarily stores information and is used as work memory by the processor.

[0021] In the following embodiments, the signed storage is one or more non-volatile storage devices that store various programs and various parameters. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes.

[0022] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0023] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0024] [First Embodiment]

[0025] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0026] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0027] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0028] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0029] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0030] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0031] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0032] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0033] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0034] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0035] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0036] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0037] The system for implementing this invention begins with the user taking a photo of the interior using a mobile device and sending the photo data to a server. Based on this photo data, the server suggests interior design ideas and then searches for and displays appropriate furniture and interior items from multiple online shopping sites. The user can then select items from the displayed list and purchase them on the spot. Each process is as follows:

[0038] User actions

[0039] The user first uses the camera function of their smartphone or other mobile device to take pictures of interior design pages in magazines or of their own home interior. Then, they use the app to upload the photos they have taken. For example, a user might take a picture of their living room and press the "Upload" button in the app.

[0040] Terminal operation

[0041] When the user presses the upload button, the device sends the selected photo data to the server. The photo data is uploaded to a cloud server via the internet.

[0042] Server analysis

[0043] The server passes the received photo data to an AI algorithm for analysis. The AI ​​algorithm extracts features such as color, shape, and arrangement from the photo and matches them with interior styles stored in a database. This allows the server to generate interior design ideas similar to the photo data. For example, based on the photo analysis, it might suggest a modern style interior.

[0044] Searching online shopping sites

[0045] The server searches for relevant furniture and interior items from multiple online shopping sites based on the proposed coordination. Criteria such as color, size, and price are used. The search results are filtered to select only the items best suited to the user.

[0046] List of items

[0047] The server organizes the search results into a list format and sends it to the user's device. The user can then select the items they wish to purchase from this list. For example, a sofa, table, and carpet that fit the suggested modern style might be displayed.

[0048] Purchase procedure

[0049] Once the user selects an item, the device proceeds with the purchase process via the online shopping site. The user can then enter their payment information and confirm the purchase.

[0050] Monetizing coordination fees

[0051] The server confirms that the purchase of the product is complete and calculates a coordination fee based on the purchase amount. For example, if the purchase amount is 100,000 yen, a fee of 1,000 yen will be generated as revenue.

[0052] This invention provides a system that allows users to easily realize interior design coordination and purchase items on the spot. Furthermore, the server can accumulate data based on the user's purchasing behavior, enabling it to further optimize future coordination suggestions.

[0053] The following describes the processing flow.

[0054] Step 1:

[0055] Users use their smartphones to take photos of interior design pages in magazines or of their own home interiors. This allows users to create photo data that serves as the basis for their interior design ideas.

[0056] Step 2:

[0057] The user taps the "Upload" button within the app, selects the photos they have taken, and sends them to the server. The device accepts this action and uploads the photo data to the server via the internet.

[0058] Step 3:

[0059] The server receives and saves the photo data, then passes it to the AI ​​module. The AI ​​module extracts features such as color, shape, and arrangement from the photo, and performs analysis to generate similar interior design ideas based on that data.

[0060] Step 4:

[0061] The server suggests similar interior styles based on the AI ​​analysis results. The suggestions are sent to the user's device, and the user can view the suggested coordination within the app.

[0062] Step 5:

[0063] The server uses the characteristics of the proposed coordination to call APIs from multiple online shopping sites to search for suitable furniture and interior items. This is done using specified criteria such as color, size, and price.

[0064] Step 6:

[0065] The server filters search results obtained from online shopping sites and selects the most suitable items. This selection is based on which items best fit the suggested outfit.

[0066] Step 7:

[0067] The server generates a list of selected items and sends this information to the user's device. The user can then view this list in the app and choose the items they wish to purchase.

[0068] Step 8:

[0069] The user selects the item they want to purchase from the list and taps the purchase button within the app. The device accepts this action and proceeds with the purchase process using the online shopping site's API.

[0070] Step 9:

[0071] The user enters their payment information and taps the purchase button. The device sends this information to the online shopping site and confirms the purchase.

[0072] Step 10:

[0073] The server confirms that the purchase process is complete and calculates a coordination fee based on the purchase amount. The calculated fee is then monetized and recorded in the system.

[0074] This system will enable users to quickly and easily coordinate their interiors and purchase necessary furniture and interior items on the spot.

[0075] (Example 1)

[0076] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0077] Traditional interior design proposal and purchasing systems require users to search for suitable items themselves, a process that is cumbersome and time-consuming. Furthermore, finding items that fit the user's budget and existing interior style is difficult, hindering efficient coordination proposals. Additionally, the purchasing process requires users to visit multiple online shopping sites individually, resulting in a poor user experience. A more efficient and user-friendly system is needed to address these challenges.

[0078] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0079] In this invention, the server includes means for a user to take a photograph of an interior using a mobile terminal and transmit the photographic data to the server; means for the terminal to transmit the photographic data selected by the user to the server; means for the server to pass the received photographic data to a generating AI model for analysis and extract features such as color, shape, and arrangement; means for the server to compare the analysis results with interior styles stored in a database and suggest similar interior coordinations; means for the server to search for appropriate furniture and interior items from multiple online shopping sites based on the suggested coordinations, filter them using criteria such as color, size, and price, and display them in a list; and means for the user to purchase selected items from the displayed list online. As a result, the user can efficiently receive interior coordination suggestions and complete the entire purchase process in a consistent manner without having to perform cumbersome search operations.

[0080] A "mobile device" is an electronic device that a user can carry with them and that has the ability to take pictures and communicate with the internet.

[0081] "Photo data" refers to digital image files taken by a user using the camera function of their mobile device.

[0082] A "server" is a computing system that exchanges data with multiple terminals via a network and performs various calculations and data management.

[0083] A "generative AI model" refers to an artificial intelligence algorithm that analyzes interior design photo data and extracts features such as color, shape, and arrangement.

[0084] "Interior coordination" refers to the proposal of arranging and selecting furniture and decorative items based on a specific interior style.

[0085] An "online shopping site" is an e-commerce platform that allows users to search for, select, and purchase products over the internet.

[0086] "Filtering" is the process of selecting data and information based on specific criteria and extracting only what is necessary.

[0087] "List view" refers to displaying multiple selected items or data in a visually clear and easy-to-understand format.

[0088] The "purchase process" refers to the series of processes related to ordering and paying for selected products.

[0089] "Coordination fee" refers to the fee charged for interior design services, which is based on the price of the items purchased by the user.

[0090] The system for carrying out the present invention begins with a user taking a photograph of the interior using a mobile device and sending the photographic data to a server. This system includes the following main components:

[0091] 1. User's device

[0092] Users take photos of their interiors using mobile devices such as smartphones. They can then upload the photos to a server using a dedicated application. A specific example would be a user taking a photo of their living room and pressing the "Upload" button within the app.

[0093] 2. Device operation

[0094] When the user presses the upload button, the device sends the selected photo data to a server in the cloud. The photo data is uploaded to the server via the internet. During this process, the photo data is properly compressed and encrypted to ensure secure transmission.

[0095] 3. Server analysis

[0096] The server passes the received photo data to a generating AI model for analysis. This AI model extracts features such as color, shape, and arrangement from the photo and matches them with interior styles stored in a database. Based on this, the server generates interior design ideas that are similar to the photo data. For example, it might suggest a modern style design based on the photo analysis results.

[0097] 4. Searching online shopping sites

[0098] The server searches for relevant furniture and interior items from multiple online shopping sites based on the proposed coordination. Criteria such as color, size, and price are used, and the search results are filtered to show only the most suitable items.

[0099] 5. Display a list of items

[0100] The server organizes the search results into a list format and sends it to the user's device. The user can then select the items they wish to purchase from this list. For example, a sofa, table, and carpet that fit the suggested modern style might be displayed.

[0101] 6. Purchase Procedure

[0102] Once the user selects the items they wish to purchase, the device proceeds with the purchase process via the online shopping site. The user can then enter their payment information and confirm the purchase.

[0103] 7. Monetizing coordination fees

[0104] The server confirms that the purchase is complete and calculates a coordination fee based on the purchase amount. For example, if the purchase amount is 100,000 yen, a fee of 1,000 yen will be generated. This revenue is used for server operation and service improvements.

[0105] Examples of specific cases and prompt statements

[0106] For example, if a user is looking for a new dining table, they would first take a photo of their dining area and upload it to the server via the app. The server would then analyze the photo, suggest a modern style, and search for and list suitable dining table sets from multiple shopping sites. The user could then select an item from the list and complete the purchase process.

[0107] Example of a prompt:

[0108] "I've taken and uploaded photos of the dining area's interior. Please use this to suggest a modern style interior design and search for a suitable dining table set."

[0109] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0110] Step 1:

[0111] The user takes a photo of the interior using their smartphone's camera function. The user presses the "Upload" button in the app and selects the photo data they have taken. The input is the photo data that has been taken, and the output is the photo data prepared within the app.

[0112] Step 2:

[0113] The device detects when the user presses the "upload" button and sends the selected photo data to a server in the cloud. The data is properly compressed and encrypted before transmission. The input is the photo data selected by the user, and the output is the photo data as it appears on the server.

[0114] Step 3:

[0115] The server analyzes the received photo data. First, it passes the photo data to a generative AI model, which extracts features such as color, shape, and arrangement. The input for this feature extraction is the photo data, and the output is the analyzed feature data.

[0116] Step 4:

[0117] The server matches the extracted feature data with interior styles stored in the database. Specifically, a generative AI model proposes interior design based on the analysis results. The input for this matching is feature data, and the output is proposed interior design data.

[0118] Step 5:

[0119] The server searches for suitable furniture and interior items from multiple online shopping sites based on the proposed interior design. Search criteria include color, size, and price. The input to this search is the interior design data, and the output is a filtered list of the best items.

[0120] Step 6:

[0121] The server organizes the search results into a list format and sends it to the user's device. The user can then select the items they wish to purchase from this list. The input is a filtered list of items, and the output is the list of items displayed on the device.

[0122] Step 7:

[0123] Once the user selects the items they wish to purchase, the device proceeds with the purchase process via the online shopping site. The user enters their payment information and confirms the purchase. The input consists of the user's selections and payment information, while the output is information about the purchased items.

[0124] Step 8:

[0125] The server confirms that the user's purchase is complete and calculates a commission based on the purchase amount. For example, if the purchase amount is 100,000 yen, a commission of 1,000 yen will be generated. The input for this commission calculation is the purchase amount, and the output is the calculated commission.

[0126] (Application Example 1)

[0127] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0128] In modern brick-and-mortar stores, customers can see products in person, but finding the right interior design requires considering numerous options, which is time-consuming and laborious. Furthermore, the purchasing process tends to be complex. There is also a lack of mechanisms to optimize future design suggestions based on online purchase history. A system is needed to address these problems.

[0129] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0130] In this invention, the server includes means for a user to take a photograph of interior decoration using an information and communication terminal and transmit the photographic data to the computer server; means for the computer server to analyze the received photographic data and suggest similar interior decorations; means for the computer server to search for appropriate decorative items and fixtures from multiple e-commerce sites based on the suggested decorations and display them in a list; means for the user to purchase selected items from the displayed list via electronic transaction; and means for suggesting the optimal coordination based on the photographed interior decorations in a physical store and searching for and displaying related products within the store. This makes it possible for users to improve their shopping experience in physical stores and to efficiently select and purchase products.

[0131] An "information and communication terminal" refers to an electronic device that allows a user to take photos and transmit that data.

[0132] "Interior decoration" refers to all decorative items and furniture applied to the indoor environment.

[0133] A "computer server" refers to a computing system that processes and analyzes received data and provides suggestions and search results.

[0134] "Similar interior decorations" refers to other decorative items and furniture that match the style and design desired by the user, based on the analyzed photographic data.

[0135] An "e-commerce site" refers to a website that facilitates the exchange of goods and services via the internet.

[0136] "Fixtures" refers to furniture, interior products, and other fixtures used for interior decoration.

[0137] "Electronic transactions" refer to the buying and selling of goods and services conducted using information and communication networks such as the internet.

[0138] A "physical store" refers to a business establishment that exists in a physical location and where users can actually visit to check and purchase products.

[0139] "Related products" refer to items that have complementary functions and designs, selected based on the suggested coordination.

[0140] The system for implementing this invention begins with the user taking a photograph of interior decoration with an information and communication terminal and transmitting the photographic data to a computer server. Based on this photographic data, the server proposes interior design ideas and further searches for and displays appropriate decorative items and fixtures from multiple e-commerce sites. The user can then select items from the displayed list and purchase them through electronic transactions. Each process is as follows:

[0141] User actions

[0142] The user first uses the camera function of their information and communication device to take pictures of the interior decorations in the physical store, and then uploads those photos through the application. For example, this might happen when a user is looking for a sofa to fit their living room in the furniture section of a physical store.

[0143] Terminal operation

[0144] The device sends the captured photo data to a computer server in response to user input. At this time, the data is uploaded to a cloud server via the internet.

[0145] Server analysis

[0146] The server uses an AI model built with TENSORFLOW® to analyze the received photo data. This model analyzes features such as color, shape, and arrangement in the photos and matches them with similar interior design styles stored in a database. This allows the server to generate optimal interior design based on the photo data.

[0147] Searching for e-commerce sites

[0148] The server searches for relevant decorative items and fixtures from multiple e-commerce sites based on the proposed coordination. Filtering is performed based on criteria such as color, size, and price. API requests are made using libraries such as Requests for the search.

[0149] View a list of items and complete the purchase process.

[0150] The server organizes the search results into a list and sends it to the user's device. The user can then select items they wish to purchase from this list and proceed with the electronic transaction. Purchase history is also stored in a database and used to optimize future outfit suggestions.

[0151] Examples of applications in physical stores

[0152] For example, suppose a user is looking for a sofa to fit their living room in the furniture section of a home improvement store. The user uses the app to take a picture of the displayed sofa and uploads the photo data to the server. The server uses an AI model to analyze the photo and suggests related interior furnishings such as tables and carpets. The user then checks the availability of the suggested items at the physical store and proceeds with the purchase on the spot.

[0153] Example of a prompt

[0154] "Please suggest a living room decor. I've attached photos below. I'd like a modern-style sofa. Please also suggest a matching table and carpet."

[0155] This invention allows users to efficiently select and purchase interior goods at physical stores, as well as receive personalized coordination suggestions based on their online shopping history.

[0156] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0157] Step 1:

[0158] Users take photos of interior decorations in physical stores using the camera on their information and communication terminals. The photos are saved to the device in common image formats such as JPEG and PNG. Then, the user opens the application, selects the photos, and presses the upload button.

[0159] Input: Photos of interior decorations

[0160] Output: Photo selection and upload button press by the application.

[0161] Step 2:

[0162] The device sends the photos selected by the user to a computer server in the cloud via the internet. An HTTP POST request is used for transmission, and the photo data is uploaded to the server in binary format.

[0163] Input: Photo data selected by the user by pressing the upload button.

[0164] Output: Photo data uploaded to a computer server in the cloud.

[0165] Step 3:

[0166] The server passes the received photo data to an AI model built with TensorFlow for analysis. The AI ​​model extracts features such as color, shape, and arrangement from the photo, and matches these features to interior styles in a database. Based on the feature information obtained from the AI ​​model, the server generates interior design ideas similar to the photo data.

[0167] Input: Uploaded photo data

[0168] Output: Analysis results by the AI ​​model, and generated interior design.

[0169] Step 4:

[0170] Based on the proposed interior design, the server searches for relevant decorative items and fixtures using the APIs of e-commerce sites. Specifically, it filters based on parameters such as color, size, and price to select the most suitable products from multiple e-commerce sites. The search is performed using libraries such as Python's requests library.

[0171] Input: Generated interior design and search parameters

[0172] Output: Product list obtained from e-commerce sites

[0173] Step 5:

[0174] The server organizes the search results in a list format and sends them to the user's device. The user can view the product list within the application and select the items they wish to purchase. The selected product information is then sent to the server via the application.

[0175] Input: Product list obtained from e-commerce site

[0176] Output: Product list and selected product information displayed on the user's device.

[0177] Step 6:

[0178] The server processes the purchase of the product selected by the user on the e-commerce site. Specifically, it receives the user's payment information and processes the purchase through the e-commerce site's API. Once the purchase is confirmed to be complete, the server saves the purchase history to the database.

[0179] Input: Selected product information and user's payment information

[0180] Output: Purchase completion confirmation and saved purchase history

[0181] Step 7:

[0182] The server optimizes future interior design suggestions based on accumulated purchase history. Purchase history data is fed back into a machine learning model, enabling more accurate design suggestions in subsequent analyses.

[0183] Input: Saved purchase history

[0184] Output: Optimized future interior design proposals

[0185] This will not only allow users to efficiently select and purchase interior goods at physical stores, but will also enable them to receive personalized coordination suggestions based on their online shopping history.

[0186] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0187] This invention begins with a user taking a photo of an interior using a mobile device and sending the photo data to a server. Based on this photo data, the server proposes interior design ideas, and then searches for and displays appropriate furniture and interior items from multiple online shopping sites. Furthermore, by incorporating an emotion engine that recognizes the user's emotional state, the quality of the suggestions is improved. Each process is as follows:

[0188] User actions

[0189] The user first uses the camera function of their smartphone or other mobile device to take pictures of interior design pages in magazines or of their own home interior. Then, they use the app to upload the photos they have taken. For example, a user might take a picture of their living room and press the "Upload" button in the app.

[0190] Terminal operation

[0191] When the user presses the upload button, the device sends the selected photo data to the server. The photo data is uploaded to a cloud server via the internet.

[0192] Server analysis

[0193] The server stores the received photo data and passes it to the AI ​​module. The AI ​​module extracts features such as color, shape, and arrangement from the photo and performs analysis to generate similar interior design ideas based on that data.

[0194] Searching online shopping sites

[0195] The server searches for relevant furniture and interior items from multiple online shopping sites based on the proposed coordination. Criteria such as color, size, and price are used. The search results are filtered to select only the items best suited to the user.

[0196] How the emotion engine works

[0197] The emotion engine analyzes data such as facial expressions, voice, and language as the user interacts with the app to determine the user's emotional state in real time. For example, it can detect emotions such as a smile or surprise when a user sees interior design suggestions.

[0198] Adjusting the interior design

[0199] The server adjusts interior design suggestions based on the user's emotional state as recognized by the emotion engine. If the user reacts positively to a suggestion, it continues to suggest similar styles; if the user reacts negatively, it suggests a different style.

[0200] List of items

[0201] The server generates a list of the final selected items and sends this information to the user's device. The user can then view this list in the app and choose the items they wish to purchase. For example, the list might display sofas, tables, carpets, and other items that fit the suggested modern style.

[0202] Purchase procedure

[0203] Once the user selects an item, the device proceeds with the purchase process via the online shopping site. The user can then enter their payment information and confirm the purchase.

[0204] Monetizing coordination fees

[0205] The server confirms that the purchase of the item has been completed and calculates a coordination fee based on the purchase amount. For example, if the purchase amount is 100,000 yen, a fee of 1,000 yen will be generated.

[0206] This system reduces the burden on users and enables quick and easy interior design coordination. Furthermore, by utilizing an emotion engine, it can provide suggestions optimized to the user's emotions. This, in turn, increases user satisfaction and provides a better purchasing experience.

[0207] The following describes the processing flow.

[0208] Step 1:

[0209] Users take photos of their interiors using their smartphone's camera function. This allows users to create photo data that serves as the basis for their interior design ideas.

[0210] Step 2:

[0211] The user taps the "Upload" button within the app, selects the photos they have taken, and sends them to the server. The device accepts this action and uploads the photo data to the server via the internet.

[0212] Step 3:

[0213] The server receives and saves the photo data, then passes it to the AI ​​module. The AI ​​module extracts features such as color, shape, and arrangement from the photo, and performs analysis to generate similar interior design ideas based on that data.

[0214] Step 4:

[0215] The server suggests similar interior styles based on the AI ​​analysis results. The suggestions are sent to the user's device, and the user can view the suggested coordination within the app.

[0216] Step 5:

[0217] The server uses the characteristics of the suggested coordination to call APIs from multiple online shopping sites to search for suitable furniture and interior items. This is done using criteria such as color, size, and price.

[0218] Step 6:

[0219] The server filters search results obtained from online shopping sites and selects the most suitable items. This selection is based on which items best fit the suggested outfit.

[0220] Step 7:

[0221] The server generates a list of selected items and sends this information to the user's device. The user can then view this list in the app and choose the items they wish to purchase.

[0222] Step 8:

[0223] The device monitors user actions and uses an emotion engine to analyze the user's emotional state. This includes facial recognition and voice analysis.

[0224] Step 9:

[0225] The emotion engine analyzes the user's emotional state in real time and sends the emotional data to the server. The server then uses this data to adjust the interior design suggestions.

[0226] Step 10:

[0227] The server sends the adjusted proposal back to the user's device. The user reviews the new proposal and, if satisfied, proceeds to the next step.

[0228] Step 11:

[0229] The user selects the item they want to purchase from the list and taps the purchase button within the app. The device accepts this action and proceeds with the purchase process using the online shopping site's API.

[0230] Step 12:

[0231] The user enters their payment information and taps the purchase button. The device sends this information to the online shopping site and confirms the purchase.

[0232] Step 13:

[0233] The server confirms that the purchase process is complete and calculates a coordination fee based on the purchase amount. The calculated fee is then monetized and recorded in the system.

[0234] This system allows users to quickly and easily create interior designs and purchase necessary furniture and interior items on the spot. Furthermore, by utilizing an emotion engine, it can provide interior design suggestions optimized to the user's emotions.

[0235] (Example 2)

[0236] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0237] In modern interior design, it is extremely difficult for users to select numerous pieces of furniture and interior items themselves and then effectively combine them. This is especially true in online shopping, where finding the right items from a vast selection requires considerable time and effort. To solve this problem, a system is needed that allows users to easily take photos of their interiors and receive suggestions for optimal furniture and interior items. Furthermore, features that adjust suggestions based on the user's emotional state and optimize future suggestions based on purchase history are also crucial.

[0238] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0239] This invention includes a server that allows a user to take photos of interiors using a mobile device and transmit the photo data to the server; a server that analyzes the received photo data and proposes similar interior coordinations using a generation AI model; a server that searches for appropriate furniture and interior items from multiple online shopping sites based on the proposed coordinations and displays them in a list; a server that analyzes the user's emotional state using an emotion engine and adjusts the interior coordination suggestions; and a server that allows the user to purchase selected items from the displayed list online. This enables users to easily achieve optimal interior coordination, improve the quality of suggestions based on their emotional state during the process, and further optimize future suggestions based on their purchase history.

[0240] A "mobile device" is a device with communication capabilities that a user can carry with them, and specifically includes smartphones and tablets.

[0241] "Photo data" refers to image information captured using the camera function of a mobile device, and is digital data that includes the colors, shapes, and arrangement of the interior.

[0242] A "server" refers to a high-performance computer system used to process, store, and deliver data over a network.

[0243] A "generative AI model" refers to an artificial intelligence system that uses machine learning algorithms to extract features from photographic data and propose interior design ideas.

[0244] An "emotion engine" refers to a software module that analyzes a user's facial expressions, voice, and language data to determine the user's emotional state in real time.

[0245] "Interior coordination" refers to proposals that create an aesthetically pleasing and functional space by considering the layout, colors, and placement of furniture in a room.

[0246] An "online shopping site" refers to a website that allows users to purchase goods over the internet.

[0247] "Items" refer to purchasable products such as furniture and decorative items used in interior design.

[0248] "List view" refers to a display format that visually shows users multiple related items.

[0249] "Purchase process" refers to the process by which a user enters payment information for selected items and confirms the purchase through an online shopping site.

[0250] "Purchase history" refers to data that records items a user has purchased in the past and their detailed information.

[0251] "The quality of the proposal" refers to the usefulness and suitability of the interior design proposal, which is optimized for the user's needs and emotional state.

[0252] "Optimization" refers to the process of adjusting suggestions and search results based on certain criteria to find the most suitable state or result.

[0253] This invention begins with a user taking a photo of an interior using a mobile device and sending the photo data to a server. The captured photo data is stored on the server and passed to an AI module (generative AI model). The AI ​​module analyzes this photo data, extracts features such as color, shape, and arrangement, and generates similar interior coordinations. The server then searches for appropriate furniture and interior items from multiple online shopping sites based on the generated coordinations and displays them in a list.

[0254] Next, the emotion engine analyzes the user's emotional state. It analyzes the user's facial expressions, voice, and language data in real time to determine how the user feels about the suggestions. Based on this data, the server adjusts the interior design suggestions. If the user shows a positive reaction, it continues to suggest similar styles; if the reaction is negative, it suggests a different style.

[0255] Specific hardware components include mobile devices such as smartphones and tablets used by users, cloud servers for storing and analyzing data, high-performance computers for running generative AI models, and emotion engines for analyzing user emotions. Software components include applications for sending photo data, algorithms for running generative AI models, data collection programs from online shopping sites, and analysis software for the emotion engine. For example, cloud services such as Google Cloud and Amazon Web Services could be used.

[0256] As a concrete example, when a user takes a photo of their living room and presses the "upload" button in the app, the photo data is sent to a server and stored in the cloud. The server then passes the photo to an AI module, which extracts features such as white walls and wooden furniture. The AI ​​module then suggests a natural-style interior design. Next, an emotion engine analyzes the user's reaction, and if it detects that the user is pleased, it suggests additional arrangements and items in a similar style.

[0257] An example of a prompt message might be: "Based on the interior photos taken by the user, suggest the optimal coordination. Describe a system that uses an emotion engine to analyze the user's reactions, adjust the suggestions, and support the final purchase process. In particular, clearly define the roles of the server, terminal, and user."

[0258] This invention allows users to quickly and easily achieve optimal interior design, and the emotional engine improves the quality of suggestions, resulting in a highly satisfying purchasing experience.

[0259] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0260] Step 1:

[0261] The user takes a photo of the interior.

[0262] Input: An image of the interior taken by the user using the camera function of their smartphone.

[0263] Output: Captured image data.

[0264] Operation: The user takes a photo of their living room or bedroom with their smartphone. Pressing the "Capture" button generates the image data.

[0265] Step 2:

[0266] Users upload photos to the app.

[0267] Input: Image data stored on a smartphone.

[0268] Output: Request to send image data to the server.

[0269] Operation: The user launches the app, selects a photo they have taken, and presses the "Upload" button. The image data is sent to the cloud server via the internet.

[0270] Step 3:

[0271] The device sends photo data to the server.

[0272] Input: Image data specified by the user.

[0273] Output: Image data stored on a cloud server.

[0274] Operation: The mobile device uploads image data to a cloud server via the internet. The properly encoded data is sent to the server.

[0275] Step 4:

[0276] The server saves the photo data and passes it to the AI ​​module.

[0277] Input: Photo data uploaded to a cloud server.

[0278] Output: Data in a format that can be processed by the AI ​​module.

[0279] Operation: The server stores the received photo data in a database and performs format conversion to pass the data to the generating AI model.

[0280] Step 5:

[0281] The AI module analyzes the photo data and generates interior coordinates

[0282] Input: Photo data passed to the AI module.

[0283] Output: Proposed interior coordinates.

[0284] Operation: The AI module extracts features such as the color, shape, and arrangement of the photo, and generates interior coordinates based on these features. The generated coordinate data is returned to the server.

[0285] Step 6:

[0286] The server searches for relevant items from online shopping sites

[0287] Input: Proposed interior coordinate data.

[0288] Output: Filtered search results.

[0289] Operation: The server searches for relevant furniture and interior items from multiple online shopping sites based on the proposed coordinate data. Apply filtering criteria such as color, size, and price to narrow down the search results.

[0290] Step 7:

[0291] The server organizes the search results and displays them to the user as a list

[0292] Input: Filtered search results.

[0293] Output: A list of items displayed on the user's app.

[0294] Action: The server selects the optimal item and sends its information to the user's terminal. The terminal displays the detailed information of the item in the app.

[0295] Step 8:

[0296] The emotion engine analyzes the user's emotional state.

[0297] Input: The user's facial expressions, voice, and language data.

[0298] Output: Data regarding the user's emotional state.

[0299] Action: The emotion engine analyzes the user's facial expressions and voice data, and determines the emotional reaction to the proposal in real time. This data is sent to the server.

[0300] Step 9:

[0301] The server adjusts the proposal based on the emotion data.

[0302] Input: The user's emotional state data.

[0303] Output: The adjusted interior coordination proposal.

[0304] Action: The server adjusts the proposal for interior coordination based on the user's emotional state recognized by the emotion engine. It maintains a similar style based on a positive reaction and proposes a different style based on a negative reaction.

[0305] Step 10:

[0306] The user proceeds with the purchase procedure. <A000966>

[0307] Input: Information on the item selected by the user.

[0308] Output: Completion of the purchase on the online shopping site.

[0309] Operation: The user selects items in the app and proceeds with the purchase process through the online shopping site. They enter payment information and confirm the purchase.

[0310] Step 11:

[0311] The server stores the purchase history and calculates the fees.

[0312] Input: Purchase completion information.

[0313] Output: Saved purchase history and calculated fees.

[0314] Operation: The server confirms that the purchase of the item has been completed and saves that information to the database. A coordination fee is calculated based on the purchase amount and monetized.

[0315] (Application Example 2)

[0316] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0317] Traditional interior design systems have faced challenges in improving user satisfaction because they only offer mechanical suggestions without considering the individual emotional state of the user. Furthermore, few systems offer the functionality to search and display suitable items from multiple online shopping sites simultaneously, requiring users to expend considerable effort to find appropriate items. Moreover, no interior design suggestion system existed that incorporated emotional analysis using user facial expressions and voice.

[0318] The identification processing by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for the user to take a picture of the interior using a mobile terminal and send the photo data to the server; means for the server to analyze the received photo data and suggest similar interior coordinations; means for the server to search for appropriate furniture and interior items from multiple online shopping sites based on the suggested coordinations and display them in a list; means for the user to purchase selected items from the displayed list online; means for detecting the user's emotional state using an emotion analysis device; and means for adjusting the suggested interior coordination based on the emotional state detected by the emotion analysis device. This makes it possible to suggest personalized interior coordinations that take into account the user's emotional state. Furthermore, since furniture and interior items can be searched and displayed from multiple online shopping sites at once, the user's effort is reduced, and they can find appropriate items more quickly and efficiently.

[0319] A "mobile device" refers to an electronic device that is portable and equipped with a camera function.

[0320] "Photo data" refers to image information of the interior taken with a mobile device.

[0321] A "server" refers to a computer device that is installed on the cloud via a network and is used for storing and analyzing data.

[0322] "Interior coordination" refers to the combination of furniture and decorative items proposed based on the colors, shapes, and arrangement of the interior shown in the photograph.

[0323] An "online shopping site" refers to a website where goods can be sold and purchased using the internet.

[0324] "Furniture and interior items" refer to items used to decorate or enhance the comfort of a room, specifically items such as sofas, tables, carpets, and lamps.

[0325] An "emotion analysis device" refers to a technological device that analyzes a user's facial expressions, voice, or language data in real time to detect the user's emotional state.

[0326] "Emotional state" refers to the results of analyzing a user's current emotions, such as joy, surprise, or dissatisfaction, based on facial expressions, voice, and other data.

[0327] "Adjustment" refers to modifying or changing the proposed content based on the user's individual emotional state and budget.

[0328] The process begins with the user taking photos of the interior using a mobile device such as a smartphone and uploading the photo data to a server via a dedicated application. It is assumed that the mobile device has a camera function.

[0329] When a device sends photo data to a server, the server receives and stores the data. The server is equipped with an AI module, which analyzes the received photo data. In the analysis process, features such as the color, shape, and arrangement of the subject are extracted, and based on these features, an interior design coordination is proposed.

[0330] Furthermore, the server searches for suitable furniture and interior items from multiple online shopping sites based on the proposed interior design. Search criteria include color, size, price, and more. The search results are filtered, and only the most suitable items are displayed to the user.

[0331] The emotion analysis device is also part of the system, analyzing facial expressions, voice, and language data in real time as the user interacts with the application. This device uses machine learning libraries such as TensorFlow and Keras to detect the user's emotional state. By analyzing the user's smiles and expressions of surprise when viewing suggested interior design, the server can adjust the interior design suggestions based on that emotional state.

[0332] For example, suppose a user takes a photo of their living room and uploads it to the server through the application. An AI module analyzes the photo and suggests a modern-style interior design. If the user smiles upon seeing the suggestion, an emotion analyzer detects the user's positive emotion. Based on this, the server will either continue suggesting a similar style or offer more in-depth suggestions. On the other hand, if the user shows expressions of surprise or dissatisfaction, the emotion analyzer detects that negative emotion, and the server will try suggesting a different style, such as a natural style.

[0333] This interior design assistant system allows users to receive personalized interior design suggestions. Furthermore, by providing information from online shopping sites in one place, it expands users' options and allows them to find suitable items quickly and efficiently.

[0334] Example of a prompt

[0335] The user takes a photo of their living room and uploads it through the app. AI analyzes the room's colors and furniture arrangement. It suggests a modern style coordination, and if the user smiles at it, it continues with the same style. If the user looks surprised or dissatisfied, it suggests a different style.

[0336] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0337] Step 1:

[0338] The user takes a photo of the interior using their mobile device and sends the photo data to a server via an application. The device receives the photo data as input and uploads it to the server via the internet. Specifically, this involves pressing the "upload" button in the application. The photo data is then sent to the server as output.

[0339] Step 2:

[0340] The server analyzes the received photo data. In this analysis process, the server uses an AI module to extract features such as color, shape, and arrangement from the photo data. The input is the received photo data, and the output is feature data. Specific operations include feature extraction using image analysis algorithms.

[0341] Step 3:

[0342] The server proposes interior design based on the analysis results. The server generates similar interior design ideas based on extracted feature data. The input is feature data, and the output is a proposed design. The specific operation includes a process of generating the optimal design using an AI model.

[0343] Step 4:

[0344] The server searches for suitable furniture and interior items from multiple online shopping sites based on the generated coordination suggestions. The server uses the coordination suggestions as input and generates search results as output. Specific operations include searching and data filtering using the APIs of each online shopping site.

[0345] Step 5:

[0346] The server sends data to the mobile device to display a list of filtered items. The server takes search results as input and sends data for display to the device as output. Specific operations include data format conversion and data transmission based on communication protocols.

[0347] Step 6:

[0348] The device displays a list of items to the user. The device uses data received from the server as input and displays the item list to the user as output. Specific operations include UI rendering.

[0349] Step 7:

[0350] The emotion analysis device detects the user's emotional state in real time. The user's facial expressions and voice data are used as input, and the detected emotional state is generated as output. The specific operation includes a process of data capture using a camera and microphone, and the application of an emotion analysis algorithm.

[0351] Step 8:

[0352] The server adjusts outfit suggestions based on the user's emotional state. The server takes emotional state data as input and generates adjusted outfit suggestions as output. Specific operations include data re-analysis and updating of suggestions to reflect the emotional state.

[0353] Step 9:

[0354] The server generates a list of the final coordinated items and sends this information to the mobile device. The server takes the coordinated coordination proposals as input and generates the data to send to the device as output. Specific operations include data format conversion and data transmission based on communication protocols.

[0355] Step 10:

[0356] The device displays the final item list to the user, and the user proceeds with the online purchase of selected items. The device takes the final item list as input and the user's selection and purchase process as output. Specific operations include UI rendering, user interaction processing, and sending purchase data to the online shopping site.

[0357] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0358] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0359] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0360] [Second Embodiment]

[0361] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0362] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0363] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0364] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0365] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0366] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0367] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0368] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0369] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0370] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0371] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0372] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0373] The system for implementing this invention begins with the user taking a photo of the interior using a mobile device and sending the photo data to a server. Based on this photo data, the server suggests interior design ideas and then searches for and displays appropriate furniture and interior items from multiple online shopping sites. The user can then select items from the displayed list and purchase them on the spot. Each process is as follows:

[0374] User actions

[0375] The user first uses the camera function of their smartphone or other mobile device to take pictures of interior design pages in magazines or of their own home interior. Then, they use the app to upload the photos they have taken. For example, a user might take a picture of their living room and press the "Upload" button in the app.

[0376] Terminal operation

[0377] When the user presses the upload button, the device sends the selected photo data to the server. The photo data is uploaded to a cloud server via the internet.

[0378] Server analysis

[0379] The server passes the received photo data to an AI algorithm for analysis. The AI ​​algorithm extracts features such as color, shape, and arrangement from the photo and matches them with interior styles stored in a database. This allows the server to generate interior design ideas similar to the photo data. For example, based on the photo analysis, it might suggest a modern style interior.

[0380] Searching online shopping sites

[0381] The server searches for relevant furniture and interior items from multiple online shopping sites based on the proposed coordination. Criteria such as color, size, and price are used. The search results are filtered to select only the items best suited to the user.

[0382] List of items

[0383] The server organizes the search results into a list format and sends it to the user's device. The user can then select the items they wish to purchase from this list. For example, a sofa, table, and carpet that fit the suggested modern style might be displayed.

[0384] Purchase procedure

[0385] Once the user selects an item, the device proceeds with the purchase process via the online shopping site. The user can then enter their payment information and confirm the purchase.

[0386] Monetizing coordination fees

[0387] The server confirms that the purchase of the product is complete and calculates a coordination fee based on the purchase amount. For example, if the purchase amount is 100,000 yen, a fee of 1,000 yen will be generated as revenue.

[0388] This invention provides a system that allows users to easily realize interior design coordination and purchase items on the spot. Furthermore, the server can accumulate data based on the user's purchasing behavior, enabling it to further optimize future coordination suggestions.

[0389] The following describes the processing flow.

[0390] Step 1:

[0391] Users use their smartphones to take photos of interior design pages in magazines or of their own home interiors. This allows users to create photo data that serves as the basis for their interior design ideas.

[0392] Step 2:

[0393] The user taps the "Upload" button within the app, selects the photos they have taken, and sends them to the server. The device accepts this action and uploads the photo data to the server via the internet.

[0394] Step 3:

[0395] The server receives and saves the photo data, then passes it to the AI ​​module. The AI ​​module extracts features such as color, shape, and arrangement from the photo, and performs analysis to generate similar interior design ideas based on that data.

[0396] Step 4:

[0397] The server suggests similar interior styles based on the AI ​​analysis results. The suggestions are sent to the user's device, and the user can view the suggested coordination within the app.

[0398] Step 5:

[0399] The server uses the characteristics of the proposed coordination to call APIs from multiple online shopping sites to search for suitable furniture and interior items. This is done using specified criteria such as color, size, and price.

[0400] Step 6:

[0401] The server filters search results obtained from online shopping sites and selects the most suitable items. This selection is based on which items best fit the suggested outfit.

[0402] Step 7:

[0403] The server generates a list of selected items and sends this information to the user's device. The user can then view this list in the app and choose the items they wish to purchase.

[0404] Step 8:

[0405] The user selects the item they want to purchase from the list and taps the purchase button within the app. The device accepts this action and proceeds with the purchase process using the online shopping site's API.

[0406] Step 9:

[0407] The user enters their payment information and taps the purchase button. The device sends this information to the online shopping site and confirms the purchase.

[0408] Step 10:

[0409] The server confirms that the purchase process is complete and calculates a coordination fee based on the purchase amount. The calculated fee is then monetized and recorded in the system.

[0410] This system will enable users to quickly and easily coordinate their interiors and purchase necessary furniture and interior items on the spot.

[0411] (Example 1)

[0412] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0413] Traditional interior design proposal and purchasing systems require users to search for suitable items themselves, a process that is cumbersome and time-consuming. Furthermore, finding items that fit the user's budget and existing interior style is difficult, hindering efficient coordination proposals. Additionally, the purchasing process requires users to visit multiple online shopping sites individually, resulting in a poor user experience. A more efficient and user-friendly system is needed to address these challenges.

[0414] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0415] In this invention, the server includes means for a user to take a photograph of an interior using a mobile terminal and transmit the photographic data to the server; means for the terminal to transmit the photographic data selected by the user to the server; means for the server to pass the received photographic data to a generating AI model for analysis and extract features such as color, shape, and arrangement; means for the server to compare the analysis results with interior styles stored in a database and suggest similar interior coordinations; means for the server to search for appropriate furniture and interior items from multiple online shopping sites based on the suggested coordinations, filter them using criteria such as color, size, and price, and display them in a list; and means for the user to purchase selected items from the displayed list online. As a result, the user can efficiently receive interior coordination suggestions and complete the entire purchase process in a consistent manner without having to perform cumbersome search operations.

[0416] A "mobile device" is an electronic device that a user can carry with them and that has the ability to take pictures and communicate with the internet.

[0417] "Photo data" refers to digital image files taken by a user using the camera function of their mobile device.

[0418] A "server" is a computing system that exchanges data with multiple terminals via a network and performs various calculations and data management.

[0419] A "generative AI model" refers to an artificial intelligence algorithm that analyzes interior design photo data and extracts features such as color, shape, and arrangement.

[0420] "Interior coordination" refers to the proposal of arranging and selecting furniture and decorative items based on a specific interior style.

[0421] An "online shopping site" is an e-commerce platform that allows users to search for, select, and purchase products over the internet.

[0422] "Filtering" is the process of selecting data and information based on specific criteria and extracting only what is necessary.

[0423] "List view" refers to displaying multiple selected items or data in a visually clear and easy-to-understand format.

[0424] The "purchase process" refers to the series of processes related to ordering and paying for selected products.

[0425] "Coordination fee" refers to the fee charged for interior design services, which is based on the price of the items purchased by the user.

[0426] The system for carrying out the present invention begins with a user taking a photograph of the interior using a mobile device and sending the photographic data to a server. This system includes the following main components:

[0427] 1. User's device

[0428] Users take photos of their interiors using mobile devices such as smartphones. They can then upload the photos to a server using a dedicated application. A specific example would be a user taking a photo of their living room and pressing the "Upload" button within the app.

[0429] 2. Device operation

[0430] When the user presses the upload button, the device sends the selected photo data to a server in the cloud. The photo data is uploaded to the server via the internet. During this process, the photo data is properly compressed and encrypted to ensure secure transmission.

[0431] 3. Server analysis

[0432] The server passes the received photo data to a generating AI model for analysis. This AI model extracts features such as color, shape, and arrangement from the photo and matches them with interior styles stored in a database. Based on this, the server generates interior design ideas that are similar to the photo data. For example, it might suggest a modern style design based on the photo analysis results.

[0433] 4. Searching online shopping sites

[0434] The server searches for relevant furniture and interior items from multiple online shopping sites based on the proposed coordination. Criteria such as color, size, and price are used, and the search results are filtered to show only the most suitable items.

[0435] 5. Display a list of items

[0436] The server organizes the search results into a list format and sends it to the user's device. The user can then select the items they wish to purchase from this list. For example, a sofa, table, and carpet that fit the suggested modern style might be displayed.

[0437] 6. Purchase Procedure

[0438] Once the user selects the items they wish to purchase, the device proceeds with the purchase process via the online shopping site. The user can then enter their payment information and confirm the purchase.

[0439] 7. Monetizing coordination fees

[0440] The server confirms that the purchase is complete and calculates a coordination fee based on the purchase amount. For example, if the purchase amount is 100,000 yen, a fee of 1,000 yen will be generated. This revenue is used for server operation and service improvements.

[0441] Examples of specific cases and prompt statements

[0442] For example, if a user is looking for a new dining table, they would first take a photo of their dining area and upload it to the server via the app. The server would then analyze the photo, suggest a modern style, and search for and list suitable dining table sets from multiple shopping sites. The user could then select an item from the list and complete the purchase process.

[0443] Example of a prompt:

[0444] "I've taken and uploaded photos of the dining area's interior. Please use this to suggest a modern style interior design and search for a suitable dining table set."

[0445] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0446] Step 1:

[0447] The user takes a photo of the interior using their smartphone's camera function. The user presses the "Upload" button in the app and selects the photo data they have taken. The input is the photo data that has been taken, and the output is the photo data prepared within the app.

[0448] Step 2:

[0449] The device detects when the user presses the "upload" button and sends the selected photo data to a server in the cloud. The data is properly compressed and encrypted before transmission. The input is the photo data selected by the user, and the output is the photo data as it appears on the server.

[0450] Step 3:

[0451] The server analyzes the received photo data. First, it passes the photo data to a generative AI model, which extracts features such as color, shape, and arrangement. The input for this feature extraction is the photo data, and the output is the analyzed feature data.

[0452] Step 4:

[0453] The server matches the extracted feature data with interior styles stored in the database. Specifically, a generative AI model proposes interior design based on the analysis results. The input for this matching is feature data, and the output is proposed interior design data.

[0454] Step 5:

[0455] The server searches for suitable furniture and interior items from multiple online shopping sites based on the proposed interior design. Search criteria include color, size, and price. The input to this search is the interior design data, and the output is a filtered list of the best items.

[0456] Step 6:

[0457] The server organizes the search results into a list format and sends it to the user's device. The user can then select the items they wish to purchase from this list. The input is a filtered list of items, and the output is the list of items displayed on the device.

[0458] Step 7:

[0459] Once the user selects the items they wish to purchase, the device proceeds with the purchase process via the online shopping site. The user enters their payment information and confirms the purchase. The input consists of the user's selections and payment information, while the output is information about the purchased items.

[0460] Step 8:

[0461] The server confirms that the user's purchase is complete and calculates a commission based on the purchase amount. For example, if the purchase amount is 100,000 yen, a commission of 1,000 yen will be generated. The input for this commission calculation is the purchase amount, and the output is the calculated commission.

[0462] (Application Example 1)

[0463] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0464] In modern brick-and-mortar stores, customers can see products in person, but finding the right interior design requires considering numerous options, which is time-consuming and laborious. Furthermore, the purchasing process tends to be complex. There is also a lack of mechanisms to optimize future design suggestions based on online purchase history. A system is needed to address these problems.

[0465] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0466] In this invention, the server includes means for a user to take a photograph of interior decoration using an information and communication terminal and transmit the photographic data to the computer server; means for the computer server to analyze the received photographic data and suggest similar interior decorations; means for the computer server to search for appropriate decorative items and fixtures from multiple e-commerce sites based on the suggested decorations and display them in a list; means for the user to purchase selected items from the displayed list via electronic transaction; and means for suggesting the optimal coordination based on the photographed interior decorations in a physical store and searching for and displaying related products within the store. This makes it possible for users to improve their shopping experience in physical stores and to efficiently select and purchase products.

[0467] An "information and communication terminal" refers to an electronic device that allows a user to take photos and transmit that data.

[0468] "Interior decoration" refers to all decorative items and furniture applied to the indoor environment.

[0469] A "computer server" refers to a computing system that processes and analyzes received data and provides suggestions and search results.

[0470] "Similar interior decorations" refers to other decorative items and furniture that match the style and design desired by the user, based on the analyzed photographic data.

[0471] An "e-commerce site" refers to a website that facilitates the exchange of goods and services via the internet.

[0472] "Fixtures" refers to furniture, interior products, and other fixtures used for interior decoration.

[0473] "Electronic transactions" refer to the buying and selling of goods and services conducted using information and communication networks such as the internet.

[0474] A "physical store" refers to a business establishment that exists in a physical location and where users can actually visit to check and purchase products.

[0475] "Related products" refer to items that have complementary functions and designs, selected based on the suggested coordination.

[0476] The system for implementing this invention begins with the user taking a photograph of interior decoration with an information and communication terminal and transmitting the photographic data to a computer server. Based on this photographic data, the server proposes interior design ideas and further searches for and displays appropriate decorative items and fixtures from multiple e-commerce sites. The user can then select items from the displayed list and purchase them through electronic transactions. Each process is as follows:

[0477] User actions

[0478] The user first uses the camera function of their information and communication device to take pictures of the interior decorations in the physical store, and then uploads those photos through the application. For example, this might happen when a user is looking for a sofa to fit their living room in the furniture section of a physical store.

[0479] Terminal operation

[0480] The device sends the captured photo data to a computer server in response to user input. At this time, the data is uploaded to a cloud server via the internet.

[0481] Server analysis

[0482] The server uses an AI model built with TensorFlow to analyze the received photo data. This model analyzes features such as color, shape, and arrangement in the photos and matches them with similar interior design styles stored in a database. This allows the server to generate optimal interior design suggestions based on the photo data.

[0483] Searching for e-commerce sites

[0484] The server searches for relevant decorative items and fixtures from multiple e-commerce sites based on the proposed coordination. Filtering is performed based on criteria such as color, size, and price. API requests are made using libraries such as Requests for the search.

[0485] View a list of items and complete the purchase process.

[0486] The server organizes the search results into a list and sends it to the user's device. The user can then select items they wish to purchase from this list and proceed with the electronic transaction. Purchase history is also stored in a database and used to optimize future outfit suggestions.

[0487] Examples of applications in physical stores

[0488] For example, suppose a user is looking for a sofa to fit their living room in the furniture section of a home improvement store. The user uses the app to take a picture of the displayed sofa and uploads the photo data to the server. The server uses an AI model to analyze the photo and suggests related interior furnishings such as tables and carpets. The user then checks the availability of the suggested items at the physical store and proceeds with the purchase on the spot.

[0489] Example of a prompt

[0490] "Please suggest a living room decor. I've attached photos below. I'd like a modern-style sofa. Please also suggest a matching table and carpet."

[0491] This invention allows users to efficiently select and purchase interior goods at physical stores, as well as receive personalized coordination suggestions based on their online shopping history.

[0492] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0493] Step 1:

[0494] Users take photos of interior decorations in physical stores using the camera on their information and communication terminals. The photos are saved to the device in common image formats such as JPEG and PNG. Then, the user opens the application, selects the photos, and presses the upload button.

[0495] Input: Photos of interior decorations

[0496] Output: Photo selection and upload button press by the application.

[0497] Step 2:

[0498] The device sends the photos selected by the user to a computer server in the cloud via the internet. An HTTP POST request is used for transmission, and the photo data is uploaded to the server in binary format.

[0499] Input: Photo data selected by the user by pressing the upload button.

[0500] Output: Photo data uploaded to a computer server in the cloud.

[0501] Step 3:

[0502] The server passes the received photo data to an AI model built with TensorFlow for analysis. The AI ​​model extracts features such as color, shape, and arrangement from the photo, and matches these features to interior styles in a database. Based on the feature information obtained from the AI ​​model, the server generates interior design ideas similar to the photo data.

[0503] Input: Uploaded photo data

[0504] Output: Analysis results by the AI ​​model, and generated interior design.

[0505] Step 4:

[0506] Based on the proposed interior design, the server searches for relevant decorative items and fixtures using the APIs of e-commerce sites. Specifically, it filters based on parameters such as color, size, and price to select the most suitable products from multiple e-commerce sites. The search is performed using libraries such as Python's requests library.

[0507] Input: Generated interior design and search parameters

[0508] Output: Product list obtained from e-commerce sites

[0509] Step 5:

[0510] The server organizes the search results in a list format and sends them to the user's device. The user can view the product list within the application and select the items they wish to purchase. The selected product information is then sent to the server via the application.

[0511] Input: Product list obtained from e-commerce site

[0512] Output: Product list and selected product information displayed on the user's device.

[0513] Step 6:

[0514] The server processes the purchase of the product selected by the user on the e-commerce site. Specifically, it receives the user's payment information and processes the purchase through the e-commerce site's API. Once the purchase is confirmed to be complete, the server saves the purchase history to the database.

[0515] Input: Selected product information and user's payment information

[0516] Output: Purchase completion confirmation and saved purchase history

[0517] Step 7:

[0518] The server optimizes future interior design suggestions based on accumulated purchase history. Purchase history data is fed back into a machine learning model, enabling more accurate design suggestions in subsequent analyses.

[0519] Input: Saved purchase history

[0520] Output: Optimized future interior design proposals

[0521] This will not only allow users to efficiently select and purchase interior goods at physical stores, but will also enable them to receive personalized coordination suggestions based on their online shopping history.

[0522] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0523] This invention begins with a user taking a photo of an interior using a mobile device and sending the photo data to a server. Based on this photo data, the server proposes interior design ideas, and then searches for and displays appropriate furniture and interior items from multiple online shopping sites. Furthermore, by incorporating an emotion engine that recognizes the user's emotional state, the quality of the suggestions is improved. Each process is as follows:

[0524] User actions

[0525] The user first uses the camera function of their smartphone or other mobile device to take pictures of interior design pages in magazines or of their own home interior. Then, they use the app to upload the photos they have taken. For example, a user might take a picture of their living room and press the "Upload" button in the app.

[0526] Terminal operation

[0527] When the user presses the upload button, the device sends the selected photo data to the server. The photo data is uploaded to a cloud server via the internet.

[0528] Server analysis

[0529] The server stores the received photo data and passes it to the AI ​​module. The AI ​​module extracts features such as color, shape, and arrangement from the photo and performs analysis to generate similar interior design ideas based on that data.

[0530] Searching online shopping sites

[0531] The server searches for relevant furniture and interior items from multiple online shopping sites based on the proposed coordination. Criteria such as color, size, and price are used. The search results are filtered to select only the items best suited to the user.

[0532] How the emotion engine works

[0533] The emotion engine analyzes data such as facial expressions, voice, and language as the user interacts with the app to determine the user's emotional state in real time. For example, it can detect emotions such as a smile or surprise when a user sees interior design suggestions.

[0534] Adjusting the interior design

[0535] The server adjusts interior design suggestions based on the user's emotional state as recognized by the emotion engine. If the user reacts positively to a suggestion, it continues to suggest similar styles; if the user reacts negatively, it suggests a different style.

[0536] List of items

[0537] The server generates a list of the final selected items and sends this information to the user's device. The user can then view this list in the app and choose the items they wish to purchase. For example, the list might display sofas, tables, carpets, and other items that fit the suggested modern style.

[0538] Purchase procedure

[0539] Once the user selects an item, the device proceeds with the purchase process via the online shopping site. The user can then enter their payment information and confirm the purchase.

[0540] Monetizing coordination fees

[0541] The server confirms that the purchase of the item has been completed and calculates a coordination fee based on the purchase amount. For example, if the purchase amount is 100,000 yen, a fee of 1,000 yen will be generated.

[0542] This system reduces the burden on users and enables quick and easy interior design coordination. Furthermore, by utilizing an emotion engine, it can provide suggestions optimized to the user's emotions. This, in turn, increases user satisfaction and provides a better purchasing experience.

[0543] The following describes the processing flow.

[0544] Step 1:

[0545] Users take photos of their interiors using their smartphone's camera function. This allows users to create photo data that serves as the basis for their interior design ideas.

[0546] Step 2:

[0547] The user taps the "Upload" button within the app, selects the photos they have taken, and sends them to the server. The device accepts this action and uploads the photo data to the server via the internet.

[0548] Step 3:

[0549] The server receives and saves the photo data, then passes it to the AI ​​module. The AI ​​module extracts features such as color, shape, and arrangement from the photo, and performs analysis to generate similar interior design ideas based on that data.

[0550] Step 4:

[0551] The server suggests similar interior styles based on the AI ​​analysis results. The suggestions are sent to the user's device, and the user can view the suggested coordination within the app.

[0552] Step 5:

[0553] The server uses the characteristics of the suggested coordination to call APIs from multiple online shopping sites to search for suitable furniture and interior items. This is done using criteria such as color, size, and price.

[0554] Step 6:

[0555] The server filters search results obtained from online shopping sites and selects the most suitable items. This selection is based on which items best fit the suggested outfit.

[0556] Step 7:

[0557] The server generates a list of selected items and sends this information to the user's device. The user can then view this list in the app and choose the items they wish to purchase.

[0558] Step 8:

[0559] The device monitors user actions and uses an emotion engine to analyze the user's emotional state. This includes facial recognition and voice analysis.

[0560] Step 9:

[0561] The emotion engine analyzes the user's emotional state in real time and sends the emotional data to the server. The server then uses this data to adjust the interior design suggestions.

[0562] Step 10:

[0563] The server sends the adjusted proposal back to the user's device. The user reviews the new proposal and, if satisfied, proceeds to the next step.

[0564] Step 11:

[0565] The user selects the item they want to purchase from the list and taps the purchase button within the app. The device accepts this action and proceeds with the purchase process using the online shopping site's API.

[0566] Step 12:

[0567] The user enters their payment information and taps the purchase button. The device sends this information to the online shopping site and confirms the purchase.

[0568] Step 13:

[0569] The server confirms that the purchase process is complete and calculates a coordination fee based on the purchase amount. The calculated fee is then monetized and recorded in the system.

[0570] This system allows users to quickly and easily create interior designs and purchase necessary furniture and interior items on the spot. Furthermore, by utilizing an emotion engine, it can provide interior design suggestions optimized to the user's emotions.

[0571] (Example 2)

[0572] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0573] In modern interior design, it is extremely difficult for users to select numerous pieces of furniture and interior items themselves and then effectively combine them. This is especially true in online shopping, where finding the right items from a vast selection requires considerable time and effort. To solve this problem, a system is needed that allows users to easily take photos of their interiors and receive suggestions for optimal furniture and interior items. Furthermore, features that adjust suggestions based on the user's emotional state and optimize future suggestions based on purchase history are also crucial.

[0574] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0575] This invention includes a server that allows a user to take photos of interiors using a mobile device and transmit the photo data to the server; a server that analyzes the received photo data and proposes similar interior coordinations using a generation AI model; a server that searches for appropriate furniture and interior items from multiple online shopping sites based on the proposed coordinations and displays them in a list; a server that analyzes the user's emotional state using an emotion engine and adjusts the interior coordination suggestions; and a server that allows the user to purchase selected items from the displayed list online. This enables users to easily achieve optimal interior coordination, improve the quality of suggestions based on their emotional state during the process, and further optimize future suggestions based on their purchase history.

[0576] A "mobile device" is a device with communication capabilities that a user can carry with them, and specifically includes smartphones and tablets.

[0577] "Photo data" refers to image information captured using the camera function of a mobile device, and is digital data that includes the colors, shapes, and arrangement of the interior.

[0578] A "server" refers to a high-performance computer system used to process, store, and deliver data over a network.

[0579] A "generative AI model" refers to an artificial intelligence system that uses machine learning algorithms to extract features from photographic data and propose interior design ideas.

[0580] An "emotion engine" refers to a software module that analyzes a user's facial expressions, voice, and language data to determine the user's emotional state in real time.

[0581] "Interior coordination" refers to proposals that create an aesthetically pleasing and functional space by considering the layout, colors, and placement of furniture in a room.

[0582] An "online shopping site" refers to a website that allows users to purchase goods over the internet.

[0583] "Items" refer to purchasable products such as furniture and decorative items used in interior design.

[0584] "List view" refers to a display format that visually shows users multiple related items.

[0585] "Purchase process" refers to the process by which a user enters payment information for selected items and confirms the purchase through an online shopping site.

[0586] "Purchase history" refers to data that records items a user has purchased in the past and their detailed information.

[0587] "The quality of the proposal" refers to the usefulness and suitability of the interior design proposal, which is optimized for the user's needs and emotional state.

[0588] "Optimization" refers to the process of adjusting suggestions and search results based on certain criteria to find the most suitable state or result.

[0589] This invention begins with a user taking a photo of an interior using a mobile device and sending the photo data to a server. The captured photo data is stored on the server and passed to an AI module (generative AI model). The AI ​​module analyzes this photo data, extracts features such as color, shape, and arrangement, and generates similar interior coordinations. The server then searches for appropriate furniture and interior items from multiple online shopping sites based on the generated coordinations and displays them in a list.

[0590] Next, the emotion engine analyzes the user's emotional state. It analyzes the user's facial expressions, voice, and language data in real time to determine how the user feels about the suggestions. Based on this data, the server adjusts the interior design suggestions. If the user shows a positive reaction, it continues to suggest similar styles; if the reaction is negative, it suggests a different style.

[0591] Specific hardware components include mobile devices such as smartphones and tablets used by users, cloud servers for storing and analyzing data, high-performance computers for running generative AI models, and emotion engines for analyzing user emotions. Software components include applications for sending photo data, algorithms for running generative AI models, data collection programs from online shopping sites, and analysis software for the emotion engine. For example, cloud services such as Google Cloud and Amazon Web Services could be used.

[0592] As a concrete example, when a user takes a photo of their living room and presses the "upload" button in the app, the photo data is sent to a server and stored in the cloud. The server then passes the photo to an AI module, which extracts features such as white walls and wooden furniture. The AI ​​module then suggests a natural-style interior design. Next, an emotion engine analyzes the user's reaction, and if it detects that the user is pleased, it suggests additional arrangements and items in a similar style.

[0593] An example of a prompt message might be: "Based on the interior photos taken by the user, suggest the optimal coordination. Describe a system that uses an emotion engine to analyze the user's reactions, adjust the suggestions, and support the final purchase process. In particular, clearly define the roles of the server, terminal, and user."

[0594] This invention allows users to quickly and easily achieve optimal interior design, and the emotional engine improves the quality of suggestions, resulting in a highly satisfying purchasing experience.

[0595] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0596] Step 1:

[0597] The user takes a photo of the interior.

[0598] Input: An image of the interior taken by the user using the camera function of their smartphone.

[0599] Output: Captured image data.

[0600] Operation: The user takes a photo of their living room or bedroom with their smartphone. Pressing the "Capture" button generates the image data.

[0601] Step 2:

[0602] Users upload photos to the app.

[0603] Input: Image data stored on a smartphone.

[0604] Output: Request to send image data to the server.

[0605] Operation: The user launches the app, selects a photo they have taken, and presses the "Upload" button. The image data is sent to the cloud server via the internet.

[0606] Step 3:

[0607] The device sends photo data to the server.

[0608] Input: Image data specified by the user.

[0609] Output: Image data stored on a cloud server.

[0610] Operation: The mobile device uploads image data to a cloud server via the internet. The properly encoded data is sent to the server.

[0611] Step 4:

[0612] The server saves the photo data and passes it to the AI ​​module.

[0613] Input: Photo data uploaded to a cloud server.

[0614] Output: Data in a format that can be processed by the AI ​​module.

[0615] Operation: The server stores the received photo data in a database and performs format conversion to pass the data to the generating AI model.

[0616] Step 5:

[0617] An AI module analyzes photo data and generates interior design coordinates.

[0618] Input: Photo data passed to the AI ​​module.

[0619] Output: Proposed interior design.

[0620] Operation: The AI ​​module extracts features such as color, shape, and arrangement from a photograph and generates interior design coordinates based on these features. The generated coordinate data is returned to the server.

[0621] Step 6:

[0622] Search for items related to the server on online shopping sites.

[0623] Input: Suggested interior design data.

[0624] Output: Filtered search results.

[0625] Operation: The server searches for relevant furniture and interior items from multiple online shopping sites based on the suggested coordination data. It then refines the search results by applying filtering criteria such as color, size, and price.

[0626] Step 7:

[0627] The server organizes the search results and displays them to the user in a list.

[0628] Input: Filtered search results.

[0629] Output: A list of items displayed in the user's app.

[0630] Operation: The server selects the most suitable item and sends that information to the user's device. The device then displays the item's details in the app.

[0631] Step 8:

[0632] The emotion engine analyzes the user's emotional state.

[0633] Input: User's facial expressions, voice, and language data.

[0634] Output: Data regarding the user's emotional state.

[0635] Operation: The emotion engine analyzes the user's facial expressions and voice data to determine their emotional response to suggestions in real time. This data is then sent to the server.

[0636] Step 9:

[0637] The server adjusts suggestions based on sentiment data.

[0638] Input: User's emotional state data.

[0639] Output: Adjusted interior design suggestions.

[0640] Operation: The server adjusts interior design suggestions based on the user's emotional state as recognized by the emotion engine. It maintains a similar style based on positive responses and suggests a different style based on negative responses.

[0641] Step 10:

[0642] The user proceeds with the purchase process.

[0643] Input: Information about the item selected by the user.

[0644] Output: Purchase completed on the online shopping site.

[0645] Operation: The user selects items in the app and proceeds with the purchase process through the online shopping site. They enter payment information and confirm the purchase.

[0646] Step 11:

[0647] The server stores the purchase history and calculates the fees.

[0648] Input: Purchase completion information.

[0649] Output: Saved purchase history and calculated fees.

[0650] Operation: The server confirms that the purchase of the item has been completed and saves that information to the database. A coordination fee is calculated based on the purchase amount and monetized.

[0651] (Application Example 2)

[0652] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0653] Traditional interior design systems have faced challenges in improving user satisfaction because they only offer mechanical suggestions without considering the individual emotional state of the user. Furthermore, few systems offer the functionality to search and display suitable items from multiple online shopping sites simultaneously, requiring users to expend considerable effort to find appropriate items. Moreover, no interior design suggestion system existed that incorporated emotional analysis using user facial expressions and voice.

[0654] The identification processing by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for the user to take a picture of the interior using a mobile terminal and send the photo data to the server; means for the server to analyze the received photo data and suggest similar interior coordinations; means for the server to search for appropriate furniture and interior items from multiple online shopping sites based on the suggested coordinations and display them in a list; means for the user to purchase selected items from the displayed list online; means for detecting the user's emotional state using an emotion analysis device; and means for adjusting the suggested interior coordination based on the emotional state detected by the emotion analysis device. This makes it possible to suggest personalized interior coordinations that take into account the user's emotional state. Furthermore, since furniture and interior items can be searched and displayed from multiple online shopping sites at once, the user's effort is reduced, and they can find appropriate items more quickly and efficiently.

[0655] A "mobile device" refers to an electronic device that is portable and equipped with a camera function.

[0656] "Photo data" refers to image information of the interior taken with a mobile device.

[0657] A "server" refers to a computer device that is installed on the cloud via a network and is used for storing and analyzing data.

[0658] "Interior coordination" refers to the combination of furniture and decorative items proposed based on the colors, shapes, and arrangement of the interior shown in the photograph.

[0659] An "online shopping site" refers to a website where goods can be sold and purchased using the internet.

[0660] "Furniture and interior items" refer to items used to decorate or enhance the comfort of a room, specifically items such as sofas, tables, carpets, and lamps.

[0661] An "emotion analysis device" refers to a technological device that analyzes a user's facial expressions, voice, or language data in real time to detect the user's emotional state.

[0662] "Emotional state" refers to the results of analyzing a user's current emotions, such as joy, surprise, or dissatisfaction, based on facial expressions, voice, and other data.

[0663] "Adjustment" refers to modifying or changing the proposed content based on the user's individual emotional state and budget.

[0664] The process begins with the user taking photos of the interior using a mobile device such as a smartphone and uploading the photo data to a server via a dedicated application. It is assumed that the mobile device has a camera function.

[0665] When a device sends photo data to a server, the server receives and stores the data. The server is equipped with an AI module, which analyzes the received photo data. In the analysis process, features such as the color, shape, and arrangement of the subject are extracted, and based on these features, an interior design coordination is proposed.

[0666] Furthermore, the server searches for suitable furniture and interior items from multiple online shopping sites based on the proposed interior design. Search criteria include color, size, price, and more. The search results are filtered, and only the most suitable items are displayed to the user.

[0667] The emotion analysis device is also part of the system, analyzing facial expressions, voice, and language data in real time as the user interacts with the application. This device uses machine learning libraries such as TensorFlow and Keras to detect the user's emotional state. By analyzing the user's smiles and expressions of surprise when viewing suggested interior design, the server can adjust the interior design suggestions based on that emotional state.

[0668] For example, suppose a user takes a photo of their living room and uploads it to the server through the application. An AI module analyzes the photo and suggests a modern-style interior design. If the user smiles upon seeing the suggestion, an emotion analyzer detects the user's positive emotion. Based on this, the server will either continue suggesting a similar style or offer more in-depth suggestions. On the other hand, if the user shows expressions of surprise or dissatisfaction, the emotion analyzer detects that negative emotion, and the server will try suggesting a different style, such as a natural style.

[0669] This interior design assistant system allows users to receive personalized interior design suggestions. Furthermore, by providing information from online shopping sites in one place, it expands users' options and allows them to find suitable items quickly and efficiently.

[0670] Example of a prompt

[0671] The user takes a photo of their living room and uploads it through the app. AI analyzes the room's colors and furniture arrangement. It suggests a modern style coordination, and if the user smiles at it, it continues with the same style. If the user looks surprised or dissatisfied, it suggests a different style.

[0672] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0673] Step 1:

[0674] The user takes a photo of the interior using their mobile device and sends the photo data to a server via an application. The device receives the photo data as input and uploads it to the server via the internet. Specifically, this involves pressing the "upload" button in the application. The photo data is then sent to the server as output.

[0675] Step 2:

[0676] The server analyzes the received photo data. In this analysis process, the server uses an AI module to extract features such as color, shape, and arrangement from the photo data. The input is the received photo data, and the output is feature data. Specific operations include feature extraction using image analysis algorithms.

[0677] Step 3:

[0678] The server proposes interior design based on the analysis results. The server generates similar interior design ideas based on extracted feature data. The input is feature data, and the output is a proposed design. The specific operation includes a process of generating the optimal design using an AI model.

[0679] Step 4:

[0680] The server searches for suitable furniture and interior items from multiple online shopping sites based on the generated coordination suggestions. The server uses the coordination suggestions as input and generates search results as output. Specific operations include searching and data filtering using the APIs of each online shopping site.

[0681] Step 5:

[0682] The server sends data to the mobile device to display a list of filtered items. The server takes search results as input and sends data for display to the device as output. Specific operations include data format conversion and data transmission based on communication protocols.

[0683] Step 6:

[0684] The device displays a list of items to the user. The device uses data received from the server as input and displays the item list to the user as output. Specific operations include UI rendering.

[0685] Step 7:

[0686] The emotion analysis device detects the user's emotional state in real time. The user's facial expressions and voice data are used as input, and the detected emotional state is generated as output. The specific operation includes a process of data capture using a camera and microphone, and the application of an emotion analysis algorithm.

[0687] Step 8:

[0688] The server adjusts outfit suggestions based on the user's emotional state. The server takes emotional state data as input and generates adjusted outfit suggestions as output. Specific operations include data re-analysis and updating of suggestions to reflect the emotional state.

[0689] Step 9:

[0690] The server generates a list of the final coordinated items and sends this information to the mobile device. The server takes the coordinated coordination proposals as input and generates the data to send to the device as output. Specific operations include data format conversion and data transmission based on communication protocols.

[0691] Step 10:

[0692] The device displays the final item list to the user, and the user proceeds with the online purchase of selected items. The device takes the final item list as input and the user's selection and purchase process as output. Specific operations include UI rendering, user interaction processing, and sending purchase data to the online shopping site.

[0693] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0694] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0695] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0696] [Third Embodiment]

[0697] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0698] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0699] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0700] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0701] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0702] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0703] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0704] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0705] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0706] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0707] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0708] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0709] The system for implementing this invention begins with the user taking a photo of the interior using a mobile device and sending the photo data to a server. Based on this photo data, the server suggests interior design ideas and then searches for and displays appropriate furniture and interior items from multiple online shopping sites. The user can then select items from the displayed list and purchase them on the spot. Each process is as follows:

[0710] User actions

[0711] The user first uses the camera function of their smartphone or other mobile device to take pictures of interior design pages in magazines or of their own home interior. Then, they use the app to upload the photos they have taken. For example, a user might take a picture of their living room and press the "Upload" button in the app.

[0712] Terminal operation

[0713] When the user presses the upload button, the device sends the selected photo data to the server. The photo data is uploaded to a cloud server via the internet.

[0714] Server analysis

[0715] The server passes the received photo data to an AI algorithm for analysis. The AI ​​algorithm extracts features such as color, shape, and arrangement from the photo and matches them with interior styles stored in a database. This allows the server to generate interior design ideas similar to the photo data. For example, based on the photo analysis, it might suggest a modern style interior.

[0716] Searching online shopping sites

[0717] The server searches for relevant furniture and interior items from multiple online shopping sites based on the proposed coordination. Criteria such as color, size, and price are used. The search results are filtered to select only the items best suited to the user.

[0718] List of items

[0719] The server organizes the search results into a list format and sends it to the user's device. The user can then select the items they wish to purchase from this list. For example, a sofa, table, and carpet that fit the suggested modern style might be displayed.

[0720] Purchase procedure

[0721] Once the user selects an item, the device proceeds with the purchase process via the online shopping site. The user can then enter their payment information and confirm the purchase.

[0722] Monetizing coordination fees

[0723] The server confirms that the purchase of the product is complete and calculates a coordination fee based on the purchase amount. For example, if the purchase amount is 100,000 yen, a fee of 1,000 yen will be generated as revenue.

[0724] This invention provides a system that allows users to easily realize interior design coordination and purchase items on the spot. Furthermore, the server can accumulate data based on the user's purchasing behavior, enabling it to further optimize future coordination suggestions.

[0725] The following describes the processing flow.

[0726] Step 1:

[0727] Users use their smartphones to take photos of interior design pages in magazines or of their own home interiors. This allows users to create photo data that serves as the basis for their interior design ideas.

[0728] Step 2:

[0729] The user taps the "Upload" button within the app, selects the photos they have taken, and sends them to the server. The device accepts this action and uploads the photo data to the server via the internet.

[0730] Step 3:

[0731] The server receives and saves the photo data, then passes it to the AI ​​module. The AI ​​module extracts features such as color, shape, and arrangement from the photo, and performs analysis to generate similar interior design ideas based on that data.

[0732] Step 4:

[0733] The server suggests similar interior styles based on the AI ​​analysis results. The suggestions are sent to the user's device, and the user can view the suggested coordination within the app.

[0734] Step 5:

[0735] The server uses the characteristics of the proposed coordination to call APIs from multiple online shopping sites to search for suitable furniture and interior items. This is done using specified criteria such as color, size, and price.

[0736] Step 6:

[0737] The server filters search results obtained from online shopping sites and selects the most suitable items. This selection is based on which items best fit the suggested outfit.

[0738] Step 7:

[0739] The server generates a list of selected items and sends this information to the user's device. The user can then view this list in the app and choose the items they wish to purchase.

[0740] Step 8:

[0741] The user selects the item they want to purchase from the list and taps the purchase button within the app. The device accepts this action and proceeds with the purchase process using the online shopping site's API.

[0742] Step 9:

[0743] The user enters their payment information and taps the purchase button. The device sends this information to the online shopping site and confirms the purchase.

[0744] Step 10:

[0745] The server confirms that the purchase process is complete and calculates a coordination fee based on the purchase amount. The calculated fee is then monetized and recorded in the system.

[0746] This system will enable users to quickly and easily coordinate their interiors and purchase necessary furniture and interior items on the spot.

[0747] (Example 1)

[0748] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0749] Traditional interior design proposal and purchasing systems require users to search for suitable items themselves, a process that is cumbersome and time-consuming. Furthermore, finding items that fit the user's budget and existing interior style is difficult, hindering efficient coordination proposals. Additionally, the purchasing process requires users to visit multiple online shopping sites individually, resulting in a poor user experience. A more efficient and user-friendly system is needed to address these challenges.

[0750] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0751] In this invention, the server includes means for a user to take a photograph of an interior using a mobile terminal and transmit the photographic data to the server; means for the terminal to transmit the photographic data selected by the user to the server; means for the server to pass the received photographic data to a generating AI model for analysis and extract features such as color, shape, and arrangement; means for the server to compare the analysis results with interior styles stored in a database and suggest similar interior coordinations; means for the server to search for appropriate furniture and interior items from multiple online shopping sites based on the suggested coordinations, filter them using criteria such as color, size, and price, and display them in a list; and means for the user to purchase selected items from the displayed list online. As a result, the user can efficiently receive interior coordination suggestions and complete the entire purchase process in a consistent manner without having to perform cumbersome search operations.

[0752] A "mobile device" is an electronic device that a user can carry with them and that has the ability to take pictures and communicate with the internet.

[0753] "Photo data" refers to digital image files taken by a user using the camera function of their mobile device.

[0754] A "server" is a computing system that exchanges data with multiple terminals via a network and performs various calculations and data management.

[0755] A "generative AI model" refers to an artificial intelligence algorithm that analyzes interior design photo data and extracts features such as color, shape, and arrangement.

[0756] "Interior coordination" refers to the proposal of arranging and selecting furniture and decorative items based on a specific interior style.

[0757] An "online shopping site" is an e-commerce platform that allows users to search for, select, and purchase products over the internet.

[0758] "Filtering" is the process of selecting data and information based on specific criteria and extracting only what is necessary.

[0759] "List view" refers to displaying multiple selected items or data in a visually clear and easy-to-understand format.

[0760] The "purchase process" refers to the series of processes related to ordering and paying for selected products.

[0761] "Coordination fee" refers to the fee charged for interior design services, which is based on the price of the items purchased by the user.

[0762] The system for carrying out the present invention begins with a user taking a photograph of the interior using a mobile device and sending the photographic data to a server. This system includes the following main components:

[0763] 1. User's device

[0764] Users take photos of their interiors using mobile devices such as smartphones. They can then upload the photos to a server using a dedicated application. A specific example would be a user taking a photo of their living room and pressing the "Upload" button within the app.

[0765] 2. Device operation

[0766] When the user presses the upload button, the device sends the selected photo data to a server in the cloud. The photo data is uploaded to the server via the internet. During this process, the photo data is properly compressed and encrypted to ensure secure transmission.

[0767] 3. Server analysis

[0768] The server passes the received photo data to a generating AI model for analysis. This AI model extracts features such as color, shape, and arrangement from the photo and matches them with interior styles stored in a database. Based on this, the server generates interior design ideas that are similar to the photo data. For example, it might suggest a modern style design based on the photo analysis results.

[0769] 4. Searching online shopping sites

[0770] The server searches for relevant furniture and interior items from multiple online shopping sites based on the proposed coordination. Criteria such as color, size, and price are used, and the search results are filtered to show only the most suitable items.

[0771] 5. Display a list of items

[0772] The server organizes the search results into a list format and sends it to the user's device. The user can then select the items they wish to purchase from this list. For example, a sofa, table, and carpet that fit the suggested modern style might be displayed.

[0773] 6. Purchase Procedure

[0774] Once the user selects the items they wish to purchase, the device proceeds with the purchase process via the online shopping site. The user can then enter their payment information and confirm the purchase.

[0775] 7. Monetizing coordination fees

[0776] The server confirms that the purchase is complete and calculates a coordination fee based on the purchase amount. For example, if the purchase amount is 100,000 yen, a fee of 1,000 yen will be generated. This revenue is used for server operation and service improvements.

[0777] Examples of specific cases and prompt statements

[0778] For example, if a user is looking for a new dining table, they would first take a photo of their dining area and upload it to the server via the app. The server would then analyze the photo, suggest a modern style, and search for and list suitable dining table sets from multiple shopping sites. The user could then select an item from the list and complete the purchase process.

[0779] Example of a prompt:

[0780] "I've taken and uploaded photos of the dining area's interior. Please use this to suggest a modern style interior design and search for a suitable dining table set."

[0781] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0782] Step 1:

[0783] The user takes a photo of the interior using their smartphone's camera function. The user presses the "Upload" button in the app and selects the photo data they have taken. The input is the photo data that has been taken, and the output is the photo data prepared within the app.

[0784] Step 2:

[0785] The device detects when the user presses the "upload" button and sends the selected photo data to a server in the cloud. The data is properly compressed and encrypted before transmission. The input is the photo data selected by the user, and the output is the photo data as it appears on the server.

[0786] Step 3:

[0787] The server analyzes the received photo data. First, it passes the photo data to a generative AI model, which extracts features such as color, shape, and arrangement. The input for this feature extraction is the photo data, and the output is the analyzed feature data.

[0788] Step 4:

[0789] The server matches the extracted feature data with interior styles stored in the database. Specifically, a generative AI model proposes interior design based on the analysis results. The input for this matching is feature data, and the output is proposed interior design data.

[0790] Step 5:

[0791] The server searches for suitable furniture and interior items from multiple online shopping sites based on the proposed interior design. Search criteria include color, size, and price. The input to this search is the interior design data, and the output is a filtered list of the best items.

[0792] Step 6:

[0793] The server organizes the search results into a list format and sends it to the user's device. The user can then select the items they wish to purchase from this list. The input is a filtered list of items, and the output is the list of items displayed on the device.

[0794] Step 7:

[0795] Once the user selects the items they wish to purchase, the device proceeds with the purchase process via the online shopping site. The user enters their payment information and confirms the purchase. The input consists of the user's selections and payment information, while the output is information about the purchased items.

[0796] Step 8:

[0797] The server confirms that the user's purchase is complete and calculates a commission based on the purchase amount. For example, if the purchase amount is 100,000 yen, a commission of 1,000 yen will be generated. The input for this commission calculation is the purchase amount, and the output is the calculated commission.

[0798] (Application Example 1)

[0799] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0800] In modern brick-and-mortar stores, customers can see products in person, but finding the right interior design requires considering numerous options, which is time-consuming and laborious. Furthermore, the purchasing process tends to be complex. There is also a lack of mechanisms to optimize future design suggestions based on online purchase history. A system is needed to address these problems.

[0801] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0802] In this invention, the server includes means for a user to take a photograph of interior decoration using an information and communication terminal and transmit the photographic data to the computer server; means for the computer server to analyze the received photographic data and suggest similar interior decorations; means for the computer server to search for appropriate decorative items and fixtures from multiple e-commerce sites based on the suggested decorations and display them in a list; means for the user to purchase selected items from the displayed list via electronic transaction; and means for suggesting the optimal coordination based on the photographed interior decorations in a physical store and searching for and displaying related products within the store. This makes it possible for users to improve their shopping experience in physical stores and to efficiently select and purchase products.

[0803] An "information and communication terminal" refers to an electronic device that allows a user to take photos and transmit that data.

[0804] "Interior decoration" refers to all decorative items and furniture applied to the indoor environment.

[0805] A "computer server" refers to a computing system that processes and analyzes received data and provides suggestions and search results.

[0806] "Similar interior decorations" refers to other decorative items and furniture that match the style and design desired by the user, based on the analyzed photographic data.

[0807] An "e-commerce site" refers to a website that facilitates the exchange of goods and services via the internet.

[0808] "Fixtures" refers to furniture, interior products, and other fixtures used for interior decoration.

[0809] "Electronic transactions" refer to the buying and selling of goods and services conducted using information and communication networks such as the internet.

[0810] A "physical store" refers to a business establishment that exists in a physical location and where users can actually visit to check and purchase products.

[0811] "Related products" refer to items that have complementary functions and designs, selected based on the suggested coordination.

[0812] The system for implementing this invention begins with the user taking a photograph of interior decoration with an information and communication terminal and transmitting the photographic data to a computer server. Based on this photographic data, the server proposes interior design ideas and further searches for and displays appropriate decorative items and fixtures from multiple e-commerce sites. The user can then select items from the displayed list and purchase them through electronic transactions. Each process is as follows:

[0813] User actions

[0814] The user first uses the camera function of their information and communication device to take pictures of the interior decorations in the physical store, and then uploads those photos through the application. For example, this might happen when a user is looking for a sofa to fit their living room in the furniture section of a physical store.

[0815] Terminal operation

[0816] The device sends the captured photo data to a computer server in response to user input. At this time, the data is uploaded to a cloud server via the internet.

[0817] Server analysis

[0818] The server uses an AI model built with TensorFlow to analyze the received photo data. This model analyzes features such as color, shape, and arrangement in the photos and matches them with similar interior design styles stored in a database. This allows the server to generate optimal interior design suggestions based on the photo data.

[0819] Searching for e-commerce sites

[0820] The server searches for relevant decorative items and fixtures from multiple e-commerce sites based on the proposed coordination. Filtering is performed based on criteria such as color, size, and price. API requests are made using libraries such as Requests for the search.

[0821] View a list of items and complete the purchase process.

[0822] The server organizes the search results into a list and sends it to the user's device. The user can then select items they wish to purchase from this list and proceed with the electronic transaction. Purchase history is also stored in a database and used to optimize future outfit suggestions.

[0823] Examples of applications in physical stores

[0824] For example, suppose a user is looking for a sofa to fit their living room in the furniture section of a home improvement store. The user uses the app to take a picture of the displayed sofa and uploads the photo data to the server. The server uses an AI model to analyze the photo and suggests related interior furnishings such as tables and carpets. The user then checks the availability of the suggested items at the physical store and proceeds with the purchase on the spot.

[0825] Example of a prompt

[0826] "Please suggest a living room decor. I've attached photos below. I'd like a modern-style sofa. Please also suggest a matching table and carpet."

[0827] This invention allows users to efficiently select and purchase interior goods at physical stores, as well as receive personalized coordination suggestions based on their online shopping history.

[0828] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0829] Step 1:

[0830] Users take photos of interior decorations in physical stores using the camera on their information and communication terminals. The photos are saved to the device in common image formats such as JPEG and PNG. Then, the user opens the application, selects the photos, and presses the upload button.

[0831] Input: Photos of interior decorations

[0832] Output: Photo selection and upload button press by the application.

[0833] Step 2:

[0834] The device sends the photos selected by the user to a computer server in the cloud via the internet. An HTTP POST request is used for transmission, and the photo data is uploaded to the server in binary format.

[0835] Input: Photo data selected by the user by pressing the upload button.

[0836] Output: Photo data uploaded to a computer server in the cloud.

[0837] Step 3:

[0838] The server passes the received photo data to an AI model built with TensorFlow for analysis. The AI ​​model extracts features such as color, shape, and arrangement from the photo, and matches these features to interior styles in a database. Based on the feature information obtained from the AI ​​model, the server generates interior design ideas similar to the photo data.

[0839] Input: Uploaded photo data

[0840] Output: Analysis results by the AI ​​model, and generated interior design.

[0841] Step 4:

[0842] Based on the proposed interior design, the server searches for relevant decorative items and fixtures using the APIs of e-commerce sites. Specifically, it filters based on parameters such as color, size, and price to select the most suitable products from multiple e-commerce sites. The search is performed using libraries such as Python's requests library.

[0843] Input: Generated interior design and search parameters

[0844] Output: Product list obtained from e-commerce sites

[0845] Step 5:

[0846] The server organizes the search results in a list format and sends them to the user's device. The user can view the product list within the application and select the items they wish to purchase. The selected product information is then sent to the server via the application.

[0847] Input: Product list obtained from e-commerce site

[0848] Output: Product list and selected product information displayed on the user's device.

[0849] Step 6:

[0850] The server processes the purchase of the product selected by the user on the e-commerce site. Specifically, it receives the user's payment information and processes the purchase through the e-commerce site's API. Once the purchase is confirmed to be complete, the server saves the purchase history to the database.

[0851] Input: Selected product information and user's payment information

[0852] Output: Purchase completion confirmation and saved purchase history

[0853] Step 7:

[0854] The server optimizes future interior design suggestions based on accumulated purchase history. Purchase history data is fed back into a machine learning model, enabling more accurate design suggestions in subsequent analyses.

[0855] Input: Saved purchase history

[0856] Output: Optimized future interior design proposals

[0857] This will not only allow users to efficiently select and purchase interior goods at physical stores, but will also enable them to receive personalized coordination suggestions based on their online shopping history.

[0858] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0859] This invention begins with a user taking a photo of an interior using a mobile device and sending the photo data to a server. Based on this photo data, the server proposes interior design ideas, and then searches for and displays appropriate furniture and interior items from multiple online shopping sites. Furthermore, by incorporating an emotion engine that recognizes the user's emotional state, the quality of the suggestions is improved. Each process is as follows:

[0860] User actions

[0861] The user first uses the camera function of their smartphone or other mobile device to take pictures of interior design pages in magazines or of their own home interior. Then, they use the app to upload the photos they have taken. For example, a user might take a picture of their living room and press the "Upload" button in the app.

[0862] Terminal operation

[0863] When the user presses the upload button, the device sends the selected photo data to the server. The photo data is uploaded to a cloud server via the internet.

[0864] Server analysis

[0865] The server stores the received photo data and passes it to the AI ​​module. The AI ​​module extracts features such as color, shape, and arrangement from the photo and performs analysis to generate similar interior design ideas based on that data.

[0866] Searching online shopping sites

[0867] The server searches for relevant furniture and interior items from multiple online shopping sites based on the proposed coordination. Criteria such as color, size, and price are used. The search results are filtered to select only the items best suited to the user.

[0868] How the emotion engine works

[0869] The emotion engine analyzes data such as facial expressions, voice, and language as the user interacts with the app to determine the user's emotional state in real time. For example, it can detect emotions such as a smile or surprise when a user sees interior design suggestions.

[0870] Adjusting the interior design

[0871] The server adjusts interior design suggestions based on the user's emotional state as recognized by the emotion engine. If the user reacts positively to a suggestion, it continues to suggest similar styles; if the user reacts negatively, it suggests a different style.

[0872] List of items

[0873] The server generates a list of the final selected items and sends this information to the user's device. The user can then view this list in the app and choose the items they wish to purchase. For example, the list might display sofas, tables, carpets, and other items that fit the suggested modern style.

[0874] Purchase procedure

[0875] Once the user selects an item, the device proceeds with the purchase process via the online shopping site. The user can then enter their payment information and confirm the purchase.

[0876] Monetizing coordination fees

[0877] The server confirms that the purchase of the item has been completed and calculates a coordination fee based on the purchase amount. For example, if the purchase amount is 100,000 yen, a fee of 1,000 yen will be generated.

[0878] This system reduces the burden on users and enables quick and easy interior design coordination. Furthermore, by utilizing an emotion engine, it can provide suggestions optimized to the user's emotions. This, in turn, increases user satisfaction and provides a better purchasing experience.

[0879] The following describes the processing flow.

[0880] Step 1:

[0881] Users take photos of their interiors using their smartphone's camera function. This allows users to create photo data that serves as the basis for their interior design ideas.

[0882] Step 2:

[0883] The user taps the "Upload" button within the app, selects the photos they have taken, and sends them to the server. The device accepts this action and uploads the photo data to the server via the internet.

[0884] Step 3:

[0885] The server receives and saves the photo data, then passes it to the AI ​​module. The AI ​​module extracts features such as color, shape, and arrangement from the photo, and performs analysis to generate similar interior design ideas based on that data.

[0886] Step 4:

[0887] The server suggests similar interior styles based on the AI ​​analysis results. The suggestions are sent to the user's device, and the user can view the suggested coordination within the app.

[0888] Step 5:

[0889] The server uses the characteristics of the suggested coordination to call APIs from multiple online shopping sites to search for suitable furniture and interior items. This is done using criteria such as color, size, and price.

[0890] Step 6:

[0891] The server filters search results obtained from online shopping sites and selects the most suitable items. This selection is based on which items best fit the suggested outfit.

[0892] Step 7:

[0893] The server generates a list of selected items and sends this information to the user's device. The user can then view this list in the app and choose the items they wish to purchase.

[0894] Step 8:

[0895] The device monitors user actions and uses an emotion engine to analyze the user's emotional state. This includes facial recognition and voice analysis.

[0896] Step 9:

[0897] The emotion engine analyzes the user's emotional state in real time and sends the emotional data to the server. The server then uses this data to adjust the interior design suggestions.

[0898] Step 10:

[0899] The server sends the adjusted proposal back to the user's device. The user reviews the new proposal and, if satisfied, proceeds to the next step.

[0900] Step 11:

[0901] The user selects the item they want to purchase from the list and taps the purchase button within the app. The device accepts this action and proceeds with the purchase process using the online shopping site's API.

[0902] Step 12:

[0903] The user enters their payment information and taps the purchase button. The device sends this information to the online shopping site and confirms the purchase.

[0904] Step 13:

[0905] The server confirms that the purchase process is complete and calculates a coordination fee based on the purchase amount. The calculated fee is then monetized and recorded in the system.

[0906] This system allows users to quickly and easily create interior designs and purchase necessary furniture and interior items on the spot. Furthermore, by utilizing an emotion engine, it can provide interior design suggestions optimized to the user's emotions.

[0907] (Example 2)

[0908] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0909] In modern interior design, it is extremely difficult for users to select numerous pieces of furniture and interior items themselves and then effectively combine them. This is especially true in online shopping, where finding the right items from a vast selection requires considerable time and effort. To solve this problem, a system is needed that allows users to easily take photos of their interiors and receive suggestions for optimal furniture and interior items. Furthermore, features that adjust suggestions based on the user's emotional state and optimize future suggestions based on purchase history are also crucial.

[0910] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0911] This invention includes a server that allows a user to take photos of interiors using a mobile device and transmit the photo data to the server; a server that analyzes the received photo data and proposes similar interior coordinations using a generation AI model; a server that searches for appropriate furniture and interior items from multiple online shopping sites based on the proposed coordinations and displays them in a list; a server that analyzes the user's emotional state using an emotion engine and adjusts the interior coordination suggestions; and a server that allows the user to purchase selected items from the displayed list online. This enables users to easily achieve optimal interior coordination, improve the quality of suggestions based on their emotional state during the process, and further optimize future suggestions based on their purchase history.

[0912] A "mobile device" is a device with communication capabilities that a user can carry with them, and specifically includes smartphones and tablets.

[0913] "Photo data" refers to image information captured using the camera function of a mobile device, and is digital data that includes the colors, shapes, and arrangement of the interior.

[0914] A "server" refers to a high-performance computer system used to process, store, and deliver data over a network.

[0915] A "generative AI model" refers to an artificial intelligence system that uses machine learning algorithms to extract features from photographic data and propose interior design ideas.

[0916] An "emotion engine" refers to a software module that analyzes a user's facial expressions, voice, and language data to determine the user's emotional state in real time.

[0917] "Interior coordination" refers to proposals that create an aesthetically pleasing and functional space by considering the layout, colors, and placement of furniture in a room.

[0918] An "online shopping site" refers to a website that allows users to purchase goods over the internet.

[0919] "Items" refer to purchasable products such as furniture and decorative items used in interior design.

[0920] "List view" refers to a display format that visually shows users multiple related items.

[0921] "Purchase process" refers to the process by which a user enters payment information for selected items and confirms the purchase through an online shopping site.

[0922] "Purchase history" refers to data that records items a user has purchased in the past and their detailed information.

[0923] "The quality of the proposal" refers to the usefulness and suitability of the interior design proposal, which is optimized for the user's needs and emotional state.

[0924] "Optimization" refers to the process of adjusting suggestions and search results based on certain criteria to find the most suitable state or result.

[0925] This invention begins with a user taking a photo of an interior using a mobile device and sending the photo data to a server. The captured photo data is stored on the server and passed to an AI module (generative AI model). The AI ​​module analyzes this photo data, extracts features such as color, shape, and arrangement, and generates similar interior coordinations. The server then searches for appropriate furniture and interior items from multiple online shopping sites based on the generated coordinations and displays them in a list.

[0926] Next, the emotion engine analyzes the user's emotional state. It analyzes the user's facial expressions, voice, and language data in real time to determine how the user feels about the suggestions. Based on this data, the server adjusts the interior design suggestions. If the user shows a positive reaction, it continues to suggest similar styles; if the reaction is negative, it suggests a different style.

[0927] Specific hardware components include mobile devices such as smartphones and tablets used by users, cloud servers for storing and analyzing data, high-performance computers for running generative AI models, and emotion engines for analyzing user emotions. Software components include applications for sending photo data, algorithms for running generative AI models, data collection programs from online shopping sites, and analysis software for the emotion engine. For example, cloud services such as Google Cloud and Amazon Web Services could be used.

[0928] As a concrete example, when a user takes a photo of their living room and presses the "upload" button in the app, the photo data is sent to a server and stored in the cloud. The server then passes the photo to an AI module, which extracts features such as white walls and wooden furniture. The AI ​​module then suggests a natural-style interior design. Next, an emotion engine analyzes the user's reaction, and if it detects that the user is pleased, it suggests additional arrangements and items in a similar style.

[0929] An example of a prompt message might be: "Based on the interior photos taken by the user, suggest the optimal coordination. Describe a system that uses an emotion engine to analyze the user's reactions, adjust the suggestions, and support the final purchase process. In particular, clearly define the roles of the server, terminal, and user."

[0930] This invention allows users to quickly and easily achieve optimal interior design, and the emotional engine improves the quality of suggestions, resulting in a highly satisfying purchasing experience.

[0931] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0932] Step 1:

[0933] The user takes a photo of the interior.

[0934] Input: An image of the interior taken by the user using the camera function of their smartphone.

[0935] Output: Captured image data.

[0936] Operation: The user takes a photo of their living room or bedroom with their smartphone. Pressing the "Capture" button generates the image data.

[0937] Step 2:

[0938] Users upload photos to the app.

[0939] Input: Image data stored on a smartphone.

[0940] Output: Request to send image data to the server.

[0941] Operation: The user launches the app, selects a photo they have taken, and presses the "Upload" button. The image data is sent to the cloud server via the internet.

[0942] Step 3:

[0943] The device sends photo data to the server.

[0944] Input: Image data specified by the user.

[0945] Output: Image data stored on a cloud server.

[0946] Operation: The mobile device uploads image data to a cloud server via the internet. The properly encoded data is sent to the server.

[0947] Step 4:

[0948] The server saves the photo data and passes it to the AI ​​module.

[0949] Input: Photo data uploaded to a cloud server.

[0950] Output: Data in a format that can be processed by the AI ​​module.

[0951] Operation: The server stores the received photo data in a database and performs format conversion to pass the data to the generating AI model.

[0952] Step 5:

[0953] An AI module analyzes photo data and generates interior design coordinates.

[0954] Input: Photo data passed to the AI ​​module.

[0955] Output: Proposed interior design.

[0956] Operation: The AI ​​module extracts features such as color, shape, and arrangement from a photograph and generates interior design coordinates based on these features. The generated coordinate data is returned to the server.

[0957] Step 6:

[0958] Search for items related to the server on online shopping sites.

[0959] Input: Suggested interior design data.

[0960] Output: Filtered search results.

[0961] Operation: The server searches for relevant furniture and interior items from multiple online shopping sites based on the suggested coordination data. It then refines the search results by applying filtering criteria such as color, size, and price.

[0962] Step 7:

[0963] The server organizes the search results and displays them to the user in a list.

[0964] Input: Filtered search results.

[0965] Output: A list of items displayed in the user's app.

[0966] Operation: The server selects the most suitable item and sends that information to the user's device. The device then displays the item's details in the app.

[0967] Step 8:

[0968] The emotion engine analyzes the user's emotional state.

[0969] Input: User's facial expressions, voice, and language data.

[0970] Output: Data regarding the user's emotional state.

[0971] Operation: The emotion engine analyzes the user's facial expressions and voice data to determine their emotional response to suggestions in real time. This data is then sent to the server.

[0972] Step 9:

[0973] The server adjusts suggestions based on sentiment data.

[0974] Input: User's emotional state data.

[0975] Output: Adjusted interior design suggestions.

[0976] Operation: The server adjusts interior design suggestions based on the user's emotional state as recognized by the emotion engine. It maintains a similar style based on positive responses and suggests a different style based on negative responses.

[0977] Step 10:

[0978] The user proceeds with the purchase process.

[0979] Input: Information about the item selected by the user.

[0980] Output: Purchase completed on the online shopping site.

[0981] Operation: The user selects items in the app and proceeds with the purchase process through the online shopping site. They enter payment information and confirm the purchase.

[0982] Step 11:

[0983] The server stores the purchase history and calculates the fees.

[0984] Input: Purchase completion information.

[0985] Output: Saved purchase history and calculated fees.

[0986] Operation: The server confirms that the purchase of the item has been completed and saves that information to the database. A coordination fee is calculated based on the purchase amount and monetized.

[0987] (Application Example 2)

[0988] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0989] Traditional interior design systems have faced challenges in improving user satisfaction because they only offer mechanical suggestions without considering the individual emotional state of the user. Furthermore, few systems offer the functionality to search and display suitable items from multiple online shopping sites simultaneously, requiring users to expend considerable effort to find appropriate items. Moreover, no interior design suggestion system existed that incorporated emotional analysis using user facial expressions and voice.

[0990] The identification processing by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for the user to take a picture of the interior using a mobile terminal and send the photo data to the server; means for the server to analyze the received photo data and suggest similar interior coordinations; means for the server to search for appropriate furniture and interior items from multiple online shopping sites based on the suggested coordinations and display them in a list; means for the user to purchase selected items from the displayed list online; means for detecting the user's emotional state using an emotion analysis device; and means for adjusting the suggested interior coordination based on the emotional state detected by the emotion analysis device. This makes it possible to suggest personalized interior coordinations that take into account the user's emotional state. Furthermore, since furniture and interior items can be searched and displayed from multiple online shopping sites at once, the user's effort is reduced, and they can find appropriate items more quickly and efficiently.

[0991] A "mobile device" refers to an electronic device that is portable and equipped with a camera function.

[0992] "Photo data" refers to image information of the interior taken with a mobile device.

[0993] A "server" refers to a computer device that is installed on the cloud via a network and is used for storing and analyzing data.

[0994] "Interior coordination" refers to the combination of furniture and decorative items proposed based on the colors, shapes, and arrangement of the interior shown in the photograph.

[0995] An "online shopping site" refers to a website where goods can be sold and purchased using the internet.

[0996] "Furniture and interior items" refer to items used to decorate or enhance the comfort of a room, specifically items such as sofas, tables, carpets, and lamps.

[0997] An "emotion analysis device" refers to a technological device that analyzes a user's facial expressions, voice, or language data in real time to detect the user's emotional state.

[0998] "Emotional state" refers to the results of analyzing a user's current emotions, such as joy, surprise, or dissatisfaction, based on facial expressions, voice, and other data.

[0999] "Adjustment" refers to modifying or changing the proposed content based on the user's individual emotional state and budget.

[1000] The process begins with the user taking photos of the interior using a mobile device such as a smartphone and uploading the photo data to a server via a dedicated application. It is assumed that the mobile device has a camera function.

[1001] When a device sends photo data to a server, the server receives and stores the data. The server is equipped with an AI module, which analyzes the received photo data. In the analysis process, features such as the color, shape, and arrangement of the subject are extracted, and based on these features, an interior design coordination is proposed.

[1002] Furthermore, the server searches for suitable furniture and interior items from multiple online shopping sites based on the proposed interior design. Search criteria include color, size, price, and more. The search results are filtered, and only the most suitable items are displayed to the user.

[1003] The emotion analysis device is also part of the system, analyzing facial expressions, voice, and language data in real time as the user interacts with the application. This device uses machine learning libraries such as TensorFlow and Keras to detect the user's emotional state. By analyzing the user's smiles and expressions of surprise when viewing suggested interior design, the server can adjust the interior design suggestions based on that emotional state.

[1004] For example, suppose a user takes a photo of their living room and uploads it to the server through the application. An AI module analyzes the photo and suggests a modern-style interior design. If the user smiles upon seeing the suggestion, an emotion analyzer detects the user's positive emotion. Based on this, the server will either continue suggesting a similar style or offer more in-depth suggestions. On the other hand, if the user shows expressions of surprise or dissatisfaction, the emotion analyzer detects that negative emotion, and the server will try suggesting a different style, such as a natural style.

[1005] This interior design assistant system allows users to receive personalized interior design suggestions. Furthermore, by providing information from online shopping sites in one place, it expands users' options and allows them to find suitable items quickly and efficiently.

[1006] Example of a prompt

[1007] The user takes a photo of their living room and uploads it through the app. AI analyzes the room's colors and furniture arrangement. It suggests a modern style coordination, and if the user smiles at it, it continues with the same style. If the user looks surprised or dissatisfied, it suggests a different style.

[1008] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[1009] Step 1:

[1010] The user takes a photo of the interior using their mobile device and sends the photo data to a server via an application. The device receives the photo data as input and uploads it to the server via the internet. Specifically, this involves pressing the "upload" button in the application. The photo data is then sent to the server as output.

[1011] Step 2:

[1012] The server analyzes the received photo data. In this analysis process, the server uses an AI module to extract features such as color, shape, and arrangement from the photo data. The input is the received photo data, and the output is feature data. Specific operations include feature extraction using image analysis algorithms.

[1013] Step 3:

[1014] The server proposes interior design based on the analysis results. The server generates similar interior design ideas based on extracted feature data. The input is feature data, and the output is a proposed design. The specific operation includes a process of generating the optimal design using an AI model.

[1015] Step 4:

[1016] The server searches for suitable furniture and interior items from multiple online shopping sites based on the generated coordination suggestions. The server uses the coordination suggestions as input and generates search results as output. Specific operations include searching and data filtering using the APIs of each online shopping site.

[1017] Step 5:

[1018] The server sends data to the mobile device to display a list of filtered items. The server takes search results as input and sends data for display to the device as output. Specific operations include data format conversion and data transmission based on communication protocols.

[1019] Step 6:

[1020] The device displays a list of items to the user. The device uses data received from the server as input and displays the item list to the user as output. Specific operations include UI rendering.

[1021] Step 7:

[1022] The emotion analysis device detects the user's emotional state in real time. The user's facial expressions and voice data are used as input, and the detected emotional state is generated as output. The specific operation includes a process of data capture using a camera and microphone, and the application of an emotion analysis algorithm.

[1023] Step 8:

[1024] The server adjusts outfit suggestions based on the user's emotional state. The server takes emotional state data as input and generates adjusted outfit suggestions as output. Specific operations include data re-analysis and updating of suggestions to reflect the emotional state.

[1025] Step 9:

[1026] The server generates a list of the final coordinated items and sends this information to the mobile device. The server takes the coordinated coordination proposals as input and generates the data to send to the device as output. Specific operations include data format conversion and data transmission based on communication protocols.

[1027] Step 10:

[1028] The device displays the final item list to the user, and the user proceeds with the online purchase of selected items. The device takes the final item list as input and the user's selection and purchase process as output. Specific operations include UI rendering, user interaction processing, and sending purchase data to the online shopping site.

[1029] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[1030] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1031] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[1032] [Fourth Embodiment]

[1033] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[1034] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1035] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1036] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[1037] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[1038] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[1039] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[1040] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[1041] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[1042] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1043] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1044] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[1045] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1046] The system for implementing this invention begins with the user taking a photo of the interior using a mobile device and sending the photo data to a server. Based on this photo data, the server suggests interior design ideas and then searches for and displays appropriate furniture and interior items from multiple online shopping sites. The user can then select items from the displayed list and purchase them on the spot. Each process is as follows:

[1047] User actions

[1048] The user first uses the camera function of their smartphone or other mobile device to take pictures of interior design pages in magazines or of their own home interior. Then, they use the app to upload the photos they have taken. For example, a user might take a picture of their living room and press the "Upload" button in the app.

[1049] Terminal operation

[1050] When the user presses the upload button, the device sends the selected photo data to the server. The photo data is uploaded to a cloud server via the internet.

[1051] Server analysis

[1052] The server passes the received photo data to an AI algorithm for analysis. The AI ​​algorithm extracts features such as color, shape, and arrangement from the photo and matches them with interior styles stored in a database. This allows the server to generate interior design ideas similar to the photo data. For example, based on the photo analysis, it might suggest a modern style interior.

[1053] Searching online shopping sites

[1054] The server searches for relevant furniture and interior items from multiple online shopping sites based on the proposed coordination. Criteria such as color, size, and price are used. The search results are filtered to select only the items best suited to the user.

[1055] List of items

[1056] The server organizes the search results into a list format and sends it to the user's device. The user can then select the items they wish to purchase from this list. For example, a sofa, table, and carpet that fit the suggested modern style might be displayed.

[1057] Purchase procedure

[1058] Once the user selects an item, the device proceeds with the purchase process via the online shopping site. The user can then enter their payment information and confirm the purchase.

[1059] Monetizing coordination fees

[1060] The server confirms that the purchase of the product is complete and calculates a coordination fee based on the purchase amount. For example, if the purchase amount is 100,000 yen, a fee of 1,000 yen will be generated as revenue.

[1061] This invention provides a system that allows users to easily realize interior design coordination and purchase items on the spot. Furthermore, the server can accumulate data based on the user's purchasing behavior, enabling it to further optimize future coordination suggestions.

[1062] The following describes the processing flow.

[1063] Step 1:

[1064] Users use their smartphones to take photos of interior design pages in magazines or of their own home interiors. This allows users to create photo data that serves as the basis for their interior design ideas.

[1065] Step 2:

[1066] The user taps the "Upload" button within the app, selects the photos they have taken, and sends them to the server. The device accepts this action and uploads the photo data to the server via the internet.

[1067] Step 3:

[1068] The server receives and saves the photo data, then passes it to the AI ​​module. The AI ​​module extracts features such as color, shape, and arrangement from the photo, and performs analysis to generate similar interior design ideas based on that data.

[1069] Step 4:

[1070] The server suggests similar interior styles based on the AI ​​analysis results. The suggestions are sent to the user's device, and the user can view the suggested coordination within the app.

[1071] Step 5:

[1072] The server uses the characteristics of the proposed coordination to call APIs from multiple online shopping sites to search for suitable furniture and interior items. This is done using specified criteria such as color, size, and price.

[1073] Step 6:

[1074] The server filters search results obtained from online shopping sites and selects the most suitable items. This selection is based on which items best fit the suggested outfit.

[1075] Step 7:

[1076] The server generates a list of selected items and sends this information to the user's device. The user can then view this list in the app and choose the items they wish to purchase.

[1077] Step 8:

[1078] The user selects the item they want to purchase from the list and taps the purchase button within the app. The device accepts this action and proceeds with the purchase process using the online shopping site's API.

[1079] Step 9:

[1080] The user enters their payment information and taps the purchase button. The device sends this information to the online shopping site and confirms the purchase.

[1081] Step 10:

[1082] The server confirms that the purchase process is complete and calculates a coordination fee based on the purchase amount. The calculated fee is then monetized and recorded in the system.

[1083] This system will enable users to quickly and easily coordinate their interiors and purchase necessary furniture and interior items on the spot.

[1084] (Example 1)

[1085] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1086] Traditional interior design proposal and purchasing systems require users to search for suitable items themselves, a process that is cumbersome and time-consuming. Furthermore, finding items that fit the user's budget and existing interior style is difficult, hindering efficient coordination proposals. Additionally, the purchasing process requires users to visit multiple online shopping sites individually, resulting in a poor user experience. A more efficient and user-friendly system is needed to address these challenges.

[1087] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[1088] In this invention, the server includes means for a user to take a photograph of an interior using a mobile terminal and transmit the photographic data to the server; means for the terminal to transmit the photographic data selected by the user to the server; means for the server to pass the received photographic data to a generating AI model for analysis and extract features such as color, shape, and arrangement; means for the server to compare the analysis results with interior styles stored in a database and suggest similar interior coordinations; means for the server to search for appropriate furniture and interior items from multiple online shopping sites based on the suggested coordinations, filter them using criteria such as color, size, and price, and display them in a list; and means for the user to purchase selected items from the displayed list online. As a result, the user can efficiently receive interior coordination suggestions and complete the entire purchase process in a consistent manner without having to perform cumbersome search operations.

[1089] A "mobile device" is an electronic device that a user can carry with them and that has the ability to take pictures and communicate with the internet.

[1090] "Photo data" refers to digital image files taken by a user using the camera function of their mobile device.

[1091] A "server" is a computing system that exchanges data with multiple terminals via a network and performs various calculations and data management.

[1092] A "generative AI model" refers to an artificial intelligence algorithm that analyzes interior design photo data and extracts features such as color, shape, and arrangement.

[1093] "Interior coordination" refers to the proposal of arranging and selecting furniture and decorative items based on a specific interior style.

[1094] An "online shopping site" is an e-commerce platform that allows users to search for, select, and purchase products over the internet.

[1095] "Filtering" is the process of selecting data and information based on specific criteria and extracting only what is necessary.

[1096] "List view" refers to displaying multiple selected items or data in a visually clear and easy-to-understand format.

[1097] The "purchase process" refers to the series of processes related to ordering and paying for selected products.

[1098] "Coordination fee" refers to the fee charged for interior design services, which is based on the price of the items purchased by the user.

[1099] The system for carrying out the present invention begins with a user taking a photograph of the interior using a mobile device and sending the photographic data to a server. This system includes the following main components:

[1100] 1. User's device

[1101] Users take photos of their interiors using mobile devices such as smartphones. They can then upload the photos to a server using a dedicated application. A specific example would be a user taking a photo of their living room and pressing the "Upload" button within the app.

[1102] 2. Device operation

[1103] When the user presses the upload button, the device sends the selected photo data to a server in the cloud. The photo data is uploaded to the server via the internet. During this process, the photo data is properly compressed and encrypted to ensure secure transmission.

[1104] 3. Server analysis

[1105] The server passes the received photo data to a generating AI model for analysis. This AI model extracts features such as color, shape, and arrangement from the photo and matches them with interior styles stored in a database. Based on this, the server generates interior design ideas that are similar to the photo data. For example, it might suggest a modern style design based on the photo analysis results.

[1106] 4. Searching online shopping sites

[1107] The server searches for relevant furniture and interior items from multiple online shopping sites based on the proposed coordination. Criteria such as color, size, and price are used, and the search results are filtered to show only the most suitable items.

[1108] 5. Display a list of items

[1109] The server organizes the search results into a list format and sends it to the user's device. The user can then select the items they wish to purchase from this list. For example, a sofa, table, and carpet that fit the suggested modern style might be displayed.

[1110] 6. Purchase Procedure

[1111] Once the user selects the items they wish to purchase, the device proceeds with the purchase process via the online shopping site. The user can then enter their payment information and confirm the purchase.

[1112] 7. Monetizing coordination fees

[1113] The server confirms that the purchase is complete and calculates a coordination fee based on the purchase amount. For example, if the purchase amount is 100,000 yen, a fee of 1,000 yen will be generated. This revenue is used for server operation and service improvements.

[1114] Examples of specific cases and prompt statements

[1115] For example, if a user is looking for a new dining table, they would first take a photo of their dining area and upload it to the server via the app. The server would then analyze the photo, suggest a modern style, and search for and list suitable dining table sets from multiple shopping sites. The user could then select an item from the list and complete the purchase process.

[1116] Example of a prompt:

[1117] "I've taken and uploaded photos of the dining area's interior. Please use this to suggest a modern style interior design and search for a suitable dining table set."

[1118] The flow of the specific processing in Example 1 will be explained using Figure 11.

[1119] Step 1:

[1120] The user takes a photo of the interior using their smartphone's camera function. The user presses the "Upload" button in the app and selects the photo data they have taken. The input is the photo data that has been taken, and the output is the photo data prepared within the app.

[1121] Step 2:

[1122] The device detects when the user presses the "upload" button and sends the selected photo data to a server in the cloud. The data is properly compressed and encrypted before transmission. The input is the photo data selected by the user, and the output is the photo data as it appears on the server.

[1123] Step 3:

[1124] The server analyzes the received photo data. First, it passes the photo data to a generative AI model, which extracts features such as color, shape, and arrangement. The input for this feature extraction is the photo data, and the output is the analyzed feature data.

[1125] Step 4:

[1126] The server matches the extracted feature data with interior styles stored in the database. Specifically, a generative AI model proposes interior design based on the analysis results. The input for this matching is feature data, and the output is proposed interior design data.

[1127] Step 5:

[1128] The server searches for suitable furniture and interior items from multiple online shopping sites based on the proposed interior design. Search criteria include color, size, and price. The input to this search is the interior design data, and the output is a filtered list of the best items.

[1129] Step 6:

[1130] The server organizes the search results into a list format and sends it to the user's device. The user can then select the items they wish to purchase from this list. The input is a filtered list of items, and the output is the list of items displayed on the device.

[1131] Step 7:

[1132] Once the user selects the items they wish to purchase, the device proceeds with the purchase process via the online shopping site. The user enters their payment information and confirms the purchase. The input consists of the user's selections and payment information, while the output is information about the purchased items.

[1133] Step 8:

[1134] The server confirms that the user's purchase is complete and calculates a commission based on the purchase amount. For example, if the purchase amount is 100,000 yen, a commission of 1,000 yen will be generated. The input for this commission calculation is the purchase amount, and the output is the calculated commission.

[1135] (Application Example 1)

[1136] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1137] In modern brick-and-mortar stores, customers can see products in person, but finding the right interior design requires considering numerous options, which is time-consuming and laborious. Furthermore, the purchasing process tends to be complex. There is also a lack of mechanisms to optimize future design suggestions based on online purchase history. A system is needed to address these problems.

[1138] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[1139] In this invention, the server includes means for a user to take a photograph of interior decoration using an information and communication terminal and transmit the photographic data to the computer server; means for the computer server to analyze the received photographic data and suggest similar interior decorations; means for the computer server to search for appropriate decorative items and fixtures from multiple e-commerce sites based on the suggested decorations and display them in a list; means for the user to purchase selected items from the displayed list via electronic transaction; and means for suggesting the optimal coordination based on the photographed interior decorations in a physical store and searching for and displaying related products within the store. This makes it possible for users to improve their shopping experience in physical stores and to efficiently select and purchase products.

[1140] An "information and communication terminal" refers to an electronic device that allows a user to take photos and transmit that data.

[1141] "Interior decoration" refers to all decorative items and furniture applied to the indoor environment.

[1142] A "computer server" refers to a computing system that processes and analyzes received data and provides suggestions and search results.

[1143] "Similar interior decorations" refers to other decorative items and furniture that match the style and design desired by the user, based on the analyzed photographic data.

[1144] An "e-commerce site" refers to a website that facilitates the exchange of goods and services via the internet.

[1145] "Fixtures" refers to furniture, interior products, and other fixtures used for interior decoration.

[1146] "Electronic transactions" refer to the buying and selling of goods and services conducted using information and communication networks such as the internet.

[1147] A "physical store" refers to a business establishment that exists in a physical location and where users can actually visit to check and purchase products.

[1148] "Related products" refer to items that have complementary functions and designs, selected based on the suggested coordination.

[1149] The system for implementing this invention begins with the user taking a photograph of interior decoration with an information and communication terminal and transmitting the photographic data to a computer server. Based on this photographic data, the server proposes interior design ideas and further searches for and displays appropriate decorative items and fixtures from multiple e-commerce sites. The user can then select items from the displayed list and purchase them through electronic transactions. Each process is as follows:

[1150] User actions

[1151] The user first uses the camera function of their information and communication device to take pictures of the interior decorations in the physical store, and then uploads those photos through the application. For example, this might happen when a user is looking for a sofa to fit their living room in the furniture section of a physical store.

[1152] Terminal operation

[1153] The device sends the captured photo data to a computer server in response to user input. At this time, the data is uploaded to a cloud server via the internet.

[1154] Server analysis

[1155] The server uses an AI model built with TensorFlow to analyze the received photo data. This model analyzes features such as color, shape, and arrangement in the photos and matches them with similar interior design styles stored in a database. This allows the server to generate optimal interior design suggestions based on the photo data.

[1156] Searching for e-commerce sites

[1157] The server searches for relevant decorative items and fixtures from multiple e-commerce sites based on the proposed coordination. Filtering is performed based on criteria such as color, size, and price. API requests are made using libraries such as Requests for the search.

[1158] View a list of items and complete the purchase process.

[1159] The server organizes the search results into a list and sends it to the user's device. The user can then select items they wish to purchase from this list and proceed with the electronic transaction. Purchase history is also stored in a database and used to optimize future outfit suggestions.

[1160] Examples of applications in physical stores

[1161] For example, suppose a user is looking for a sofa to fit their living room in the furniture section of a home improvement store. The user uses the app to take a picture of the displayed sofa and uploads the photo data to the server. The server uses an AI model to analyze the photo and suggests related interior furnishings such as tables and carpets. The user then checks the availability of the suggested items at the physical store and proceeds with the purchase on the spot.

[1162] Example of a prompt

[1163] "Please suggest a living room decor. I've attached photos below. I'd like a modern-style sofa. Please also suggest a matching table and carpet."

[1164] This invention allows users to efficiently select and purchase interior goods at physical stores, as well as receive personalized coordination suggestions based on their online shopping history.

[1165] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[1166] Step 1:

[1167] Users take photos of interior decorations in physical stores using the camera on their information and communication terminals. The photos are saved to the device in common image formats such as JPEG and PNG. Then, the user opens the application, selects the photos, and presses the upload button.

[1168] Input: Photos of interior decorations

[1169] Output: Photo selection and upload button press by the application.

[1170] Step 2:

[1171] The device sends the photos selected by the user to a computer server in the cloud via the internet. An HTTP POST request is used for transmission, and the photo data is uploaded to the server in binary format.

[1172] Input: Photo data selected by the user by pressing the upload button.

[1173] Output: Photo data uploaded to a computer server in the cloud.

[1174] Step 3:

[1175] The server passes the received photo data to an AI model built with TensorFlow for analysis. The AI ​​model extracts features such as color, shape, and arrangement from the photo, and matches these features to interior styles in a database. Based on the feature information obtained from the AI ​​model, the server generates interior design ideas similar to the photo data.

[1176] Input: Uploaded photo data

[1177] Output: Analysis results by the AI ​​model, and generated interior design.

[1178] Step 4:

[1179] Based on the proposed interior design, the server searches for relevant decorative items and fixtures using the APIs of e-commerce sites. Specifically, it filters based on parameters such as color, size, and price to select the most suitable products from multiple e-commerce sites. The search is performed using libraries such as Python's requests library.

[1180] Input: Generated interior design and search parameters

[1181] Output: Product list obtained from e-commerce sites

[1182] Step 5:

[1183] The server organizes the search results in a list format and sends them to the user's device. The user can view the product list within the application and select the items they wish to purchase. The selected product information is then sent to the server via the application.

[1184] Input: Product list obtained from e-commerce site

[1185] Output: Product list and selected product information displayed on the user's device.

[1186] Step 6:

[1187] The server processes the purchase of the product selected by the user on the e-commerce site. Specifically, it receives the user's payment information and processes the purchase through the e-commerce site's API. Once the purchase is confirmed to be complete, the server saves the purchase history to the database.

[1188] Input: Selected product information and user's payment information

[1189] Output: Purchase completion confirmation and saved purchase history

[1190] Step 7:

[1191] The server optimizes future interior design suggestions based on accumulated purchase history. Purchase history data is fed back into a machine learning model, enabling more accurate design suggestions in subsequent analyses.

[1192] Input: Saved purchase history

[1193] Output: Optimized future interior design proposals

[1194] This will not only allow users to efficiently select and purchase interior goods at physical stores, but will also enable them to receive personalized coordination suggestions based on their online shopping history.

[1195] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[1196] This invention begins with a user taking a photo of an interior using a mobile device and sending the photo data to a server. Based on this photo data, the server proposes interior design ideas, and then searches for and displays appropriate furniture and interior items from multiple online shopping sites. Furthermore, by incorporating an emotion engine that recognizes the user's emotional state, the quality of the suggestions is improved. Each process is as follows:

[1197] User actions

[1198] The user first uses the camera function of their smartphone or other mobile device to take pictures of interior design pages in magazines or of their own home interior. Then, they use the app to upload the photos they have taken. For example, a user might take a picture of their living room and press the "Upload" button in the app.

[1199] Terminal operation

[1200] When the user presses the upload button, the device sends the selected photo data to the server. The photo data is uploaded to a cloud server via the internet.

[1201] Server analysis

[1202] The server stores the received photo data and passes it to the AI ​​module. The AI ​​module extracts features such as color, shape, and arrangement from the photo and performs analysis to generate similar interior design ideas based on that data.

[1203] Searching online shopping sites

[1204] The server searches for relevant furniture and interior items from multiple online shopping sites based on the proposed coordination. Criteria such as color, size, and price are used. The search results are filtered to select only the items best suited to the user.

[1205] How the emotion engine works

[1206] The emotion engine analyzes data such as facial expressions, voice, and language as the user interacts with the app to determine the user's emotional state in real time. For example, it can detect emotions such as a smile or surprise when a user sees interior design suggestions.

[1207] Adjusting the interior design

[1208] The server adjusts interior design suggestions based on the user's emotional state as recognized by the emotion engine. If the user reacts positively to a suggestion, it continues to suggest similar styles; if the user reacts negatively, it suggests a different style.

[1209] List of items

[1210] The server generates a list of the final selected items and sends this information to the user's device. The user can then view this list in the app and choose the items they wish to purchase. For example, the list might display sofas, tables, carpets, and other items that fit the suggested modern style.

[1211] Purchase procedure

[1212] Once the user selects an item, the device proceeds with the purchase process via the online shopping site. The user can then enter their payment information and confirm the purchase.

[1213] Monetizing coordination fees

[1214] The server confirms that the purchase of the item has been completed and calculates a coordination fee based on the purchase amount. For example, if the purchase amount is 100,000 yen, a fee of 1,000 yen will be generated.

[1215] This system reduces the burden on users and enables quick and easy interior design coordination. Furthermore, by utilizing an emotion engine, it can provide suggestions optimized to the user's emotions. This, in turn, increases user satisfaction and provides a better purchasing experience.

[1216] The following describes the processing flow.

[1217] Step 1:

[1218] Users take photos of their interiors using their smartphone's camera function. This allows users to create photo data that serves as the basis for their interior design ideas.

[1219] Step 2:

[1220] The user taps the "Upload" button within the app, selects the photos they have taken, and sends them to the server. The device accepts this action and uploads the photo data to the server via the internet.

[1221] Step 3:

[1222] The server receives and saves the photo data, then passes it to the AI ​​module. The AI ​​module extracts features such as color, shape, and arrangement from the photo, and performs analysis to generate similar interior design ideas based on that data.

[1223] Step 4:

[1224] The server suggests similar interior styles based on the AI ​​analysis results. The suggestions are sent to the user's device, and the user can view the suggested coordination within the app.

[1225] Step 5:

[1226] The server uses the characteristics of the suggested coordination to call APIs from multiple online shopping sites to search for suitable furniture and interior items. This is done using criteria such as color, size, and price.

[1227] Step 6:

[1228] The server filters search results obtained from online shopping sites and selects the most suitable items. This selection is based on which items best fit the suggested outfit.

[1229] Step 7:

[1230] The server generates a list of selected items and sends this information to the user's device. The user can then view this list in the app and choose the items they wish to purchase.

[1231] Step 8:

[1232] The device monitors user actions and uses an emotion engine to analyze the user's emotional state. This includes facial recognition and voice analysis.

[1233] Step 9:

[1234] The emotion engine analyzes the user's emotional state in real time and sends the emotional data to the server. The server then uses this data to adjust the interior design suggestions.

[1235] Step 10:

[1236] The server sends the adjusted proposal back to the user's device. The user reviews the new proposal and, if satisfied, proceeds to the next step.

[1237] Step 11:

[1238] The user selects the item they want to purchase from the list and taps the purchase button within the app. The device accepts this action and proceeds with the purchase process using the online shopping site's API.

[1239] Step 12:

[1240] The user enters their payment information and taps the purchase button. The device sends this information to the online shopping site and confirms the purchase.

[1241] Step 13:

[1242] The server confirms that the purchase process is complete and calculates a coordination fee based on the purchase amount. The calculated fee is then monetized and recorded in the system.

[1243] This system allows users to quickly and easily create interior designs and purchase necessary furniture and interior items on the spot. Furthermore, by utilizing an emotion engine, it can provide interior design suggestions optimized to the user's emotions.

[1244] (Example 2)

[1245] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1246] In modern interior design, it is extremely difficult for users to select numerous pieces of furniture and interior items themselves and then effectively combine them. This is especially true in online shopping, where finding the right items from a vast selection requires considerable time and effort. To solve this problem, a system is needed that allows users to easily take photos of their interiors and receive suggestions for optimal furniture and interior items. Furthermore, features that adjust suggestions based on the user's emotional state and optimize future suggestions based on purchase history are also crucial.

[1247] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[1248] This invention includes a server that allows a user to take photos of interiors using a mobile device and transmit the photo data to the server; a server that analyzes the received photo data and proposes similar interior coordinations using a generation AI model; a server that searches for appropriate furniture and interior items from multiple online shopping sites based on the proposed coordinations and displays them in a list; a server that analyzes the user's emotional state using an emotion engine and adjusts the interior coordination suggestions; and a server that allows the user to purchase selected items from the displayed list online. This enables users to easily achieve optimal interior coordination, improve the quality of suggestions based on their emotional state during the process, and further optimize future suggestions based on their purchase history.

[1249] A "mobile device" is a device with communication capabilities that a user can carry with them, and specifically includes smartphones and tablets.

[1250] "Photo data" refers to image information captured using the camera function of a mobile device, and is digital data that includes the colors, shapes, and arrangement of the interior.

[1251] A "server" refers to a high-performance computer system used to process, store, and deliver data over a network.

[1252] A "generative AI model" refers to an artificial intelligence system that uses machine learning algorithms to extract features from photographic data and propose interior design ideas.

[1253] An "emotion engine" refers to a software module that analyzes a user's facial expressions, voice, and language data to determine the user's emotional state in real time.

[1254] "Interior coordination" refers to proposals that create an aesthetically pleasing and functional space by considering the layout, colors, and placement of furniture in a room.

[1255] An "online shopping site" refers to a website that allows users to purchase goods over the internet.

[1256] "Items" refer to purchasable products such as furniture and decorative items used in interior design.

[1257] "List view" refers to a display format that visually shows users multiple related items.

[1258] "Purchase process" refers to the process by which a user enters payment information for selected items and confirms the purchase through an online shopping site.

[1259] "Purchase history" refers to data that records items a user has purchased in the past and their detailed information.

[1260] "The quality of the proposal" refers to the usefulness and suitability of the interior design proposal, which is optimized for the user's needs and emotional state.

[1261] "Optimization" refers to the process of adjusting suggestions and search results based on certain criteria to find the most suitable state or result.

[1262] This invention begins with a user taking a photo of an interior using a mobile device and sending the photo data to a server. The captured photo data is stored on the server and passed to an AI module (generative AI model). The AI ​​module analyzes this photo data, extracts features such as color, shape, and arrangement, and generates similar interior coordinations. The server then searches for appropriate furniture and interior items from multiple online shopping sites based on the generated coordinations and displays them in a list.

[1263] Next, the emotion engine analyzes the user's emotional state. It analyzes the user's facial expressions, voice, and language data in real time to determine how the user feels about the suggestions. Based on this data, the server adjusts the interior design suggestions. If the user shows a positive reaction, it continues to suggest similar styles; if the reaction is negative, it suggests a different style.

[1264] Specific hardware components include mobile devices such as smartphones and tablets used by users, cloud servers for storing and analyzing data, high-performance computers for running generative AI models, and emotion engines for analyzing user emotions. Software components include applications for sending photo data, algorithms for running generative AI models, data collection programs from online shopping sites, and analysis software for the emotion engine. For example, cloud services such as Google Cloud and Amazon Web Services could be used.

[1265] As a concrete example, when a user takes a photo of their living room and presses the "upload" button in the app, the photo data is sent to a server and stored in the cloud. The server then passes the photo to an AI module, which extracts features such as white walls and wooden furniture. The AI ​​module then suggests a natural-style interior design. Next, an emotion engine analyzes the user's reaction, and if it detects that the user is pleased, it suggests additional arrangements and items in a similar style.

[1266] An example of a prompt message might be: "Based on the interior photos taken by the user, suggest the optimal coordination. Describe a system that uses an emotion engine to analyze the user's reactions, adjust the suggestions, and support the final purchase process. In particular, clearly define the roles of the server, terminal, and user."

[1267] This invention allows users to quickly and easily achieve optimal interior design, and the emotional engine improves the quality of suggestions, resulting in a highly satisfying purchasing experience.

[1268] The flow of the specific processing in Example 2 will be explained using Figure 13.

[1269] Step 1:

[1270] The user takes a photo of the interior.

[1271] Input: An image of the interior taken by the user using the camera function of their smartphone.

[1272] Output: Captured image data.

[1273] Operation: The user takes a photo of their living room or bedroom with their smartphone. Pressing the "Capture" button generates the image data.

[1274] Step 2:

[1275] Users upload photos to the app.

[1276] Input: Image data stored on a smartphone.

[1277] Output: Request to send image data to the server.

[1278] Operation: The user launches the app, selects a photo they have taken, and presses the "Upload" button. The image data is sent to the cloud server via the internet.

[1279] Step 3:

[1280] The device sends photo data to the server.

[1281] Input: Image data specified by the user.

[1282] Output: Image data stored on a cloud server.

[1283] Operation: The mobile device uploads image data to a cloud server via the internet. The properly encoded data is sent to the server.

[1284] Step 4:

[1285] The server saves the photo data and passes it to the AI ​​module.

[1286] Input: Photo data uploaded to a cloud server.

[1287] Output: Data in a format that can be processed by the AI ​​module.

[1288] Operation: The server stores the received photo data in a database and performs format conversion to pass the data to the generating AI model.

[1289] Step 5:

[1290] An AI module analyzes photo data and generates interior design coordinates.

[1291] Input: Photo data passed to the AI ​​module.

[1292] Output: Proposed interior design.

[1293] Operation: The AI ​​module extracts features such as color, shape, and arrangement from a photograph and generates interior design coordinates based on these features. The generated coordinate data is returned to the server.

[1294] Step 6:

[1295] Search for items related to the server on online shopping sites.

[1296] Input: Suggested interior design data.

[1297] Output: Filtered search results.

[1298] Operation: The server searches for relevant furniture and interior items from multiple online shopping sites based on the suggested coordination data. It then refines the search results by applying filtering criteria such as color, size, and price.

[1299] Step 7:

[1300] The server organizes the search results and displays them to the user in a list.

[1301] Input: Filtered search results.

[1302] Output: A list of items displayed in the user's app.

[1303] Operation: The server selects the most suitable item and sends that information to the user's device. The device then displays the item's details in the app.

[1304] Step 8:

[1305] The emotion engine analyzes the user's emotional state.

[1306] Input: User's facial expressions, voice, and language data.

[1307] Output: Data regarding the user's emotional state.

[1308] Operation: The emotion engine analyzes the user's facial expressions and voice data to determine their emotional response to suggestions in real time. This data is then sent to the server.

[1309] Step 9:

[1310] The server adjusts suggestions based on sentiment data.

[1311] Input: User's emotional state data.

[1312] Output: Adjusted interior design suggestions.

[1313] Operation: The server adjusts interior design suggestions based on the user's emotional state as recognized by the emotion engine. It maintains a similar style based on positive responses and suggests a different style based on negative responses.

[1314] Step 10:

[1315] The user proceeds with the purchase process.

[1316] Input: Information about the item selected by the user.

[1317] Output: Purchase completed on the online shopping site.

[1318] Operation: The user selects items in the app and proceeds with the purchase process through the online shopping site. They enter payment information and confirm the purchase.

[1319] Step 11:

[1320] The server stores the purchase history and calculates the fees.

[1321] Input: Purchase completion information.

[1322] Output: Saved purchase history and calculated fees.

[1323] Operation: The server confirms that the purchase of the item has been completed and saves that information to the database. A coordination fee is calculated based on the purchase amount and monetized.

[1324] (Application Example 2)

[1325] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1326] Traditional interior design systems have faced challenges in improving user satisfaction because they only offer mechanical suggestions without considering the individual emotional state of the user. Furthermore, few systems offer the functionality to search and display suitable items from multiple online shopping sites simultaneously, requiring users to expend considerable effort to find appropriate items. Moreover, no interior design suggestion system existed that incorporated emotional analysis using user facial expressions and voice.

[1327] The identification processing by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for the user to take a picture of the interior using a mobile terminal and send the photo data to the server; means for the server to analyze the received photo data and suggest similar interior coordinations; means for the server to search for appropriate furniture and interior items from multiple online shopping sites based on the suggested coordinations and display them in a list; means for the user to purchase selected items from the displayed list online; means for detecting the user's emotional state using an emotion analysis device; and means for adjusting the suggested interior coordination based on the emotional state detected by the emotion analysis device. This makes it possible to suggest personalized interior coordinations that take into account the user's emotional state. Furthermore, since furniture and interior items can be searched and displayed from multiple online shopping sites at once, the user's effort is reduced, and they can find appropriate items more quickly and efficiently.

[1328] A "mobile device" refers to an electronic device that is portable and equipped with a camera function.

[1329] "Photo data" refers to image information of the interior taken with a mobile device.

[1330] A "server" refers to a computer device that is installed on the cloud via a network and is used for storing and analyzing data.

[1331] "Interior coordination" refers to the combination of furniture and decorative items proposed based on the colors, shapes, and arrangement of the interior shown in the photograph.

[1332] An "online shopping site" refers to a website where goods can be sold and purchased using the internet.

[1333] "Furniture and interior items" refer to items used to decorate or enhance the comfort of a room, specifically items such as sofas, tables, carpets, and lamps.

[1334] An "emotion analysis device" refers to a technological device that analyzes a user's facial expressions, voice, or language data in real time to detect the user's emotional state.

[1335] "Emotional state" refers to the results of analyzing a user's current emotions, such as joy, surprise, or dissatisfaction, based on facial expressions, voice, and other data.

[1336] "Adjustment" refers to modifying or changing the proposed content based on the user's individual emotional state and budget.

[1337] The process begins with the user taking photos of the interior using a mobile device such as a smartphone and uploading the photo data to a server via a dedicated application. It is assumed that the mobile device has a camera function.

[1338] When a device sends photo data to a server, the server receives and stores the data. The server is equipped with an AI module, which analyzes the received photo data. In the analysis process, features such as the color, shape, and arrangement of the subject are extracted, and based on these features, an interior design coordination is proposed.

[1339] Furthermore, the server searches for suitable furniture and interior items from multiple online shopping sites based on the proposed interior design. Search criteria include color, size, price, and more. The search results are filtered, and only the most suitable items are displayed to the user.

[1340] The emotion analysis device is also part of the system, analyzing facial expressions, voice, and language data in real time as the user interacts with the application. This device uses machine learning libraries such as TensorFlow and Keras to detect the user's emotional state. By analyzing the user's smiles and expressions of surprise when viewing suggested interior design, the server can adjust the interior design suggestions based on that emotional state.

[1341] For example, suppose a user takes a photo of their living room and uploads it to the server through the application. An AI module analyzes the photo and suggests a modern-style interior design. If the user smiles upon seeing the suggestion, an emotion analyzer detects the user's positive emotion. Based on this, the server will either continue suggesting a similar style or offer more in-depth suggestions. On the other hand, if the user shows expressions of surprise or dissatisfaction, the emotion analyzer detects that negative emotion, and the server will try suggesting a different style, such as a natural style.

[1342] This interior design assistant system allows users to receive personalized interior design suggestions. Furthermore, by providing information from online shopping sites in one place, it expands users' options and allows them to find suitable items quickly and efficiently.

[1343] Example of a prompt

[1344] The user takes a photo of their living room and uploads it through the app. AI analyzes the room's colors and furniture arrangement. It suggests a modern style coordination, and if the user smiles at it, it continues with the same style. If the user looks surprised or dissatisfied, it suggests a different style.

[1345] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[1346] Step 1:

[1347] The user takes a photo of the interior using their mobile device and sends the photo data to a server via an application. The device receives the photo data as input and uploads it to the server via the internet. Specifically, this involves pressing the "upload" button in the application. The photo data is then sent to the server as output.

[1348] Step 2:

[1349] The server analyzes the received photo data. In this analysis process, the server uses an AI module to extract features such as color, shape, and arrangement from the photo data. The input is the received photo data, and the output is feature data. Specific operations include feature extraction using image analysis algorithms.

[1350] Step 3:

[1351] The server proposes interior design based on the analysis results. The server generates similar interior design ideas based on extracted feature data. The input is feature data, and the output is a proposed design. The specific operation includes a process of generating the optimal design using an AI model.

[1352] Step 4:

[1353] The server searches for suitable furniture and interior items from multiple online shopping sites based on the generated coordination suggestions. The server uses the coordination suggestions as input and generates search results as output. Specific operations include searching and data filtering using the APIs of each online shopping site.

[1354] Step 5:

[1355] The server sends data to the mobile device to display a list of filtered items. The server takes search results as input and sends data for display to the device as output. Specific operations include data format conversion and data transmission based on communication protocols.

[1356] Step 6:

[1357] The device displays a list of items to the user. The device uses data received from the server as input and displays the item list to the user as output. Specific operations include UI rendering.

[1358] Step 7:

[1359] The emotion analysis device detects the user's emotional state in real time. The user's facial expressions and voice data are used as input, and the detected emotional state is generated as output. The specific operation includes a process of data capture using a camera and microphone, and the application of an emotion analysis algorithm.

[1360] Step 8:

[1361] The server adjusts outfit suggestions based on the user's emotional state. The server takes emotional state data as input and generates adjusted outfit suggestions as output. Specific operations include data re-analysis and updating of suggestions to reflect the emotional state.

[1362] Step 9:

[1363] The server generates a list of the final coordinated items and sends this information to the mobile device. The server takes the coordinated coordination proposals as input and generates the data to send to the device as output. Specific operations include data format conversion and data transmission based on communication protocols.

[1364] Step 10:

[1365] The device displays the final item list to the user, and the user proceeds with the online purchase of selected items. The device takes the final item list as input and the user's selection and purchase process as output. Specific operations include UI rendering, user interaction processing, and sending purchase data to the online shopping site.

[1366] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[1367] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1368] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[1369] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1370] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[1371] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[1372] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[1373] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated based, for example, on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[1374] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[1375] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[1376] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[1377] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[1378] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[1379] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1380] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[1381] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[1382] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[1383] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[1384] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[1385] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[1386] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted as being incorporated by reference.

[1387] The following is further disclosed regarding the embodiments described above.

[1388] (Claim 1)

[1389] A means for a user to take a photo of the interior using a mobile device and send that photo data to a server,

[1390] The aforementioned server analyzes the received photo data and provides a means for suggesting similar interior design ideas.

[1391] The aforementioned server has a means of searching for and displaying in a list appropriate furniture and interior items from multiple online shopping sites based on the proposed coordination.

[1392] A means by which the aforementioned user can purchase online the product they selected from the displayed list of items,

[1393] A system that includes this.

[1394] (Claim 2)

[1395] The system according to claim 1, comprising means for adjusting the interior design proposed by the server based on the user's budget.

[1396] (Claim 3)

[1397] The system according to claim 1, comprising means for the server to proceed with the purchase procedure of a product selected by the user on an online shopping site, and means for optimizing future outfit suggestions based on the purchase history.

[1398] (Claim 4)

[1399] The system according to claim 1, which includes means for the server to calculate a coordination fee based on the purchase amount and to monetize the said fee.

[1400] "Example 1"

[1401] (Claim 1)

[1402] A means for a user to take a photo of the interior using a mobile device and send that photo data to a server,

[1403] The terminal provides means for transmitting photo data selected by the user to the server,

[1404] The aforementioned server passes the received photographic data to a generating AI model for analysis and extracts features such as color, shape, and arrangement.

[1405] The server, based on the analysis results, compares them with interior styles stored in the database and provides means for suggesting similar interior coordinations.

[1406] The aforementioned server searches for appropriate furniture and interior items from multiple online shopping sites based on the proposed coordination, filters them using criteria such as color, size, and price, and displays them in a list.

[1407] A means by which the aforementioned user can purchase online the product they selected from the displayed list of items,

[1408] A system that includes this.

[1409] (Claim 2)

[1410] The system according to claim 1, comprising means for adjusting the interior design proposed by the server based on the user's budget.

[1411] (Claim 3)

[1412] The system according to claim 1, comprising means for the server to proceed with the purchase procedure of a product selected by the user on an online shopping site, and means for optimizing future outfit suggestions based on the purchase history.

[1413] "Application Example 1"

[1414] (Claim 1)

[1415] A means for a user to take a photograph of interior decoration using an information and communication terminal and transmit the photographic data to a computer server,

[1416] The computer server analyzes the received photographic data and provides means for suggesting similar interior decorations,

[1417] The aforementioned computer server has means for searching for and displaying in a list appropriate decorative items and fixtures from multiple e-commerce sites based on the proposed decorations,

[1418] A means by which the aforementioned user can purchase items selected from a list of displayed items via electronic transaction,

[1419] In a physical store, a system is in place to suggest the optimal coordination based on photographs of interior decorations, and to search for and display related products within the store.

[1420] A system that includes this.

[1421] (Claim 2)

[1422] The system according to claim 1, comprising means for adjusting the interior decoration proposed by the computer server based on the user's budget.

[1423] (Claim 3)

[1424] The system according to claim 1, comprising means for the computer server to proceed with the purchase procedure of a product selected by the user on an e-commerce site, and means for optimizing future decoration suggestions based on the purchase history.

[1425] "Example 2 of combining an emotion engine"

[1426] (Claim 1)

[1427] A means for a user to take a photo of the interior using a mobile device and send that photo data to a server,

[1428] The aforementioned server analyzes the received photo data and proposes similar interior design using a generating AI model.

[1429] The aforementioned server has a means of searching for and displaying in a list appropriate furniture and interior items from multiple online shopping sites based on the proposed coordination.

[1430] A means for analyzing the user's emotional state using the aforementioned emotion engine and adjusting the interior design proposals accordingly,

[1431] A means by which the aforementioned user can purchase online the product they selected from the displayed list of items,

[1432] A system that includes this.

[1433] (Claim 2)

[1434] The system according to claim 1, comprising means for adjusting the interior design proposed by the server based on the user's budget.

[1435] (Claim 3)

[1436] The system according to claim 1, comprising means for the server to proceed with the purchase procedure of a product selected by the user on an online shopping site, and means for optimizing future outfit suggestions based on the purchase history.

[1437] "Application example 2 when combining with an emotional engine"

[1438] (Claim 1)

[1439] A means for a user to take a photo of the interior using a mobile device and send that photo data to a server,

[1440] The aforementioned server analyzes the received photo data and provides a means for suggesting similar interior design ideas.

[1441] The aforementioned server has a means of searching for and displaying in a list appropriate furniture and interior items from multiple online shopping sites based on the proposed coordination.

[1442] A means by which the aforementioned user can purchase online the product they selected from the displayed list of items,

[1443] A means for detecting a user's emotional state using an emotion analysis device,

[1444] Means for adjusting the proposed interior design based on the emotional state detected by the emotion analysis device,

[1445] A system that includes this.

[1446] (Claim 2)

[1447] A means for adjusting the interior design proposed by the aforementioned server based on the user's budget,

[1448] The emotion analysis device includes means for analyzing data such as the user's facial expressions, voice, and language in real time.

[1449] The system according to claim 1.

[1450] (Claim 3)

[1451] The aforementioned server provides a means for the user to proceed with the purchase process of the selected product on an online shopping site,

[1452] A means to optimize future outfit suggestions based on purchase history,

[1453] The means includes updating the suggested content based on the user's emotions detected by the emotion analysis device.

[1454] The system according to claim 1. [Explanation of Symbols]

[1455] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. A means for a user to take a photo of the interior using a mobile device and send that photo data to a server, The aforementioned server analyzes the received photo data and provides a means for suggesting similar interior design ideas. The aforementioned server has a means of searching for and displaying in a list appropriate furniture and interior items from multiple online shopping sites based on the proposed coordination. A means by which the aforementioned user can purchase online the product they selected from the displayed list of items, A system that includes this.

2. The system according to claim 1, which includes means for adjusting the interior design proposed by the server based on the user's budget.

3. The system according to claim 1, comprising means for the server to proceed with the purchase procedure of a product selected by the user on an online shopping site, and means for optimizing future outfit suggestions based on the purchase history.

4. The system according to claim 1, which includes means for the server to calculate a coordination fee based on the purchase amount and to monetize the said fee.

Images