System

Abstract

A system is provided.SOLUTION: This system includes a means for allowing a user to upload the image data of a room, a means for receiving the uploaded image data, and for performing image analysis, a means for selecting a product suitable for the taste and budget of the user based on the image analysis result, and a means for generating the proposal contents of the selected product, and for providing the user with the proposal contents.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] With traditional interior design tools, individual users lack specialized knowledge and find it difficult to design the ideal room in their home. It also takes time and effort to perform spatial analysis using room photos and floor plans, and to select products that fit the user's preferences and budget. Furthermore, finding the right product link requires searching multiple websites, which places a significant burden on users. This creates a need for a method that allows individual users to design interiors efficiently and effectively. [Means for solving the problem]

[0005] The present invention provides a system that includes a means for a user to upload image data of a room, a means for receiving the uploaded image data and performing image analysis, a means for selecting products that match the user's preferences and budget based on the image analysis results, and a means for generating and providing user recommendations for the selected products. This system allows users to solve their own interior design challenges. Specifically, the image analysis means grasps the characteristics of the room and detects the furniture arrangement, wall color, and amount of light, thereby enabling accurate interior design recommendations. Furthermore, the product selection means searches an external database and selects appropriate products based on the user's preferences and budget, allowing users to find their ideal interior items without any hassle.

[0006] "User" refers to an individual or corporation that uses the system to receive interior design proposals.

[0007] "Image data" refers to data containing visual information, such as photos of rooms or floor plans uploaded by users.

[0008] "Uploading means" refers to an interface and function that allows a user to send image data to the system.

[0009] The "means for receiving" refers to an interface and function for the server to obtain uploaded image data.

[0010] "Image analysis" refers to the technology and process of detecting room characteristics, furniture placement, wall color, amount of lighting, etc. from uploaded image data.

[0011] "Preferences" refer to interior style and color preferences specified by the user.

[0012] "Budget" refers to the maximum amount of money a user can spend on purchasing interior items.

[0013] "Means for selecting a product" refers to a function for searching and selecting a product that matches the user's preferences and budget from an external database or online shop.

[0014] "Means for generating proposal content" refers to the function of generating interior design proposals to be provided to users based on the image analysis results and product selection results.

[0015] The "means for providing" refers to a function for displaying the generated proposal content to the user and providing it to the user through an appropriate interface.

[0016] The term "system" refers to a set of components realized by combining the above means. [Brief explanation of the drawings]

[0017] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10]1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0018] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0019] First, the terms used in the following description will be explained.

[0020] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0021] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0022] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0023] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0024] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0025] [First embodiment]

[0026] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0027] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0028] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0029] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0030] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0031] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0032] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0033] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0034] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0035] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0036] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0037] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0038] The present invention begins with the user uploading image data of a room. Using a terminal connected to the Internet, the user sends photos and floor plans of the rooms in their home to the system. The user also operates an interface to input their preferred interior style and budget.

[0039] The device sends the image data, preferences, and budget information entered by the user to the server. The server temporarily stores the received data and performs preprocessing on the image data, including adjusting the resolution and converting the file format.

[0040] The server then loads the pre-processed image data into a multimodal AI system for image analysis, which detects characteristics such as the room size, furniture arrangement, wall color, window position, and amount of light, which are then stored in an internal database.

[0041] Based on the analysis results, the server applies the user's preferences and budget information and performs calculations to suggest an appropriate interior style and furniture arrangement. At this stage, the server accesses an external database (e.g., an online shopping site) to search for products that match the user's preferences and budget. Specific keyword search and filtering techniques are used to select products.

[0042] The server selects appropriate products from the search results and obtains detailed information (price, images, and purchase links) for each product. Based on this information, the server generates interior design suggestions suitable for the user. Specifically, the suggestions include furniture placement, color combinations, and accessory selection.

[0043] The generated proposals are sent to the device in HTML or JSON format. The device displays the received proposals on a user interface. The user can check the proposed interior designs and click on a product they like to be taken to an online shopping site that offers that product and complete the purchase process.

[0044] For example, if a user uploads a photo of their living room and inputs a natural style and a budget of 100,000 yen, the server will analyze the living room's characteristics (size, lighting conditions, etc.) and select furniture (wooden sofa, beige rug, etc.) and accessories (green potted plants, etc.) that are suitable for the natural style and within the user's budget. It will then provide an interior design proposal including specific placements for these items and links to purchase them.

[0045] This allows users to receive interior design suggestions based on their preferences and budget, and easily find and purchase the ideal interior products. This system solves users' interior design concerns and significantly reduces time and effort.

[0046] The processing flow will be explained below.

[0047] Step 1:

[0048] The user uploads photos and floor plans of the room using the device, and the device displays an interface for selecting the image data and inputting the user's preferred interior style (modern, natural, Scandinavian, etc.) and budget (e.g., 100,000 yen).

[0049] Step 2:

[0050] The device sends the uploaded image data, along with information about the user's preferences and budget, to the server, where it is packaged in an appropriate format and transmitted using a secure communication protocol.

[0051] Step 3:

[0052] The server temporarily stores the received data. The server then performs pre-processing on the received image data. This pre-processing includes adjusting the image resolution and converting the file format.

[0053] Step 4:

[0054] The server inputs the preprocessed image data into the multimodal AI for image analysis. Here, the AI ​​identifies the characteristics of the room. Specifically, it analyzes the room's size, furniture layout, wall color, window position, amount of lighting, etc. The results of this analysis are stored in the server's internal database.

[0055] Step 5:

[0056] The server performs calculations to propose an appropriate interior style based on the analysis results and the user's preferences. The calculations include furniture placement and color selection according to the characteristics of the room. The server searches an external database (such as an API of an online shopping site) to select appropriate interior items based on the user's preferences and budget.

[0057] Step 6:

[0058] The server filters the list of products retrieved from an external database and selects the best product that matches the user's criteria, taking into account factors such as price, design, and user reviews.

[0059] Step 7:

[0060] The server generates recommendations based on the selected product details (price, images, purchase links, etc.), including specific furniture placement, color schemes, and accessory selections.

[0061] Step 8:

[0062] The server sends the generated proposal to the device. The sent data is in HTML or JSON format and is organized in a format that the device can display appropriately.

[0063] Step 9:

[0064] The device displays the received proposals on a user interface, allowing the user to review the proposed interior designs and click on the desired product to view more information and a link to purchase it.

[0065] Step 10:

[0066] When the user clicks on the purchase link for the suggested product, the terminal will be redirected to the page of the corresponding online shopping site, where the user can purchase the product directly.

[0067] This process allows users to easily find interior designs and products that fit their tastes and budget, and smoothly move on to actual purchases.

[0068] Example 1

[0069] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0070] Conventional interior design suggestion systems require users to manually select products, making it easy for them to become overwhelmed by the vast amount of information, and making it difficult to efficiently find an interior design that suits their tastes and budget. Furthermore, insufficient image analysis can lead to the inability to accurately grasp the characteristics of a room and make appropriate suggestions. This can make it difficult for users to obtain satisfactory interior design suggestions, requiring a great deal of time and effort.

[0071] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0072] In this invention, the server includes means for users to upload image data of a space, means for receiving the uploaded image data and performing data preprocessing, means for analyzing the preprocessed image data and extracting features of the space, means for searching for and selecting products that match the user's preferences and budget based on the analysis results using a generative AI model, and means for generating and providing proposals for the selected products to the user. This allows users to easily obtain interior proposals that match their preferences and budget, significantly reducing time and effort.

[0073] "User" refers to an individual or group that operates the system and inputs image data of a room, preferences regarding the interior, and budget.

[0074] "Spatial image data" refers to digital images, including photographs and floor plans, of a user's own room or other space.

[0075] "Means for uploading" refers to an interface or function that allows a user to send image data or text information to a server via a terminal.

[0076] "Means for performing data preprocessing" refers to a function for performing preprocessing such as adjusting the resolution and converting the file format on received image data.

[0077] "Means for analyzing and extracting spatial characteristics" refers to the function of recognizing and identifying characteristics such as the size of a room, furniture arrangement, wall color, and amount of light based on preprocessed image data.

[0078] A "generative AI model" refers to an artificial intelligence model that can generate and analyze information from a variety of data.

[0079] "Means for searching and selecting products" refers to a function that uses an external database to find and select products that match the user's preferences and budget.

[0080] "Means for generating proposals and providing them to users" refers to the interface and functions that create interior proposals based on the selected products and present them to users.

[0081] "External Database" means a digital database containing product information and service data accessible on the Internet.

[0082] This invention is a system that uses a generative AI model to make appropriate interior design suggestions by allowing users to upload image data of their room and input their interior style and budget. The system begins by users accessing the system's web application using a device connected to the Internet.

[0083] Users upload photos and floor plans of their rooms via their devices, and input their preferred interior style and budget. For example, consider a case where a user uploads a photo of their living room and inputs a natural style and a budget of 100,000 yen. The user clicks the "Upload Image" button on the web interface to select a photo from their local device, and inputs their preferred style and budget using drop-down menus and text boxes.

[0084] The device sends the uploaded image data and the entered information to the server using the HTTP POST method. The server temporarily stores the received image data and performs preprocessing such as adjusting the resolution and converting the file format.

[0085] The server then passes the preprocessed image data to the multimodal AI, which uses deep learning models to identify characteristics such as room size, furniture placement, wall color, window position, and light intensity, and stores this data in an internal database.

[0086] Based on the analysis results, the server runs a recommendation algorithm using the user's preferences and budget information to calculate the appropriate interior style and furniture layout. During this process, the server sends an API request to an external database to search for products that match the user's preferences and budget. An example of an external database is the API of an online shopping site. In this process, keyword searches and price filtering are used to retrieve appropriate product information. The retrieved information includes the product's image URL, price, and purchase link.

[0087] Based on this information, the server creates interior design proposals, including specific furniture placement locations, color combinations to be used, and accessory selections, and generates data to provide to the user in HTML or JSON format.

[0088] The server sends the generated interior design proposals to the user's device. The device analyzes the received HTML and JSON data and displays the proposals on a user interface. The user can check the displayed interior designs and click on a product they like to be taken to the online shopping site that provides the product and complete the purchase process.

[0089] For example, by inputting the following prompt sentence into a generative AI model, interior design suggestions based on the user's requirements can be obtained.

[0090] Prompt: Analyze an image of a living room and create a natural-style interior design proposal that fits within a budget of ¥100,000. Attach an image of the living room, select items that are available within your budget, and suggest placement, color combinations, and accessory choices. Also provide a link to purchase each item.

[0091] This method allows users to easily receive interior design suggestions that match their preferences and budget, and also simplifies the purchasing process, making it possible to efficiently create the ideal interior.

[0092] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0093] Step 1:

[0094] Users use an internet-connected device to upload photos and floor plans of their rooms to the system, and also operate the interface to input their preferred interior style and budget.

[0095] Input: room image data, preferred interior style, budget

[0096] Output: The HTTP request that the user input is sent to the server.

[0097] Step 2:

[0098] The device sends the image data, preferences, and budget information entered by the user to the server using the HTTP POST method.

[0099] Input: User-entered image data, preferences, and budget

[0100] Output: Image data and user information arrive at the server as an HTTP request

[0101] Step 3:

[0102] The server temporarily stores the received data and performs pre-processing on the image data, including adjusting the resolution and converting the file format to JPEG or PNG.

[0103] Input: Image data received from the user

[0104] Output: Preprocessed image data (resolution adjusted, format converted)

[0105] Step 4:

[0106] The server then passes the preprocessed image data to a multimodal AI system that begins analyzing the image, using deep learning models to recognize characteristics such as the room's size, furniture placement, wall color, window position, and amount of light.

[0107] Input: Preprocessed image data

[0108] Output: Room characteristics data (area, furniture layout, wall color, window position, amount of light, etc.)

[0109] Step 5:

[0110] The server stores the results of the image analysis in an internal database.

[0111] Input: Room characteristic data

[0112] Output: Room characteristics data stored in a database

[0113] Step 6:

[0114] The server runs a recommendation algorithm based on the user's preferences and budget information to calculate the appropriate interior style and furniture layout. This calculation process involves sending API requests to an external database to search for products that match the user's preferences and budget.

[0115] Input: User preferences, budget information, room characteristics data

[0116] Output: Suitable interior style, furniture layout information, product candidates

[0117] Step 7:

[0118] The server selects appropriate products from the search results and retrieves detailed information for each product (price, image, purchase link).

[0119] Input: Product information as search results

[0120] Output: Selected product details (price, image, purchase link)

[0121] Step 8:

[0122] The server creates an interior design proposal, including specific furniture placement, color combinations, and accessory selection, and generates the proposal in HTML or JSON format.

[0123] Input: Selected product details, user preferences, budget information, room characteristics data

[0124] Output: Interior proposals in HTML or JSON format

[0125] Step 9:

[0126] The server sends the generated proposal to the user's device using an HTTP response.

[0127] Input: Interior proposal in HTML or JSON format

[0128] Output: HTTP response containing the proposal

[0129] Step 10:

[0130] The device parses the received suggestions and displays them in a user interface, while the front end parses the HTML and JSON data and renders images and text appropriately.

[0131] Input: HTTP response from the server (proposal)

[0132] Output: Interior proposals displayed on the user interface

[0133] Step 11:

[0134] The user can check the proposed interior designs and click on the product they like. When they click, the device opens a browser and goes to the corresponding product page on the online shopping site.

[0135] Input: Interior proposal displayed on the user interface

[0136] Output: Transition to online shopping site and start of purchase process

[0137] (Application example 1)

[0138] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0139] Many conventional interior design support systems analyze image data and propose products for a specific room in a user's home. However, providing a similar service in a physical store presents challenges, such as the unique size and characteristics of the store, as well as the appropriate placement of display items. This makes it difficult for store owners and interior designers to quickly determine effective product placement and design. The present invention aims to solve these problems and provide a system that easily proposes interior designs and product placements for physical stores.

[0140] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0141] In this invention, the server includes means for allowing a user to upload image data of a room, means for receiving the uploaded image data and performing image analysis, means for selecting products that match the user's preferences and budget based on the image analysis results, means for generating and providing suggestions for the selected products to the user, means for capturing and analyzing image data of the store, and means for suggesting display items for the store based on the image analysis results. This makes it easy to arrange and design products in a physical store in accordance with the user's preferences and budget.

[0142] A "user" is a person or organization that uses the system to receive proposals for interior design of a room or store.

[0143] A "living room" is a space in a residence where a user actually lives, and includes a room, a living room, a bedroom, etc.

[0144] "Image data" refers to digitally stored still or video images that contain visual information about a room or store.

[0145] "Upload" refers to the operation of a user sending image data from their own device to the system's server via the Internet.

[0146] "Means" refers to a method or component that enables a system or device to achieve a specific function or purpose.

[0147] "Receiving" refers to the process in which the server takes in image data sent by the user.

[0148] "Image analysis" is the process of analyzing received image data to extract characteristics of a room or store, furniture layout, wall color, amount of light, etc.

[0149] "Results" refers to the data or information obtained after performing image analysis.

[0150] "Preferences" refers to personal preference information such as interior style, color, and theme that the user has input into the system in advance.

[0151] "Budget" refers to the range of amounts set by the user for interior design and product purchases.

[0152] "Products" refers to furniture, decorations, and display items necessary for the interior design of a user's room or store.

[0153] "Selection" refers to the process of choosing products that suit the user's preferences and budget based on the results of image analysis.

[0154] "Proposal content" refers to the interior design and product placement plan shown to the user, including specific placement proposals and detailed information about the products to be used.

[0155] "Providing" refers to the process by which the system presents the suggestions to the user through the user interface.

[0156] The present invention is a system that allows users to upload image data of a room or a brick-and-mortar store, and then proposes interior designs and product layouts based on that data. This system involves a series of processes: sending image data from the user's device to a server, analyzing it, generating proposals, and providing them to the user.

[0157] First, the user captures image data of a room in their home or a brick-and-mortar store using a device and uploads it to the system. The device is equipped with a camera, and can be, for example, a smartphone or a head-mounted display (HMD). In addition to the image data, the user also enters their preferred interior style and budget information. This information is then sent to the server by the program and temporarily stored.

[0158] The server preprocesses the received image data, which includes adjusting the image resolution and converting the file format. To perform this processing, the server uses the OpenCV library.

[0159] The server then loads the preprocessed image data into a multimodal AI model for image analysis. The analysis results are converted into a dataset containing room or store characteristics (size, furniture arrangement, wall color, amount of light, etc.). The AI ​​model used here is a pre-trained generative AI model.

[0160] Based on the analysis results, the server accesses internal and external databases to search for products and display items that match the user's preferences and budget. Product selection uses specific keyword search and filtering techniques, which are implemented by the request library. For example, if a user enters a prompt such as "Please suggest items in a natural style that cost less than 100,000 yen," the server will perform a search based on this information.

[0161] The server selects appropriate products and display items from the search results and retrieves their detailed information (price, images, purchase links, etc.). Based on this, the server generates a suitable interior design proposal for the user, including specific furniture and display item placement, color combinations, accessory selection, etc.

[0162] The generated proposals are sent to the device in HTML or JSON format. The proposals are visually displayed on the user's device based on the received data, allowing the user to check the proposed interior design. By clicking on a product they like, they can be taken to the online shopping site that offers that product and complete the purchase process.

[0163] For example, if a user uploads a photo of their living room and enters "natural style with a budget of 100,000 yen," the server analyzes the characteristics of the living room and selects furniture (e.g., a wooden sofa, a beige rug) and accessories (e.g., green potted plants) suitable for the natural style within the user's budget. It then provides an interior design proposal, including specific placements for these items and links to purchase them.

[0164] This system allows users to easily receive store interior designs using their smartphones or HMDs, and quickly find and purchase the ideal interior products. This process is an effective way to resolve users' concerns about interior design and significantly reduce time and effort.

[0165] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0166] Step 1:

[0167] The user captures image data of a room or store using a terminal and uploads it to the system.

[0168] Inputs are image data captured by a camera, preferred interior style, and budget information.

[0169] In operation, the user takes an image using a smartphone or head-mounted display (HMD) and sends it to the server using the application's upload function.

[0170] The output is image data and user preference data sent to the server.

[0171] Step 2:

[0172] The server preprocesses the received image data.

[0173] The input is image data and preference information submitted by the user.

[0174] In operation, the server uses the OpenCV library to adjust the image resolution and convert it to the appropriate file format. This preprocessing makes the image suitable for analysis.

[0175] The output is pre-processed image data.

[0176] Step 3:

[0177] The server loads the preprocessed image data into a multimodal AI model and performs image analysis.

[0178] The input is the preprocessed image data.

[0179] In operation, the server inputs data into a multimodal AI model and performs image analysis, which extracts characteristics of the room or store (size, furniture arrangement, wall color, amount of light, etc.).

[0180] The output is characteristic data based on the analysis results.

[0181] Step 4:

[0182] Based on the image analysis results, the server selects products that suit the user's preferences and budget.

[0183] The inputs are the image analysis results data, user preferences, and budget information.

[0184] In operation, the server accesses internal and external databases to search for products and display items that meet the user's preferences and budget, using specific keyword search and filtering techniques.

[0185] The output is a list of selected products with their details (price, image, and purchase link).

[0186] Step 5:

[0187] The server generates and provides selected product and display item suggestions to the user.

[0188] The input is a product list and detailed information.

[0189] The server then sends the generated proposals in HTML or JSON format to the device, and creates an interior design including specific layouts and color combinations.

[0190] The output is the proposal sent to the user terminal.

[0191] Step 6:

[0192] The user checks the proposed content on the terminal, clicks on the product they like, and is taken to an online shopping site where they can complete the purchase.

[0193] The input is the proposal sent by the server.

[0194] The user visually checks the recommendations and browses detailed information about the recommended products. When the user clicks on a product they like, they are taken to a shopping site where they can complete the purchase process.

[0195] The output is to complete the purchase of the product.

[0196] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0197] The present invention provides a system that allows users to upload image data of their rooms and proposes interior design suggestions based on their preferences, budget, and emotional state. The image data uploaded by the user includes photos and floor plans of the rooms, which the user sends to the system through an interface. The user also inputs or has the system detect their preferred interior style, budget, and real-time emotional state.

[0198] When the device receives user input data, it sends it to the server. The server preprocesses the received image data (adjusting resolution, converting formats, etc.), and then uses multimodal AI to analyze the room's characteristics. This analysis includes the room's size, furniture layout, wall color, amount of light, etc. The analysis results are stored in an internal database.

[0199] Next, the device uses an emotion engine to recognize the user's emotional state. The emotion engine analyzes the user's input data and biometric data (facial expressions, voice tone, heart rate, etc.) to detect whether the user is feeling joy, surprise, sadness, anger, etc. The detected emotional state is also sent to the server.

[0200] The server generates appropriate interior style and furniture layout suggestions by taking into consideration the room's characteristics, the user's emotional state, as well as their preferences and budget. In this process, optimization is performed according to the user's emotional state; for example, if the user's emotional state is positive, suggestions for a bright and lively style are prioritized, whereas if the user's emotional state is negative, suggestions for a more subdued style are prioritized.

[0201] The server also searches external databases (such as the API of an online shopping site) to select interior products that meet the user's requirements, taking into consideration factors such as price, design, and user reviews. Detailed information about the selected products (price, images, purchase links, etc.) is also obtained.

[0202] Once the proposals are generated, the server sends them to the device. The device then displays the received proposals on its user interface. The user can review the displayed interior designs and view detailed information and purchase links for products they like. By clicking on a product, they can be taken directly to an online shopping site and complete the purchase process.

[0203] For example, suppose a user uploads a photo of their living room, specifies a preference for a natural style, and sets a budget of 100,000 yen. Furthermore, suppose the emotion engine recognizes the user's smile and detects a happy emotional state. The server analyzes the characteristics of the living room and suggests natural-style furniture and accessories (e.g., a wooden sofa or a beige rug). These suggestions are chosen to create a bright and cheerful atmosphere. Detailed information and purchase links for the selected products are also included, allowing the user to easily purchase them.

[0204] In this way, by using the system of the present invention, users can receive interior design suggestions that take into consideration their own emotional state, and easily find and purchase their ideal interior design items.

[0205] The processing flow will be explained below.

[0206] Step 1:

[0207] The user uploads image data of the room (e.g., a photo of the living room) to the system using a terminal. In addition, the user inputs their preferred interior style (e.g., natural) and budget (e.g., 100,000 yen) through the interface.

[0208] Step 2:

[0209] The device sends the uploaded image data, user preferences, and budget information to a server, where the data is packaged in a suitable format and transmitted using a secure communication protocol.

[0210] Step 3:

[0211] The server temporarily stores the received data. The server then preprocesses the received image data, adjusting the resolution and converting the format.

[0212] Step 4:

[0213] The server inputs the preprocessed image data into the multimodal AI for image analysis, which detects the room's characteristics (size, furniture arrangement, wall color, window position, amount of light, etc.). The analyzed characteristic data is stored in an internal database.

[0214] Step 5:

[0215] The device uses an emotion engine to recognize the user's emotional state. The emotion engine analyzes the user's biometric data (facial expressions, tone of voice, heart rate, etc.) to detect emotional states such as joy, surprise, sadness, and anger. The detected emotional data is sent to the server.

[0216] Step 6:

[0217] The server combines and analyzes the room's characteristic data, the user's preferences, budget, and emotional data to generate appropriate interior design suggestions. If the emotional state is positive, a bright and colorful interior style is prioritized, while if the emotional state is negative, a calm style is prioritized.

[0218] Step 7:

[0219] The server searches an external database (e.g., an API for an online shopping site) to select interior products that match the user's tastes and budget. The external database contains information such as prices, designs, and user reviews. Detailed information about the selected products (price, images, purchase links, etc.) is also retrieved.

[0220] Step 8:

[0221] The server generates interior design suggestions, including furniture placement, color schemes, and accessory selections based on the user's preferences, budget, and emotional state. The suggestions are formatted in HTML or JSON.

[0222] Step 9:

[0223] The server sends the generated proposal to the terminal, which displays the received data on the user interface, and the user checks the interior design displayed on the interface.

[0224] Step 10:

[0225] When the user clicks on the suggested product details or purchase link, the device will be redirected to the corresponding page on the online shopping site, where the user can purchase the product directly.

[0226] For example, if a user uploads a photo of their living room, selects a natural style and a budget of 100,000 yen, and the emotion engine detects the user's smile and recognizes their emotional state of joy, the server will suggest natural-style interior items that create a bright and cheerful atmosphere (e.g., a wooden sofa, a beige rug, etc.). These suggestions also include specific placement locations, color schemes, and accessories (e.g., green potted plants), allowing users to easily find and purchase the ideal interior items.

[0227] Example 2

[0228] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0229] Conventional interior design suggestion systems generally suggest interior styles based on the user's preferences and budget, but they are unable to take into account the user's real-time emotional state. As a result, they are unable to provide optimal interior suggestions that reflect the user's momentary psychological state. This makes it difficult to meet the diverse needs of users, and more personalized suggestions are needed.

[0230] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0231] In this invention, the server includes means for a user to upload image data of a room, means for adjusting the resolution of the image and converting the format, means for analyzing the room size, furniture layout, wall color, amount of light, etc., means for analyzing the user's biometric data to recognize the user's emotional state, means for generating suggestions for interior styles and furniture layouts based on the analysis results and the user's preferences, budget, and emotional state, means for searching an external database to obtain detailed information and purchase links for products that meet the user's requirements, and means for displaying the generated suggestions on a user interface. This enables personalized interior suggestions that take the user's real-time emotional state into consideration.

[0232] "User" refers to an individual who uses the system to receive room interior suggestions.

[0233] "Room image data" refers to image files such as room photos and floor plans that users upload to the system.

[0234] "Resolution adjustment" refers to the process of changing the resolution of an input image to an appropriate size.

[0235] "Format conversion" refers to the process of converting the file format of input image data into a format that can be used by the system.

[0236] "Room size" refers to information indicating the physical size of the room obtained from the analyzed image.

[0237] "Furniture arrangement" refers to information indicating the position and arrangement of furniture within a room obtained from the analyzed image.

[0238] "Wall color" refers to information indicating the color of the walls of a room obtained from the analyzed image.

[0239] "Amount of light" refers to information indicating the brightness of light in a room and the amount of light incident thereon, obtained from the analyzed image.

[0240] "Biometric data" refers to physiological data such as a user's facial expression, voice tone, heart rate, etc.

[0241] "Emotional state" refers to the emotion the user is feeling (such as joy, surprise, sadness, anger, etc.).

[0242] "Interior style" refers to interior design styles such as natural, modern, and classic.

[0243] "Furniture arrangement suggestions" refers to specific suggestions showing the optimal furniture arrangement and interior style for the user's room.

[0244] "External Database" refers to a database for obtaining product information from online shopping sites and other external resources.

[0245] "Detailed Information" refers to information about the selected product, including price, images, and purchase links.

[0246] "User interface" refers to the screen and operation method that allows a user to interact with a system.

[0247] The present invention is a system that allows users to upload image data of a room and receives interior design suggestions based on the user's preferences, budget, and real-time emotional state. Specific hardware and software used to implement the invention, as well as methods for processing and calculating data, and specific examples are described below.

[0248] Hardware and Software

[0249] A terminal is a computer or mobile device that provides a user interface and accepts data input from a user. The terminal has image processing software and an emotion engine installed.

[0250] The server is a computer system that processes and analyzes the received data. The following software is used for this analysis:

[0251] Use an image processing library (e.g., OpenCV, Pillow) to adjust image resolution and convert image formats.

[0252] A multimodal AI model (e.g., CNN) analyzes room characteristics (size, furniture arrangement, wall color, amount of light, etc.).

[0253] Recognize the user's emotional state with an emotion recognition library (e.g., OpenFace or other facial expression recognition libraries).

[0254] Data processing and calculation

[0255] When a user launches the system, an interface appears. The user uploads a photo of the room (e.g., a photo of the living room) and a floor plan, inputs the interior style (e.g., natural style) and budget (e.g., 100,000 yen), and provides facial expressions and voice data to detect emotional states.

[0256] The device sends the received data to the server, which then preprocesses the received image data, adjusting the resolution and converting the format, and analyzes the room size, furniture layout, wall color, amount of light, etc.

[0257] Based on this data, the server uses a generative AI model to generate optimal interior style and furniture arrangement suggestions tailored to the user's preferences, budget, and emotional state, with example prompts such as:

[0258] Prompt: "Picture of a living room, natural style, budget of ¥100,000, user smiling (happy)"

[0259] The server searches an external database to select interior products that meet the user's criteria. It selects the most suitable product by comprehensively considering factors such as price, design, and user reviews. It also obtains detailed information about the selected product (price, images, purchase links, etc.).

[0260] Finally, the server sends the generated recommendations to the terminal, which displays them on the user interface. The user can then check the recommendations, view detailed information and purchase links, and purchase the products they are interested in directly from the online shopping site.

[0261] For example, suppose a user uploads a photo of their living room, specifies a preference for a natural style, and sets a budget of 100,000 yen. Furthermore, suppose the emotion engine recognizes the user's smile and detects a happy emotional state. The server analyzes the characteristics of the living room and suggests natural-style furniture and accessories (e.g., a wooden sofa or a beige rug). These suggestions are chosen to create a bright and cheerful atmosphere. Detailed information and purchase links for the selected products are also included, allowing the user to easily purchase them.

[0262] In this way, by using the system of the present invention, users can receive interior design suggestions that take into consideration their own emotional state, and easily find and purchase their ideal interior design items.

[0263] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0264] Step 1: System startup and data entry

[0265] When the user boots the system, the interface is displayed.

[0266] Users upload photos of their rooms (e.g., photos of their living rooms) and floor plans.

[0267] The user inputs the interior style (e.g., natural style) and budget (e.g., 100,000 yen) through the interface.

[0268] The user provides facial expression and voice data to detect the emotional state, which then provides the emotional data.

[0269] Step 2: Receiving data and sending it to the server

[0270] The terminal receives input data from the user (room image data, preferences, budget, and emotional data).

[0271] The terminal sends the received data to the server. Input is the user's input data, which the terminal sends to the server.

[0272] Step 3: Preprocessing the image data

[0273] The server preprocesses the image data it receives, specifically adjusting the image resolution and converting the format.

[0274] The server uses the OpenCV and Pillow libraries for this process.

[0275] For example, downscaling a high-resolution image and converting it to an appropriate image format (JPEG or PNG). The input is the original image data and the output is the preprocessed image data.

[0276] Step 4: Analyzing the image data

[0277] The server uses the pre-processed image data to analyze the characteristics of the room.

[0278] The server uses a multimodal AI model (e.g., CNN) to determine the size of the room, furniture arrangement, wall color, amount of light, etc.

[0279] The analysis results are stored in an internal database. The input is preprocessed image data, and the output is the analysis results.

[0280] Step 5: Recognizing your emotional state

[0281] The device uses an emotion engine to recognize the user's emotional state, using data such as facial expressions, voice tone, and heart rate.

[0282] The device uses OpenFace and other facial recognition libraries for emotion recognition.

[0283] The emotional state is recognized and sent to the server. The input is the user's biometric data and the output is the emotional state data.

[0284] Step 6: Generate proposals

[0285] The server generates appropriate interior style and furniture arrangement suggestions taking into account the room's characteristics, the user's emotional state, preferences, and budget.

[0286] The server uses a generative AI model to create suggestions based on the input (prompt).

[0287] Prompt: "Picture of a living room, natural style, budget of ¥100,000, user smiling (happy)"

[0288] A proposal is generated. The inputs are the room characteristic data, the emotional state data, the user's preferences and budget, and the output is the generated proposal.

[0289] Step 7: Product Selection

[0290] The server searches an external database and selects interior products that meet the user's requirements.

[0291] The server selects the most suitable product by comprehensively considering price, design, user reviews, etc. Detailed information about the selected product (price, image, purchase link, etc.) is obtained.

[0292] The input is the user's conditions and proposal details, and the output is the selected product information.

[0293] Step 8: View the results

[0294] The server sends the generated proposal to the terminal.

[0295] The terminal displays the received proposal on a user interface.

[0296] Users can review the suggestions displayed, view detailed information and purchase links, and select the appropriate product.

[0297] The input is the proposal content and product information, and the output is the proposal and product information displayed on the user interface.

[0298] (Application example 2)

[0299] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0300] Conventional interior design suggestion systems can select interior products based on the user's preferences and budget, but they do not consider the user's emotional state when making suggestions. This makes it difficult to provide the optimal interior design that matches the user's psychological state. Furthermore, there is a lack of a way to easily access detailed information and purchase links on the spot.

[0301] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0302] In this invention, the server includes means for a user to upload image data of a room, means for receiving the uploaded image data and performing image analysis, means for selecting products that match the user's preferences, budget, and emotional state based on the image analysis results, means for generating proposals for the selected products and providing them to the user, means for analyzing the user's facial expression images to detect the user's emotional state, means including an emotion engine with the function of analyzing the emotional state in real time, and means for recommending interior styles according to the emotional state. This makes it possible to propose an optimal interior design that matches the user's real-time emotional state, and further allows the user to access detailed information and a purchase link on the spot.

[0303] "User" refers to an individual or organization that uses the system to upload image data of a room and receive interior design suggestions.

[0304] "Image data" refers to digital image files such as room photos and floor plans uploaded by users.

[0305] "Upload" refers to the act of a user sending image data from their device to a server.

[0306] "Image analysis" refers to the process by which the server preprocesses the image data it receives and analyzes the characteristics of the room.

[0307] "Preferences" refer to personal preferences regarding the interior style and design desired by the user.

[0308] "Budget" refers to the upper limit of costs set by the user when selecting interior products.

[0309] "Emotional state" refers to the psychological state of the user, which is analyzed using the emotion engine.

[0310] An "emotion engine" refers to an algorithm or software that analyzes a user's facial expression images and biometric data to detect their emotional state.

[0311] "Server" refers to a computer system that processes image data and other information received from users and generates and provides interior design proposals.

[0312] "Suggestions" refers to the interior design and product recommendations generated by the system.

[0313] "External databases" refer to online shopping sites and other information sources on the Internet that provide reference information for product selection.

[0314] "Product selection" refers to the process by which the system selects the most suitable interior products based on the user's preferences, budget, and emotional state.

[0315] "Real-time" refers to the process by which a system processes user input or detected information immediately and generates and delivers results.

[0316] This invention provides a system that allows a user to upload image data of a room and proposes interior design ideas based on the user's preferences, budget, and emotional state. The specific system design and operation are described below.

[0317] server

[0318] The server is a computer system that processes image data and other information received from users and generates and provides interior design proposals. The server includes the following means:

[0319] 1. A function to receive image data and perform preprocessing (resolution adjustment, format conversion, etc.).

[0320] 2. Room characteristic analysis function using multimodal AI (analyzing room size, furniture arrangement, wall color, amount of light, etc.).

[0321] 3. A function that uses an emotion engine to analyze the user's emotional state (analyzing facial expressions, voice, heart rate, etc. to detect joy, surprise, sadness, anger, etc.).

[0322] 4. A feature that suggests the best interior style and furniture arrangement based on the user's preferences, budget, and emotional state.

[0323] 5. The ability to search external databases (e.g., online shopping sites) and select appropriate interior products.

[0324] Terminal

[0325] A terminal is a device that allows users to access the system and input information. This terminal can be a smartphone, tablet, or PC. The specific operation steps are as follows:

[0326] 1. The user takes a photo of the room using the device and uploads the image data.

[0327] 2. The user inputs their preferred interior style and budget through the interface.

[0328] 3. The device captures the user's facial expressions and sends the data to the emotion engine in real time.

[0329] Processing Flow

[0330] Hardware and software configuration

[0331] Hardware: Smartphone (camera function, network connection), server (high performance computer).

[0332] Software: Python, PIL (image preprocessing), Emotion Engine (sentiment analysis), Interior Style Recommender (interior suggestions).

[0333] The server first preprocesses the image data of the room received from the user (adjusting the resolution, converting the format, etc.). It then uses multimodal AI to analyze the room's size, furniture arrangement, wall color, amount of light, etc. The analysis results are stored in a database on the server. At the same time, the device uses an emotion engine to analyze the user's emotional state, and this information is also sent to the server.

[0334] The server takes into account the room's characteristics, the user's emotional state, preferences, and budget to generate suggestions for appropriate interior styles and furniture arrangements. These suggestions also include product information (price, images, purchase links, etc.) retrieved from an external database. Once the suggestions are generated, the server sends them to the device and provides them to the user.

[0335] Examples and prompts

[0336] For example, if a user uploads a photo of their living room with a natural style and a budget of 100,000 yen, the system works as follows: The emotion engine recognizes the user's smile and detects a happy emotional state. The server analyzes the characteristics of the living room and suggests natural-style furniture and accessories (e.g., a wooden sofa, a beige rug). The suggestions are selected to create a bright and cheerful atmosphere.

[0337] Prompt Sentence Examples

[0338] "Upload a picture of your room: room.jpg"

[0339] "Please enter your preferred interior style: Natural"

[0340] "Please enter your budget: 100,000 yen"

[0341] "Upload a photo of your face: user.jpg"

[0342] This allows users to easily find and purchase their ideal interior items through specific interior suggestions.

[0343] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0344] Step 1:

[0345] The user uses the terminal to take a picture of the room and upload it.

[0346] Input: A photo of the room.

[0347] Output: The uploaded image data.

[0348] Specific operation: The user takes a photo of the room using the camera function of their smartphone, and then presses the upload button in the application to send the image data to the server.

[0349] Step 2:

[0350] The server pre-processes the image data received from the user.

[0351] Input: Uploaded image data.

[0352] Output: Preprocessed image data.

[0353] What it does: It adjusts the image resolution on the server and converts it to the required format (e.g. resize to 256x256 pixels). It uses Python and the PIL library for preprocessing.

[0354] Step 3:

[0355] The server analyzes the pre-processed image data to understand the characteristics of the room.

[0356] Input: Preprocessed image data.

[0357] Output: Analysis results such as room size, furniture placement, wall color, and amount of light.

[0358] Specific operation: A multimodal AI system on the server analyzes image data and extracts room characteristics. The extracted data is stored in an internal database.

[0359] Step 4:

[0360] The user inputs their preferred interior style and budget through the interface.

[0361] Input: The user's interior style preferences and budget.

[0362] Output: Interior style preferences, budget data.

[0363] What it does: Enter style and budget through a user interface using drop-down menus and text input forms.

[0364] Step 5:

[0365] The terminal uses an emotion engine to detect the user's emotional state.

[0366] Input: An image of the user's facial expression.

[0367] Output: Detected emotional state data.

[0368] Specific operation: The smartphone camera captures the user's facial expression and sends it to the server in real time. The emotion engine analyzes the facial expression data and detects the user's emotional state (e.g., joy, sadness, surprise, etc.).

[0369] Step 6:

[0370] The server suggests interior styles and furniture arrangements based on the room's characteristics, the user's preferences, budget, and emotional state.

[0371] Input: Room characteristic data, interior style preference, budget data, emotional state data.

[0372] Output: Interior design proposals (style, furniture arrangement, specific products).

[0373] What it does: The algorithm in the server generates the optimal interior design based on all input data, and then retrieves and finalizes the proposal from internal and external databases.

[0374] Step 7:

[0375] The server sends the selected product details and a link to purchase the product to the device.

[0376] Input: Interior design proposal.

[0377] Output: The suggestions that will be displayed on the user's device.

[0378] Specific operation: The server sends the generated interior design proposal to the device and displays detailed information (price, image, purchase link) on the interface.

[0379] Step 8:

[0380] The user reviews the offer and clicks the purchase link to purchase the product.

[0381] Input: Proposed interior product.

[0382] Output: Transition to online shopping site and purchase process.

[0383] Specific actions: The user checks the suggestions displayed on the device and clicks on the purchase link for the product they are interested in. They then complete the purchase at the linked online shopping site.

[0384] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0385] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0386] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0387] [Second embodiment]

[0388] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0389] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0390] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0391] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0392] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0393] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0394] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0395] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0396] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0397] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0398] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0399] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0400] The present invention begins with the user uploading image data of a room. Using a terminal connected to the Internet, the user sends photos and floor plans of the rooms in their home to the system. The user also operates an interface to input their preferred interior style and budget.

[0401] The device sends the image data, preferences, and budget information entered by the user to the server. The server temporarily stores the received data and performs preprocessing on the image data, including adjusting the resolution and converting the file format.

[0402] The server then loads the pre-processed image data into a multimodal AI system for image analysis, which detects characteristics such as the room size, furniture arrangement, wall color, window position, and amount of light, which are then stored in an internal database.

[0403] Based on the analysis results, the server applies the user's preferences and budget information and performs calculations to suggest an appropriate interior style and furniture arrangement. At this stage, the server accesses an external database (e.g., an online shopping site) to search for products that match the user's preferences and budget. Specific keyword search and filtering techniques are used to select products.

[0404] The server selects appropriate products from the search results and obtains detailed information (price, images, and purchase links) for each product. Based on this information, the server generates interior design suggestions suitable for the user. Specifically, the suggestions include furniture placement, color combinations, and accessory selection.

[0405] The generated proposals are sent to the device in HTML or JSON format. The device displays the received proposals on a user interface. The user can check the proposed interior designs and click on a product they like to be taken to an online shopping site that offers that product and complete the purchase process.

[0406] For example, if a user uploads a photo of their living room and inputs a natural style and a budget of 100,000 yen, the server will analyze the living room's characteristics (size, lighting conditions, etc.) and select furniture (wooden sofa, beige rug, etc.) and accessories (green potted plants, etc.) that are suitable for the natural style and within the user's budget. It will then provide an interior design proposal including specific placements for these items and links to purchase them.

[0407] This allows users to receive interior design suggestions based on their preferences and budget, and easily find and purchase the ideal interior products. This system solves users' interior design concerns and significantly reduces time and effort.

[0408] The processing flow will be explained below.

[0409] Step 1:

[0410] The user uploads photos and floor plans of the room using the device, and the device displays an interface for selecting the image data and inputting the user's preferred interior style (modern, natural, Scandinavian, etc.) and budget (e.g., 100,000 yen).

[0411] Step 2:

[0412] The device sends the uploaded image data, along with information about the user's preferences and budget, to the server, where it is packaged in an appropriate format and transmitted using a secure communication protocol.

[0413] Step 3:

[0414] The server temporarily stores the received data. The server then performs pre-processing on the received image data. This pre-processing includes adjusting the image resolution and converting the file format.

[0415] Step 4:

[0416] The server inputs the preprocessed image data into the multimodal AI for image analysis. Here, the AI ​​identifies the characteristics of the room. Specifically, it analyzes the room's size, furniture layout, wall color, window position, amount of lighting, etc. The results of this analysis are stored in the server's internal database.

[0417] Step 5:

[0418] The server performs calculations to propose an appropriate interior style based on the analysis results and the user's preferences. The calculations include furniture placement and color selection according to the characteristics of the room. The server searches an external database (such as an API of an online shopping site) to select appropriate interior items based on the user's preferences and budget.

[0419] Step 6:

[0420] The server filters the list of products retrieved from an external database and selects the best product that matches the user's criteria, taking into account factors such as price, design, and user reviews.

[0421] Step 7:

[0422] The server generates recommendations based on the selected product details (price, images, purchase links, etc.), including specific furniture placement, color schemes, and accessory selections.

[0423] Step 8:

[0424] The server sends the generated proposal to the device. The sent data is in HTML or JSON format and is organized in a format that the device can display appropriately.

[0425] Step 9:

[0426] The device displays the received proposals on a user interface, allowing the user to review the proposed interior designs and click on the desired product to view more information and a link to purchase it.

[0427] Step 10:

[0428] When the user clicks on the purchase link for the suggested product, the terminal will be redirected to the page of the corresponding online shopping site, where the user can purchase the product directly.

[0429] This process allows users to easily find interior designs and products that fit their tastes and budget, and smoothly move on to actual purchases.

[0430] Example 1

[0431] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0432] Conventional interior design suggestion systems require users to manually select products, making it easy for them to become overwhelmed by the vast amount of information, and making it difficult to efficiently find an interior design that suits their tastes and budget. Furthermore, insufficient image analysis can lead to the inability to accurately grasp the characteristics of a room and make appropriate suggestions. This can make it difficult for users to obtain satisfactory interior design suggestions, requiring a great deal of time and effort.

[0433] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0434] In this invention, the server includes means for users to upload image data of a space, means for receiving the uploaded image data and performing data preprocessing, means for analyzing the preprocessed image data and extracting features of the space, means for searching for and selecting products that match the user's preferences and budget based on the analysis results using a generative AI model, and means for generating and providing proposals for the selected products to the user. This allows users to easily obtain interior proposals that match their preferences and budget, significantly reducing time and effort.

[0435] "User" refers to an individual or group that operates the system and inputs image data of a room, preferences regarding the interior, and budget.

[0436] "Spatial image data" refers to digital images, including photographs and floor plans, of a user's own room or other space.

[0437] "Means for uploading" refers to an interface or function that allows a user to send image data or text information to a server via a terminal.

[0438] "Means for performing data preprocessing" refers to a function for performing preprocessing such as adjusting the resolution and converting the file format on received image data.

[0439] "Means for analyzing and extracting spatial characteristics" refers to the function of recognizing and identifying characteristics such as the size of a room, furniture arrangement, wall color, and amount of light based on preprocessed image data.

[0440] A "generative AI model" refers to an artificial intelligence model that can generate and analyze information from a variety of data.

[0441] "Means for searching and selecting products" refers to a function that uses an external database to find and select products that match the user's preferences and budget.

[0442] "Means for generating proposals and providing them to users" refers to the interface and functions that create interior proposals based on the selected products and present them to users.

[0443] "External Database" means a digital database containing product information and service data accessible on the Internet.

[0444] This invention is a system that uses a generative AI model to make appropriate interior design suggestions by allowing users to upload image data of their room and input their interior style and budget. The system begins by users accessing the system's web application using a device connected to the Internet.

[0445] Users upload photos and floor plans of their rooms via their devices, and input their preferred interior style and budget. For example, consider a case where a user uploads a photo of their living room and inputs a natural style and a budget of 100,000 yen. The user clicks the "Upload Image" button on the web interface to select a photo from their local device, and inputs their preferred style and budget using drop-down menus and text boxes.

[0446] The device sends the uploaded image data and the entered information to the server using the HTTP POST method. The server temporarily stores the received image data and performs preprocessing such as adjusting the resolution and converting the file format.

[0447] The server then passes the preprocessed image data to the multimodal AI, which uses deep learning models to identify characteristics such as room size, furniture placement, wall color, window position, and light intensity, and stores this data in an internal database.

[0448] Based on the analysis results, the server runs a recommendation algorithm using the user's preferences and budget information to calculate the appropriate interior style and furniture layout. During this process, the server sends an API request to an external database to search for products that match the user's preferences and budget. An example of an external database is the API of an online shopping site. In this process, keyword searches and price filtering are used to retrieve appropriate product information. The retrieved information includes the product's image URL, price, and purchase link.

[0449] Based on this information, the server creates interior design proposals, including specific furniture placement locations, color combinations to be used, and accessory selections, and generates data to provide to the user in HTML or JSON format.

[0450] The server sends the generated interior design proposals to the user's device. The device analyzes the received HTML and JSON data and displays the proposals on a user interface. The user can check the displayed interior designs and click on a product they like to be taken to the online shopping site that provides the product and complete the purchase process.

[0451] For example, by inputting the following prompt sentence into a generative AI model, interior design suggestions based on the user's requirements can be obtained.

[0452] Prompt: Analyze an image of a living room and create a natural-style interior design proposal that fits within a budget of ¥100,000. Attach an image of the living room, select items that are available within your budget, and suggest placement, color combinations, and accessory choices. Also provide a link to purchase each item.

[0453] This method allows users to easily receive interior design suggestions that match their preferences and budget, and also simplifies the purchasing process, making it possible to efficiently create the ideal interior.

[0454] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0455] Step 1:

[0456] Users use an internet-connected device to upload photos and floor plans of their rooms to the system, and also operate the interface to input their preferred interior style and budget.

[0457] Input: room image data, preferred interior style, budget

[0458] Output: The HTTP request that the user input is sent to the server.

[0459] Step 2:

[0460] The device sends the image data, preferences, and budget information entered by the user to the server using the HTTP POST method.

[0461] Input: User-entered image data, preferences, and budget

[0462] Output: Image data and user information arrive at the server as an HTTP request

[0463] Step 3:

[0464] The server temporarily stores the received data and performs pre-processing on the image data, including adjusting the resolution and converting the file format to JPEG or PNG.

[0465] Input: Image data received from the user

[0466] Output: Preprocessed image data (resolution adjusted, format converted)

[0467] Step 4:

[0468] The server then passes the preprocessed image data to a multimodal AI system that begins analyzing the image, using deep learning models to recognize characteristics such as the room's size, furniture placement, wall color, window position, and amount of light.

[0469] Input: Preprocessed image data

[0470] Output: Room characteristics data (area, furniture layout, wall color, window position, amount of light, etc.)

[0471] Step 5:

[0472] The server stores the results of the image analysis in an internal database.

[0473] Input: Room characteristic data

[0474] Output: Room characteristics data stored in a database

[0475] Step 6:

[0476] The server runs a recommendation algorithm based on the user's preferences and budget information to calculate the appropriate interior style and furniture layout. This calculation process involves sending API requests to an external database to search for products that match the user's preferences and budget.

[0477] Input: User preferences, budget information, room characteristics data

[0478] Output: Suitable interior style, furniture layout information, product candidates

[0479] Step 7:

[0480] The server selects appropriate products from the search results and retrieves detailed information for each product (price, image, purchase link).

[0481] Input: Product information as search results

[0482] Output: Selected product details (price, image, purchase link)

[0483] Step 8:

[0484] The server creates an interior design proposal, including specific furniture placement, color combinations, and accessory selection, and generates the proposal in HTML or JSON format.

[0485] Input: Selected product details, user preferences, budget information, room characteristics data

[0486] Output: Interior proposals in HTML or JSON format

[0487] Step 9:

[0488] The server sends the generated proposal to the user's device using an HTTP response.

[0489] Input: Interior proposal in HTML or JSON format

[0490] Output: HTTP response containing the proposal

[0491] Step 10:

[0492] The device parses the received suggestions and displays them in a user interface, while the front end parses the HTML and JSON data and renders images and text appropriately.

[0493] Input: HTTP response from the server (proposal)

[0494] Output: Interior proposals displayed on the user interface

[0495] Step 11:

[0496] The user can check the proposed interior designs and click on the product they like. When they click, the device opens a browser and goes to the corresponding product page on the online shopping site.

[0497] Input: Interior proposal displayed on the user interface

[0498] Output: Transition to online shopping site and start of purchase process

[0499] (Application example 1)

[0500] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0501] Many conventional interior design support systems analyze image data and propose products for a specific room in a user's home. However, providing a similar service in a physical store presents challenges, such as the unique size and characteristics of the store, as well as the appropriate placement of display items. This makes it difficult for store owners and interior designers to quickly determine effective product placement and design. The present invention aims to solve these problems and provide a system that easily proposes interior designs and product placements for physical stores.

[0502] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0503] In this invention, the server includes means for allowing a user to upload image data of a room, means for receiving the uploaded image data and performing image analysis, means for selecting products that match the user's preferences and budget based on the image analysis results, means for generating and providing suggestions for the selected products to the user, means for capturing and analyzing image data of the store, and means for suggesting display items for the store based on the image analysis results. This makes it easy to arrange and design products in a physical store in accordance with the user's preferences and budget.

[0504] A "user" is a person or organization that uses the system to receive proposals for interior design of a room or store.

[0505] A "living room" is a space in a residence where a user actually lives, and includes a room, a living room, a bedroom, etc.

[0506] "Image data" refers to digitally stored still or video images that contain visual information about a room or store.

[0507] "Upload" refers to the operation of a user sending image data from their own device to the system's server via the Internet.

[0508] "Means" refers to a method or component that enables a system or device to achieve a specific function or purpose.

[0509] "Receiving" refers to the process in which the server takes in image data sent by the user.

[0510] "Image analysis" is the process of analyzing received image data to extract characteristics of a room or store, furniture layout, wall color, amount of light, etc.

[0511] "Results" refers to the data or information obtained after performing image analysis.

[0512] "Preferences" refers to personal preference information such as interior style, color, and theme that the user has input into the system in advance.

[0513] "Budget" refers to the range of amounts set by the user for interior design and product purchases.

[0514] "Products" refers to furniture, decorations, and display items necessary for the interior design of a user's room or store.

[0515] "Selection" refers to the process of choosing products that suit the user's preferences and budget based on the results of image analysis.

[0516] "Proposal content" refers to the interior design and product placement plan shown to the user, including specific placement proposals and detailed information about the products to be used.

[0517] "Providing" refers to the process by which the system presents the suggestions to the user through the user interface.

[0518] The present invention is a system that allows users to upload image data of a room or a brick-and-mortar store, and then proposes interior designs and product layouts based on that data. This system involves a series of processes: sending image data from the user's device to a server, analyzing it, generating proposals, and providing them to the user.

[0519] First, the user captures image data of a room in their home or a brick-and-mortar store using a device and uploads it to the system. The device is equipped with a camera, and can be, for example, a smartphone or a head-mounted display (HMD). In addition to the image data, the user also enters their preferred interior style and budget information. This information is then sent to the server by the program and temporarily stored.

[0520] The server preprocesses the received image data, which includes adjusting the image resolution and converting the file format. To perform this processing, the server uses the OpenCV library.

[0521] The server then loads the preprocessed image data into a multimodal AI model for image analysis. The analysis results are converted into a dataset containing room or store characteristics (size, furniture arrangement, wall color, amount of light, etc.). The AI ​​model used here is a pre-trained generative AI model.

[0522] Based on the analysis results, the server accesses internal and external databases to search for products and display items that match the user's preferences and budget. Product selection uses specific keyword search and filtering techniques, which are implemented by the request library. For example, if a user enters a prompt such as "Please suggest items in a natural style that cost less than 100,000 yen," the server will perform a search based on this information.

[0523] The server selects appropriate products and display items from the search results and retrieves their detailed information (price, images, purchase links, etc.). Based on this, the server generates a suitable interior design proposal for the user, including specific furniture and display item placement, color combinations, accessory selection, etc.

[0524] The generated proposals are sent to the device in HTML or JSON format. The proposals are visually displayed on the user's device based on the received data, allowing the user to check the proposed interior design. By clicking on a product they like, they can be taken to the online shopping site that offers that product and complete the purchase process.

[0525] For example, if a user uploads a photo of their living room and enters "natural style with a budget of 100,000 yen," the server analyzes the characteristics of the living room and selects furniture (e.g., a wooden sofa, a beige rug) and accessories (e.g., green potted plants) suitable for the natural style within the user's budget. It then provides an interior design proposal, including specific placements for these items and links to purchase them.

[0526] This system allows users to easily receive store interior designs using their smartphones or HMDs, and quickly find and purchase the ideal interior products. This process is an effective way to resolve users' concerns about interior design and significantly reduce time and effort.

[0527] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0528] Step 1:

[0529] The user captures image data of a room or store using a terminal and uploads it to the system.

[0530] Inputs are image data captured by a camera, preferred interior style, and budget information.

[0531] In operation, the user takes an image using a smartphone or head-mounted display (HMD) and sends it to the server using the application's upload function.

[0532] The output is image data and user preference data sent to the server.

[0533] Step 2:

[0534] The server preprocesses the received image data.

[0535] The input is image data and preference information submitted by the user.

[0536] In operation, the server uses the OpenCV library to adjust the image resolution and convert it to the appropriate file format. This preprocessing makes the image suitable for analysis.

[0537] The output is pre-processed image data.

[0538] Step 3:

[0539] The server loads the preprocessed image data into a multimodal AI model and performs image analysis.

[0540] The input is the preprocessed image data.

[0541] In operation, the server inputs data into a multimodal AI model and performs image analysis, which extracts characteristics of the room or store (size, furniture arrangement, wall color, amount of light, etc.).

[0542] The output is characteristic data based on the analysis results.

[0543] Step 4:

[0544] Based on the image analysis results, the server selects products that suit the user's preferences and budget.

[0545] The inputs are the image analysis results data, user preferences, and budget information.

[0546] In operation, the server accesses internal and external databases to search for products and display items that meet the user's preferences and budget, using specific keyword search and filtering techniques.

[0547] The output is a list of selected products with their details (price, image, and purchase link).

[0548] Step 5:

[0549] The server generates and provides selected product and display item suggestions to the user.

[0550] The input is a product list and detailed information.

[0551] The server then sends the generated proposals in HTML or JSON format to the device, and creates an interior design including specific layouts and color combinations.

[0552] The output is the proposal sent to the user terminal.

[0553] Step 6:

[0554] The user checks the proposed content on the terminal, clicks on the product they like, and is taken to an online shopping site where they can complete the purchase.

[0555] The input is the proposal sent by the server.

[0556] The user visually checks the recommendations and browses detailed information about the recommended products. When the user clicks on a product they like, they are taken to a shopping site where they can complete the purchase process.

[0557] The output is to complete the purchase of the product.

[0558] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0559] The present invention provides a system that allows users to upload image data of their rooms and proposes interior design suggestions based on their preferences, budget, and emotional state. The image data uploaded by the user includes photos and floor plans of the rooms, which the user sends to the system through an interface. The user also inputs or has the system detect their preferred interior style, budget, and real-time emotional state.

[0560] When the device receives user input data, it sends it to the server. The server preprocesses the received image data (adjusting resolution, converting formats, etc.), and then uses multimodal AI to analyze the room's characteristics. This analysis includes the room's size, furniture layout, wall color, amount of light, etc. The analysis results are stored in an internal database.

[0561] Next, the device uses an emotion engine to recognize the user's emotional state. The emotion engine analyzes the user's input data and biometric data (facial expressions, voice tone, heart rate, etc.) to detect whether the user is feeling joy, surprise, sadness, anger, etc. The detected emotional state is also sent to the server.

[0562] The server generates appropriate interior style and furniture layout suggestions by taking into consideration the room's characteristics, the user's emotional state, as well as their preferences and budget. In this process, optimization is performed according to the user's emotional state; for example, if the user's emotional state is positive, suggestions for a bright and lively style are prioritized, whereas if the user's emotional state is negative, suggestions for a more subdued style are prioritized.

[0563] The server also searches external databases (such as the API of an online shopping site) to select interior products that meet the user's requirements, taking into consideration factors such as price, design, and user reviews. Detailed information about the selected products (price, images, purchase links, etc.) is also obtained.

[0564] Once the proposals are generated, the server sends them to the device. The device then displays the received proposals on its user interface. The user can review the displayed interior designs and view detailed information and purchase links for products they like. By clicking on a product, they can be taken directly to an online shopping site and complete the purchase process.

[0565] For example, suppose a user uploads a photo of their living room, specifies a preference for a natural style, and sets a budget of 100,000 yen. Furthermore, suppose the emotion engine recognizes the user's smile and detects a happy emotional state. The server analyzes the characteristics of the living room and suggests natural-style furniture and accessories (e.g., a wooden sofa or a beige rug). These suggestions are chosen to create a bright and cheerful atmosphere. Detailed information and purchase links for the selected products are also included, allowing the user to easily purchase them.

[0566] In this way, by using the system of the present invention, users can receive interior design suggestions that take into consideration their own emotional state, and easily find and purchase their ideal interior design items.

[0567] The processing flow will be explained below.

[0568] Step 1:

[0569] The user uploads image data of the room (e.g., a photo of the living room) to the system using a terminal. In addition, the user inputs their preferred interior style (e.g., natural) and budget (e.g., 100,000 yen) through the interface.

[0570] Step 2:

[0571] The device sends the uploaded image data, user preferences, and budget information to a server, where the data is packaged in a suitable format and transmitted using a secure communication protocol.

[0572] Step 3:

[0573] The server temporarily stores the received data. The server then preprocesses the received image data, adjusting the resolution and converting the format.

[0574] Step 4:

[0575] The server inputs the preprocessed image data into the multimodal AI for image analysis, which detects the room's characteristics (size, furniture arrangement, wall color, window position, amount of light, etc.). The analyzed characteristic data is stored in an internal database.

[0576] Step 5:

[0577] The device uses an emotion engine to recognize the user's emotional state. The emotion engine analyzes the user's biometric data (facial expressions, tone of voice, heart rate, etc.) to detect emotional states such as joy, surprise, sadness, and anger. The detected emotional data is sent to the server.

[0578] Step 6:

[0579] The server combines and analyzes the room's characteristic data, the user's preferences, budget, and emotional data to generate appropriate interior design suggestions. If the emotional state is positive, a bright and colorful interior style is prioritized, while if the emotional state is negative, a calm style is prioritized.

[0580] Step 7:

[0581] The server searches an external database (e.g., an API for an online shopping site) to select interior products that match the user's tastes and budget. The external database contains information such as prices, designs, and user reviews. Detailed information about the selected products (price, images, purchase links, etc.) is also retrieved.

[0582] Step 8:

[0583] The server generates interior design suggestions, including furniture placement, color schemes, and accessory selections based on the user's preferences, budget, and emotional state. The suggestions are formatted in HTML or JSON.

[0584] Step 9:

[0585] The server sends the generated proposal to the terminal, which displays the received data on the user interface, and the user checks the interior design displayed on the interface.

[0586] Step 10:

[0587] When the user clicks on the suggested product details or purchase link, the device will be redirected to the corresponding page on the online shopping site, where the user can purchase the product directly.

[0588] For example, if a user uploads a photo of their living room, selects a natural style and a budget of 100,000 yen, and the emotion engine detects the user's smile and recognizes their emotional state of joy, the server will suggest natural-style interior items that create a bright and cheerful atmosphere (e.g., a wooden sofa, a beige rug, etc.). These suggestions also include specific placement locations, color schemes, and accessories (e.g., green potted plants), allowing users to easily find and purchase the ideal interior items.

[0589] Example 2

[0590] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0591] Conventional interior design suggestion systems generally suggest interior styles based on the user's preferences and budget, but they are unable to take into account the user's real-time emotional state. As a result, they are unable to provide optimal interior suggestions that reflect the user's momentary psychological state. This makes it difficult to meet the diverse needs of users, and more personalized suggestions are needed.

[0592] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0593] In this invention, the server includes means for a user to upload image data of a room, means for adjusting the resolution of the image and converting the format, means for analyzing the room size, furniture layout, wall color, amount of light, etc., means for analyzing the user's biometric data to recognize the user's emotional state, means for generating suggestions for interior styles and furniture layouts based on the analysis results and the user's preferences, budget, and emotional state, means for searching an external database to obtain detailed information and purchase links for products that meet the user's requirements, and means for displaying the generated suggestions on a user interface. This enables personalized interior suggestions that take the user's real-time emotional state into consideration.

[0594] "User" refers to an individual who uses the system to receive room interior suggestions.

[0595] "Room image data" refers to image files such as room photos and floor plans that users upload to the system.

[0596] "Resolution adjustment" refers to the process of changing the resolution of an input image to an appropriate size.

[0597] "Format conversion" refers to the process of converting the file format of input image data into a format that can be used by the system.

[0598] "Room size" refers to information indicating the physical size of the room obtained from the analyzed image.

[0599] "Furniture arrangement" refers to information indicating the position and arrangement of furniture within a room obtained from the analyzed image.

[0600] "Wall color" refers to information indicating the color of the walls of a room obtained from the analyzed image.

[0601] "Amount of light" refers to information indicating the brightness of light in a room and the amount of light incident thereon, obtained from the analyzed image.

[0602] "Biometric data" refers to physiological data such as a user's facial expression, voice tone, heart rate, etc.

[0603] "Emotional state" refers to the emotion the user is feeling (such as joy, surprise, sadness, anger, etc.).

[0604] "Interior style" refers to interior design styles such as natural, modern, and classic.

[0605] "Furniture arrangement suggestions" refers to specific suggestions showing the optimal furniture arrangement and interior style for the user's room.

[0606] "External Database" refers to a database for obtaining product information from online shopping sites and other external resources.

[0607] "Detailed Information" refers to information about the selected product, including price, images, and purchase links.

[0608] "User interface" refers to the screen and operation method that allows a user to interact with a system.

[0609] The present invention is a system that allows users to upload image data of a room and receives interior design suggestions based on the user's preferences, budget, and real-time emotional state. Specific hardware and software used to implement the invention, as well as methods for processing and calculating data, and specific examples are described below.

[0610] Hardware and Software

[0611] A terminal is a computer or mobile device that provides a user interface and accepts data input from a user. The terminal has image processing software and an emotion engine installed.

[0612] The server is a computer system that processes and analyzes the received data. The following software is used for this analysis:

[0613] Use an image processing library (e.g., OpenCV, Pillow) to adjust image resolution and convert image formats.

[0614] A multimodal AI model (e.g., CNN) analyzes room characteristics (size, furniture arrangement, wall color, amount of light, etc.).

[0615] Recognize the user's emotional state with an emotion recognition library (e.g., OpenFace or other facial expression recognition libraries).

[0616] Data processing and calculation

[0617] When a user launches the system, an interface appears. The user uploads a photo of the room (e.g., a photo of the living room) and a floor plan, inputs the interior style (e.g., natural style) and budget (e.g., 100,000 yen), and provides facial expressions and voice data to detect emotional states.

[0618] The device sends the received data to the server, which then preprocesses the received image data, adjusting the resolution and converting the format, and analyzes the room size, furniture layout, wall color, amount of light, etc.

[0619] Based on this data, the server uses a generative AI model to generate optimal interior style and furniture arrangement suggestions tailored to the user's preferences, budget, and emotional state, with example prompts such as:

[0620] Prompt: "Picture of a living room, natural style, budget of ¥100,000, user smiling (happy)"

[0621] The server searches an external database to select interior products that meet the user's criteria. It selects the most suitable product by comprehensively considering factors such as price, design, and user reviews. It also obtains detailed information about the selected product (price, images, purchase links, etc.).

[0622] Finally, the server sends the generated recommendations to the terminal, which displays them on the user interface. The user can then check the recommendations, view detailed information and purchase links, and purchase the products they are interested in directly from the online shopping site.

[0623] For example, suppose a user uploads a photo of their living room, specifies a preference for a natural style, and sets a budget of 100,000 yen. Furthermore, suppose the emotion engine recognizes the user's smile and detects a happy emotional state. The server analyzes the characteristics of the living room and suggests natural-style furniture and accessories (e.g., a wooden sofa or a beige rug). These suggestions are chosen to create a bright and cheerful atmosphere. Detailed information and purchase links for the selected products are also included, allowing the user to easily purchase them.

[0624] In this way, by using the system of the present invention, users can receive interior design suggestions that take into consideration their own emotional state, and easily find and purchase their ideal interior design items.

[0625] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0626] Step 1: System startup and data entry

[0627] When the user boots the system, the interface is displayed.

[0628] Users upload photos of their rooms (e.g., photos of their living rooms) and floor plans.

[0629] The user inputs the interior style (e.g., natural style) and budget (e.g., 100,000 yen) through the interface.

[0630] The user provides facial expression and voice data to detect the emotional state, which then provides the emotional data.

[0631] Step 2: Receiving data and sending it to the server

[0632] The terminal receives input data from the user (room image data, preferences, budget, and emotional data).

[0633] The terminal sends the received data to the server. Input is the user's input data, which the terminal sends to the server.

[0634] Step 3: Preprocessing the image data

[0635] The server preprocesses the image data it receives, specifically adjusting the image resolution and converting the format.

[0636] The server uses the OpenCV and Pillow libraries for this process.

[0637] For example, downscaling a high-resolution image and converting it to an appropriate image format (JPEG or PNG). The input is the original image data and the output is the preprocessed image data.

[0638] Step 4: Analyzing the image data

[0639] The server uses the pre-processed image data to analyze the characteristics of the room.

[0640] The server uses a multimodal AI model (e.g., CNN) to determine the size of the room, furniture arrangement, wall color, amount of light, etc.

[0641] The analysis results are stored in an internal database. The input is preprocessed image data, and the output is the analysis results.

[0642] Step 5: Recognizing your emotional state

[0643] The device uses an emotion engine to recognize the user's emotional state, using data such as facial expressions, voice tone, and heart rate.

[0644] The device uses OpenFace and other facial recognition libraries for emotion recognition.

[0645] The emotional state is recognized and sent to the server. The input is the user's biometric data and the output is the emotional state data.

[0646] Step 6: Generate proposals

[0647] The server generates appropriate interior style and furniture arrangement suggestions taking into account the room's characteristics, the user's emotional state, preferences, and budget.

[0648] The server uses a generative AI model to create suggestions based on the input (prompt).

[0649] Prompt: "Picture of a living room, natural style, budget of ¥100,000, user smiling (happy)"

[0650] A proposal is generated. The inputs are the room characteristic data, the emotional state data, the user's preferences and budget, and the output is the generated proposal.

[0651] Step 7: Product Selection

[0652] The server searches an external database and selects interior products that meet the user's requirements.

[0653] The server selects the most suitable product by comprehensively considering price, design, user reviews, etc. Detailed information about the selected product (price, image, purchase link, etc.) is obtained.

[0654] The input is the user's conditions and proposal details, and the output is the selected product information.

[0655] Step 8: View the results

[0656] The server sends the generated proposal to the terminal.

[0657] The terminal displays the received proposal on a user interface.

[0658] Users can review the suggestions displayed, view detailed information and purchase links, and select the appropriate product.

[0659] The input is the proposal content and product information, and the output is the proposal and product information displayed on the user interface.

[0660] (Application example 2)

[0661] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0662] Conventional interior design suggestion systems can select interior products based on the user's preferences and budget, but they do not consider the user's emotional state when making suggestions. This makes it difficult to provide the optimal interior design that matches the user's psychological state. Furthermore, there is a lack of a way to easily access detailed information and purchase links on the spot.

[0663] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0664] In this invention, the server includes means for a user to upload image data of a room, means for receiving the uploaded image data and performing image analysis, means for selecting products that match the user's preferences, budget, and emotional state based on the image analysis results, means for generating proposals for the selected products and providing them to the user, means for analyzing the user's facial expression images to detect the user's emotional state, means including an emotion engine with the function of analyzing the emotional state in real time, and means for recommending interior styles according to the emotional state. This makes it possible to propose an optimal interior design that matches the user's real-time emotional state, and further allows the user to access detailed information and a purchase link on the spot.

[0665] "User" refers to an individual or organization that uses the system to upload image data of a room and receive interior design suggestions.

[0666] "Image data" refers to digital image files such as room photos and floor plans uploaded by users.

[0667] "Upload" refers to the act of a user sending image data from their device to a server.

[0668] "Image analysis" refers to the process by which the server preprocesses the image data it receives and analyzes the characteristics of the room.

[0669] "Preferences" refer to personal preferences regarding the interior style and design desired by the user.

[0670] "Budget" refers to the upper limit of costs set by the user when selecting interior products.

[0671] "Emotional state" refers to the psychological state of the user, which is analyzed using the emotion engine.

[0672] An "emotion engine" refers to an algorithm or software that analyzes a user's facial expression images and biometric data to detect their emotional state.

[0673] "Server" refers to a computer system that processes image data and other information received from users and generates and provides interior design proposals.

[0674] "Suggestions" refers to the interior design and product recommendations generated by the system.

[0675] "External databases" refer to online shopping sites and other information sources on the Internet that provide reference information for product selection.

[0676] "Product selection" refers to the process by which the system selects the most suitable interior products based on the user's preferences, budget, and emotional state.

[0677] "Real-time" refers to the process by which a system processes user input or detected information immediately and generates and delivers results.

[0678] This invention provides a system that allows a user to upload image data of a room and proposes interior design ideas based on the user's preferences, budget, and emotional state. The specific system design and operation are described below.

[0679] server

[0680] The server is a computer system that processes image data and other information received from users and generates and provides interior design proposals. The server includes the following means:

[0681] 1. A function to receive image data and perform preprocessing (resolution adjustment, format conversion, etc.).

[0682] 2. Room characteristic analysis function using multimodal AI (analyzing room size, furniture arrangement, wall color, amount of light, etc.).

[0683] 3. A function that uses an emotion engine to analyze the user's emotional state (analyzing facial expressions, voice, heart rate, etc. to detect joy, surprise, sadness, anger, etc.).

[0684] 4. A feature that suggests the best interior style and furniture arrangement based on the user's preferences, budget, and emotional state.

[0685] 5. The ability to search external databases (e.g., online shopping sites) and select appropriate interior products.

[0686] Terminal

[0687] A terminal is a device that allows users to access the system and input information. This terminal can be a smartphone, tablet, or PC. The specific operation steps are as follows:

[0688] 1. The user takes a photo of the room using the device and uploads the image data.

[0689] 2. The user inputs their preferred interior style and budget through the interface.

[0690] 3. The device captures the user's facial expressions and sends the data to the emotion engine in real time.

[0691] Processing Flow

[0692] Hardware and software configuration

[0693] Hardware: Smartphone (camera function, network connection), server (high performance computer).

[0694] Software: Python, PIL (image preprocessing), Emotion Engine (sentiment analysis), Interior Style Recommender (interior suggestions).

[0695] The server first preprocesses the image data of the room received from the user (adjusting the resolution, converting the format, etc.). It then uses multimodal AI to analyze the room's size, furniture arrangement, wall color, amount of light, etc. The analysis results are stored in a database on the server. At the same time, the device uses an emotion engine to analyze the user's emotional state, and this information is also sent to the server.

[0696] The server takes into account the room's characteristics, the user's emotional state, preferences, and budget to generate suggestions for appropriate interior styles and furniture arrangements. These suggestions also include product information (price, images, purchase links, etc.) retrieved from an external database. Once the suggestions are generated, the server sends them to the device and provides them to the user.

[0697] Examples and prompts

[0698] For example, if a user uploads a photo of their living room with a natural style and a budget of 100,000 yen, the system works as follows: The emotion engine recognizes the user's smile and detects a happy emotional state. The server analyzes the characteristics of the living room and suggests natural-style furniture and accessories (e.g., a wooden sofa, a beige rug). The suggestions are selected to create a bright and cheerful atmosphere.

[0699] Prompt Sentence Examples

[0700] "Upload a picture of your room: room.jpg"

[0701] "Please enter your preferred interior style: Natural"

[0702] "Please enter your budget: 100,000 yen"

[0703] "Upload a photo of your face: user.jpg"

[0704] This allows users to easily find and purchase their ideal interior items through specific interior suggestions.

[0705] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0706] Step 1:

[0707] The user uses the terminal to take a picture of the room and upload it.

[0708] Input: A photo of the room.

[0709] Output: The uploaded image data.

[0710] Specific operation: The user takes a photo of the room using the camera function of their smartphone, and then presses the upload button in the application to send the image data to the server.

[0711] Step 2:

[0712] The server pre-processes the image data received from the user.

[0713] Input: Uploaded image data.

[0714] Output: Preprocessed image data.

[0715] What it does: It adjusts the image resolution on the server and converts it to the required format (e.g. resize to 256x256 pixels). It uses Python and the PIL library for preprocessing.

[0716] Step 3:

[0717] The server analyzes the pre-processed image data to understand the characteristics of the room.

[0718] Input: Preprocessed image data.

[0719] Output: Analysis results such as room size, furniture placement, wall color, and amount of light.

[0720] Specific operation: A multimodal AI system on the server analyzes image data and extracts room characteristics. The extracted data is stored in an internal database.

[0721] Step 4:

[0722] The user inputs their preferred interior style and budget through the interface.

[0723] Input: The user's interior style preferences and budget.

[0724] Output: Interior style preferences, budget data.

[0725] What it does: Enter style and budget through a user interface using drop-down menus and text input forms.

[0726] Step 5:

[0727] The terminal uses an emotion engine to detect the user's emotional state.

[0728] Input: An image of the user's facial expression.

[0729] Output: Detected emotional state data.

[0730] Specific operation: The smartphone camera captures the user's facial expression and sends it to the server in real time. The emotion engine analyzes the facial expression data and detects the user's emotional state (e.g., joy, sadness, surprise, etc.).

[0731] Step 6:

[0732] The server suggests interior styles and furniture arrangements based on the room's characteristics, the user's preferences, budget, and emotional state.

[0733] Input: Room characteristic data, interior style preference, budget data, emotional state data.

[0734] Output: Interior design proposals (style, furniture arrangement, specific products).

[0735] What it does: The algorithm in the server generates the optimal interior design based on all input data, and then retrieves and finalizes the proposal from internal and external databases.

[0736] Step 7:

[0737] The server sends the selected product details and a link to purchase the product to the device.

[0738] Input: Interior design proposal.

[0739] Output: The suggestions that will be displayed on the user's device.

[0740] Specific operation: The server sends the generated interior design proposal to the device and displays detailed information (price, image, purchase link) on the interface.

[0741] Step 8:

[0742] The user reviews the offer and clicks the purchase link to purchase the product.

[0743] Input: Proposed interior product.

[0744] Output: Transition to online shopping site and purchase process.

[0745] Specific actions: The user checks the suggestions displayed on the device and clicks on the purchase link for the product they are interested in. They then complete the purchase at the linked online shopping site.

[0746] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0747] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0748] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0749] [Third embodiment]

[0750] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0751] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0752] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0753] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0754] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0755] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0756] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0757] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0758] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0759] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0760] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0761] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0762] The present invention begins with the user uploading image data of a room. Using a terminal connected to the Internet, the user sends photos and floor plans of the rooms in their home to the system. The user also operates an interface to input their preferred interior style and budget.

[0763] The device sends the image data, preferences, and budget information entered by the user to the server. The server temporarily stores the received data and performs preprocessing on the image data, including adjusting the resolution and converting the file format.

[0764] The server then loads the pre-processed image data into a multimodal AI system for image analysis, which detects characteristics such as the room size, furniture arrangement, wall color, window position, and amount of light, which are then stored in an internal database.

[0765] Based on the analysis results, the server applies the user's preferences and budget information and performs calculations to suggest an appropriate interior style and furniture arrangement. At this stage, the server accesses an external database (e.g., an online shopping site) to search for products that match the user's preferences and budget. Specific keyword search and filtering techniques are used to select products.

[0766] The server selects appropriate products from the search results and obtains detailed information (price, images, and purchase links) for each product. Based on this information, the server generates interior design suggestions suitable for the user. Specifically, the suggestions include furniture placement, color combinations, and accessory selection.

[0767] The generated proposals are sent to the device in HTML or JSON format. The device displays the received proposals on a user interface. The user can check the proposed interior designs and click on a product they like to be taken to an online shopping site that offers that product and complete the purchase process.

[0768] For example, if a user uploads a photo of their living room and inputs a natural style and a budget of 100,000 yen, the server will analyze the living room's characteristics (size, lighting conditions, etc.) and select furniture (wooden sofa, beige rug, etc.) and accessories (green potted plants, etc.) that are suitable for the natural style and within the user's budget. It will then provide an interior design proposal including specific placements for these items and links to purchase them.

[0769] This allows users to receive interior design suggestions based on their preferences and budget, and easily find and purchase the ideal interior products. This system solves users' interior design concerns and significantly reduces time and effort.

[0770] The processing flow will be explained below.

[0771] Step 1:

[0772] The user uploads photos and floor plans of the room using the device, and the device displays an interface for selecting the image data and inputting the user's preferred interior style (modern, natural, Scandinavian, etc.) and budget (e.g., 100,000 yen).

[0773] Step 2:

[0774] The device sends the uploaded image data, along with information about the user's preferences and budget, to the server, where it is packaged in an appropriate format and transmitted using a secure communication protocol.

[0775] Step 3:

[0776] The server temporarily stores the received data. The server then performs pre-processing on the received image data. This pre-processing includes adjusting the image resolution and converting the file format.

[0777] Step 4:

[0778] The server inputs the preprocessed image data into the multimodal AI for image analysis. Here, the AI ​​identifies the characteristics of the room. Specifically, it analyzes the room's size, furniture layout, wall color, window position, amount of lighting, etc. The results of this analysis are stored in the server's internal database.

[0779] Step 5:

[0780] The server performs calculations to propose an appropriate interior style based on the analysis results and the user's preferences. The calculations include furniture placement and color selection according to the characteristics of the room. The server searches an external database (such as an API of an online shopping site) to select appropriate interior items based on the user's preferences and budget.

[0781] Step 6:

[0782] The server filters the list of products retrieved from an external database and selects the best product that matches the user's criteria, taking into account factors such as price, design, and user reviews.

[0783] Step 7:

[0784] The server generates recommendations based on the selected product details (price, images, purchase links, etc.), including specific furniture placement, color schemes, and accessory selections.

[0785] Step 8:

[0786] The server sends the generated proposal to the device. The sent data is in HTML or JSON format and is organized in a format that the device can display appropriately.

[0787] Step 9:

[0788] The device displays the received proposals on a user interface, allowing the user to review the proposed interior designs and click on the desired product to view more information and a link to purchase it.

[0789] Step 10:

[0790] When the user clicks on the purchase link for the suggested product, the terminal will be redirected to the page of the corresponding online shopping site, where the user can purchase the product directly.

[0791] This process allows users to easily find interior designs and products that fit their tastes and budget, and smoothly move on to actual purchases.

[0792] Example 1

[0793] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0794] Conventional interior design suggestion systems require users to manually select products, making it easy for them to become overwhelmed by the vast amount of information, and making it difficult to efficiently find an interior design that suits their tastes and budget. Furthermore, insufficient image analysis can lead to the inability to accurately grasp the characteristics of a room and make appropriate suggestions. This can make it difficult for users to obtain satisfactory interior design suggestions, requiring a great deal of time and effort.

[0795] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0796] In this invention, the server includes means for users to upload image data of a space, means for receiving the uploaded image data and performing data preprocessing, means for analyzing the preprocessed image data and extracting features of the space, means for searching for and selecting products that match the user's preferences and budget based on the analysis results using a generative AI model, and means for generating and providing proposals for the selected products to the user. This allows users to easily obtain interior proposals that match their preferences and budget, significantly reducing time and effort.

[0797] "User" refers to an individual or group that operates the system and inputs image data of a room, preferences regarding the interior, and budget.

[0798] "Spatial image data" refers to digital images, including photographs and floor plans, of a user's own room or other space.

[0799] "Means for uploading" refers to an interface or function that allows a user to send image data or text information to a server via a terminal.

[0800] "Means for performing data preprocessing" refers to a function for performing preprocessing such as adjusting the resolution and converting the file format on received image data.

[0801] "Means for analyzing and extracting spatial characteristics" refers to the function of recognizing and identifying characteristics such as the size of a room, furniture arrangement, wall color, and amount of light based on preprocessed image data.

[0802] A "generative AI model" refers to an artificial intelligence model that can generate and analyze information from a variety of data.

[0803] "Means for searching and selecting products" refers to a function that uses an external database to find and select products that match the user's preferences and budget.

[0804] "Means for generating proposals and providing them to users" refers to the interface and functions that create interior proposals based on the selected products and present them to users.

[0805] "External Database" means a digital database containing product information and service data accessible on the Internet.

[0806] This invention is a system that uses a generative AI model to make appropriate interior design suggestions by allowing users to upload image data of their room and input their interior style and budget. The system begins by users accessing the system's web application using a device connected to the Internet.

[0807] Users upload photos and floor plans of their rooms via their devices, and input their preferred interior style and budget. For example, consider a case where a user uploads a photo of their living room and inputs a natural style and a budget of 100,000 yen. The user clicks the "Upload Image" button on the web interface to select a photo from their local device, and inputs their preferred style and budget using drop-down menus and text boxes.

[0808] The device sends the uploaded image data and the entered information to the server using the HTTP POST method. The server temporarily stores the received image data and performs preprocessing such as adjusting the resolution and converting the file format.

[0809] The server then passes the preprocessed image data to the multimodal AI, which uses deep learning models to identify characteristics such as room size, furniture placement, wall color, window position, and light intensity, and stores this data in an internal database.

[0810] Based on the analysis results, the server runs a recommendation algorithm using the user's preferences and budget information to calculate the appropriate interior style and furniture layout. During this process, the server sends an API request to an external database to search for products that match the user's preferences and budget. An example of an external database is the API of an online shopping site. In this process, keyword searches and price filtering are used to retrieve appropriate product information. The retrieved information includes the product's image URL, price, and purchase link.

[0811] Based on this information, the server creates interior design proposals, including specific furniture placement locations, color combinations to be used, and accessory selections, and generates data to provide to the user in HTML or JSON format.

[0812] The server sends the generated interior design proposals to the user's device. The device analyzes the received HTML and JSON data and displays the proposals on a user interface. The user can check the displayed interior designs and click on a product they like to be taken to the online shopping site that provides the product and complete the purchase process.

[0813] For example, by inputting the following prompt sentence into a generative AI model, interior design suggestions based on the user's requirements can be obtained.

[0814] Prompt: Analyze an image of a living room and create a natural-style interior design proposal that fits within a budget of ¥100,000. Attach an image of the living room, select items that are available within your budget, and suggest placement, color combinations, and accessory choices. Also provide a link to purchase each item.

[0815] This method allows users to easily receive interior design suggestions that match their preferences and budget, and also simplifies the purchasing process, making it possible to efficiently create the ideal interior.

[0816] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0817] Step 1:

[0818] Users use an internet-connected device to upload photos and floor plans of their rooms to the system, and also operate the interface to input their preferred interior style and budget.

[0819] Input: room image data, preferred interior style, budget

[0820] Output: The HTTP request that the user input is sent to the server.

[0821] Step 2:

[0822] The device sends the image data, preferences, and budget information entered by the user to the server using the HTTP POST method.

[0823] Input: User-entered image data, preferences, and budget

[0824] Output: Image data and user information arrive at the server as an HTTP request

[0825] Step 3:

[0826] The server temporarily stores the received data and performs pre-processing on the image data, including adjusting the resolution and converting the file format to JPEG or PNG.

[0827] Input: Image data received from the user

[0828] Output: Preprocessed image data (resolution adjusted, format converted)

[0829] Step 4:

[0830] The server then passes the preprocessed image data to a multimodal AI system that begins analyzing the image, using deep learning models to recognize characteristics such as the room's size, furniture placement, wall color, window position, and amount of light.

[0831] Input: Preprocessed image data

[0832] Output: Room characteristics data (area, furniture layout, wall color, window position, amount of light, etc.)

[0833] Step 5:

[0834] The server stores the results of the image analysis in an internal database.

[0835] Input: Room characteristic data

[0836] Output: Room characteristics data stored in a database

[0837] Step 6:

[0838] The server runs a recommendation algorithm based on the user's preferences and budget information to calculate the appropriate interior style and furniture layout. This calculation process involves sending API requests to an external database to search for products that match the user's preferences and budget.

[0839] Input: User preferences, budget information, room characteristics data

[0840] Output: Suitable interior style, furniture layout information, product candidates

[0841] Step 7:

[0842] The server selects appropriate products from the search results and retrieves detailed information for each product (price, image, purchase link).

[0843] Input: Product information as search results

[0844] Output: Selected product details (price, image, purchase link)

[0845] Step 8:

[0846] The server creates an interior design proposal, including specific furniture placement, color combinations, and accessory selection, and generates the proposal in HTML or JSON format.

[0847] Input: Selected product details, user preferences, budget information, room characteristics data

[0848] Output: Interior proposals in HTML or JSON format

[0849] Step 9:

[0850] The server sends the generated proposal to the user's device using an HTTP response.

[0851] Input: Interior proposal in HTML or JSON format

[0852] Output: HTTP response containing the proposal

[0853] Step 10:

[0854] The device parses the received suggestions and displays them in a user interface, while the front end parses the HTML and JSON data and renders images and text appropriately.

[0855] Input: HTTP response from the server (proposal)

[0856] Output: Interior proposals displayed on the user interface

[0857] Step 11:

[0858] The user can check the proposed interior designs and click on the product they like. When they click, the device opens a browser and goes to the corresponding product page on the online shopping site.

[0859] Input: Interior proposal displayed on the user interface

[0860] Output: Transition to online shopping site and start of purchase process

[0861] (Application example 1)

[0862] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0863] Many conventional interior design support systems analyze image data and propose products for a specific room in a user's home. However, providing a similar service in a physical store presents challenges, such as the unique size and characteristics of the store, as well as the appropriate placement of display items. This makes it difficult for store owners and interior designers to quickly determine effective product placement and design. The present invention aims to solve these problems and provide a system that easily proposes interior designs and product placements for physical stores.

[0864] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0865] In this invention, the server includes means for allowing a user to upload image data of a room, means for receiving the uploaded image data and performing image analysis, means for selecting products that match the user's preferences and budget based on the image analysis results, means for generating and providing suggestions for the selected products to the user, means for capturing and analyzing image data of the store, and means for suggesting display items for the store based on the image analysis results. This makes it easy to arrange and design products in a physical store in accordance with the user's preferences and budget.

[0866] A "user" is a person or organization that uses the system to receive proposals for interior design of a room or store.

[0867] A "living room" is a space in a residence where a user actually lives, and includes a room, a living room, a bedroom, etc.

[0868] "Image data" refers to digitally stored still or video images that contain visual information about a room or store.

[0869] "Upload" refers to the operation of a user sending image data from their own device to the system's server via the Internet.

[0870] "Means" refers to a method or component that enables a system or device to achieve a specific function or purpose.

[0871] "Receiving" refers to the process in which the server takes in image data sent by the user.

[0872] "Image analysis" is the process of analyzing received image data to extract characteristics of a room or store, furniture layout, wall color, amount of light, etc.

[0873] "Results" refers to the data or information obtained after performing image analysis.

[0874] "Preferences" refers to personal preference information such as interior style, color, and theme that the user has input into the system in advance.

[0875] "Budget" refers to the range of amounts set by the user for interior design and product purchases.

[0876] "Products" refers to furniture, decorations, and display items necessary for the interior design of a user's room or store.

[0877] "Selection" refers to the process of choosing products that suit the user's preferences and budget based on the results of image analysis.

[0878] "Proposal content" refers to the interior design and product placement plan shown to the user, including specific placement proposals and detailed information about the products to be used.

[0879] "Providing" refers to the process by which the system presents the suggestions to the user through the user interface.

[0880] The present invention is a system that allows users to upload image data of a room or a brick-and-mortar store, and then proposes interior designs and product layouts based on that data. This system involves a series of processes: sending image data from the user's device to a server, analyzing it, generating proposals, and providing them to the user.

[0881] First, the user captures image data of a room in their home or a brick-and-mortar store using a device and uploads it to the system. The device is equipped with a camera, and can be, for example, a smartphone or a head-mounted display (HMD). In addition to the image data, the user also enters their preferred interior style and budget information. This information is then sent to the server by the program and temporarily stored.

[0882] The server preprocesses the received image data, which includes adjusting the image resolution and converting the file format. To perform this processing, the server uses the OpenCV library.

[0883] The server then loads the preprocessed image data into a multimodal AI model for image analysis. The analysis results are converted into a dataset containing room or store characteristics (size, furniture arrangement, wall color, amount of light, etc.). The AI ​​model used here is a pre-trained generative AI model.

[0884] Based on the analysis results, the server accesses internal and external databases to search for products and display items that match the user's preferences and budget. Product selection uses specific keyword search and filtering techniques, which are implemented by the request library. For example, if a user enters a prompt such as "Please suggest items in a natural style that cost less than 100,000 yen," the server will perform a search based on this information.

[0885] The server selects appropriate products and display items from the search results and retrieves their detailed information (price, images, purchase links, etc.). Based on this, the server generates a suitable interior design proposal for the user, including specific furniture and display item placement, color combinations, accessory selection, etc.

[0886] The generated proposals are sent to the device in HTML or JSON format. The proposals are visually displayed on the user's device based on the received data, allowing the user to check the proposed interior design. By clicking on a product they like, they can be taken to the online shopping site that offers that product and complete the purchase process.

[0887] For example, if a user uploads a photo of their living room and enters "natural style with a budget of 100,000 yen," the server analyzes the characteristics of the living room and selects furniture (e.g., a wooden sofa, a beige rug) and accessories (e.g., green potted plants) suitable for the natural style within the user's budget. It then provides an interior design proposal, including specific placements for these items and links to purchase them.

[0888] This system allows users to easily receive store interior designs using their smartphones or HMDs, and quickly find and purchase the ideal interior products. This process is an effective way to resolve users' concerns about interior design and significantly reduce time and effort.

[0889] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0890] Step 1:

[0891] The user captures image data of a room or store using a terminal and uploads it to the system.

[0892] Inputs are image data captured by a camera, preferred interior style, and budget information.

[0893] In operation, the user takes an image using a smartphone or head-mounted display (HMD) and sends it to the server using the application's upload function.

[0894] The output is image data and user preference data sent to the server.

[0895] Step 2:

[0896] The server preprocesses the received image data.

[0897] The input is image data and preference information submitted by the user.

[0898] In operation, the server uses the OpenCV library to adjust the image resolution and convert it to the appropriate file format. This preprocessing makes the image suitable for analysis.

[0899] The output is pre-processed image data.

[0900] Step 3:

[0901] The server loads the preprocessed image data into a multimodal AI model and performs image analysis.

[0902] The input is the preprocessed image data.

[0903] In operation, the server inputs data into a multimodal AI model and performs image analysis, which extracts characteristics of the room or store (size, furniture arrangement, wall color, amount of light, etc.).

[0904] The output is characteristic data based on the analysis results.

[0905] Step 4:

[0906] Based on the image analysis results, the server selects products that suit the user's preferences and budget.

[0907] The inputs are the image analysis results data, user preferences, and budget information.

[0908] In operation, the server accesses internal and external databases to search for products and display items that meet the user's preferences and budget, using specific keyword search and filtering techniques.

[0909] The output is a list of selected products with their details (price, image, and purchase link).

[0910] Step 5:

[0911] The server generates and provides selected product and display item suggestions to the user.

[0912] The input is a product list and detailed information.

[0913] The server then sends the generated proposals in HTML or JSON format to the device, and creates an interior design including specific layouts and color combinations.

[0914] The output is the proposal sent to the user terminal.

[0915] Step 6:

[0916] The user checks the proposed content on the terminal, clicks on the product they like, and is taken to an online shopping site where they can complete the purchase.

[0917] The input is the proposal sent by the server.

[0918] The user visually checks the recommendations and browses detailed information about the recommended products. When the user clicks on a product they like, they are taken to a shopping site where they can complete the purchase process.

[0919] The output is to complete the purchase of the product.

[0920] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0921] The present invention provides a system that allows users to upload image data of their rooms and proposes interior design suggestions based on their preferences, budget, and emotional state. The image data uploaded by the user includes photos and floor plans of the rooms, which the user sends to the system through an interface. The user also inputs or has the system detect their preferred interior style, budget, and real-time emotional state.

[0922] When the device receives user input data, it sends it to the server. The server preprocesses the received image data (adjusting resolution, converting formats, etc.), and then uses multimodal AI to analyze the room's characteristics. This analysis includes the room's size, furniture layout, wall color, amount of light, etc. The analysis results are stored in an internal database.

[0923] Next, the device uses an emotion engine to recognize the user's emotional state. The emotion engine analyzes the user's input data and biometric data (facial expressions, voice tone, heart rate, etc.) to detect whether the user is feeling joy, surprise, sadness, anger, etc. The detected emotional state is also sent to the server.

[0924] The server generates appropriate interior style and furniture layout suggestions by taking into consideration the room's characteristics, the user's emotional state, as well as their preferences and budget. In this process, optimization is performed according to the user's emotional state; for example, if the user's emotional state is positive, suggestions for a bright and lively style are prioritized, whereas if the user's emotional state is negative, suggestions for a more subdued style are prioritized.

[0925] The server also searches external databases (such as the API of an online shopping site) to select interior products that meet the user's requirements, taking into consideration factors such as price, design, and user reviews. Detailed information about the selected products (price, images, purchase links, etc.) is also obtained.

[0926] Once the proposals are generated, the server sends them to the device. The device then displays the received proposals on its user interface. The user can review the displayed interior designs and view detailed information and purchase links for products they like. By clicking on a product, they can be taken directly to an online shopping site and complete the purchase process.

[0927] For example, suppose a user uploads a photo of their living room, specifies a preference for a natural style, and sets a budget of 100,000 yen. Furthermore, suppose the emotion engine recognizes the user's smile and detects a happy emotional state. The server analyzes the characteristics of the living room and suggests natural-style furniture and accessories (e.g., a wooden sofa or a beige rug). These suggestions are chosen to create a bright and cheerful atmosphere. Detailed information and purchase links for the selected products are also included, allowing the user to easily purchase them.

[0928] In this way, by using the system of the present invention, users can receive interior design suggestions that take into consideration their own emotional state, and easily find and purchase their ideal interior design items.

[0929] The processing flow will be explained below.

[0930] Step 1:

[0931] The user uploads image data of the room (e.g., a photo of the living room) to the system using a terminal. In addition, the user inputs their preferred interior style (e.g., natural) and budget (e.g., 100,000 yen) through the interface.

[0932] Step 2:

[0933] The device sends the uploaded image data, user preferences, and budget information to a server, where the data is packaged in a suitable format and transmitted using a secure communication protocol.

[0934] Step 3:

[0935] The server temporarily stores the received data. The server then preprocesses the received image data, adjusting the resolution and converting the format.

[0936] Step 4:

[0937] The server inputs the preprocessed image data into the multimodal AI for image analysis, which detects the room's characteristics (size, furniture arrangement, wall color, window position, amount of light, etc.). The analyzed characteristic data is stored in an internal database.

[0938] Step 5:

[0939] The device uses an emotion engine to recognize the user's emotional state. The emotion engine analyzes the user's biometric data (facial expressions, tone of voice, heart rate, etc.) to detect emotional states such as joy, surprise, sadness, and anger. The detected emotional data is sent to the server.

[0940] Step 6:

[0941] The server combines and analyzes the room's characteristic data, the user's preferences, budget, and emotional data to generate appropriate interior design suggestions. If the emotional state is positive, a bright and colorful interior style is prioritized, while if the emotional state is negative, a calm style is prioritized.

[0942] Step 7:

[0943] The server searches an external database (e.g., an API for an online shopping site) to select interior products that match the user's tastes and budget. The external database contains information such as prices, designs, and user reviews. Detailed information about the selected products (price, images, purchase links, etc.) is also retrieved.

[0944] Step 8:

[0945] The server generates interior design suggestions, including furniture placement, color schemes, and accessory selections based on the user's preferences, budget, and emotional state. The suggestions are formatted in HTML or JSON.

[0946] Step 9:

[0947] The server sends the generated proposal to the terminal, which displays the received data on the user interface, and the user checks the interior design displayed on the interface.

[0948] Step 10:

[0949] When the user clicks on the suggested product details or purchase link, the device will be redirected to the corresponding page on the online shopping site, where the user can purchase the product directly.

[0950] For example, if a user uploads a photo of their living room, selects a natural style and a budget of 100,000 yen, and the emotion engine detects the user's smile and recognizes their emotional state of joy, the server will suggest natural-style interior items that create a bright and cheerful atmosphere (e.g., a wooden sofa, a beige rug, etc.). These suggestions also include specific placement locations, color schemes, and accessories (e.g., green potted plants), allowing users to easily find and purchase the ideal interior items.

[0951] Example 2

[0952] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0953] Conventional interior design suggestion systems generally suggest interior styles based on the user's preferences and budget, but they are unable to take into account the user's real-time emotional state. As a result, they are unable to provide optimal interior suggestions that reflect the user's momentary psychological state. This makes it difficult to meet the diverse needs of users, and more personalized suggestions are needed.

[0954] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0955] In this invention, the server includes means for a user to upload image data of a room, means for adjusting the resolution of the image and converting the format, means for analyzing the room size, furniture layout, wall color, amount of light, etc., means for analyzing the user's biometric data to recognize the user's emotional state, means for generating suggestions for interior styles and furniture layouts based on the analysis results and the user's preferences, budget, and emotional state, means for searching an external database to obtain detailed information and purchase links for products that meet the user's requirements, and means for displaying the generated suggestions on a user interface. This enables personalized interior suggestions that take the user's real-time emotional state into consideration.

[0956] "User" refers to an individual who uses the system to receive room interior suggestions.

[0957] "Room image data" refers to image files such as room photos and floor plans that users upload to the system.

[0958] "Resolution adjustment" refers to the process of changing the resolution of an input image to an appropriate size.

[0959] "Format conversion" refers to the process of converting the file format of input image data into a format that can be used by the system.

[0960] "Room size" refers to information indicating the physical size of the room obtained from the analyzed image.

[0961] "Furniture arrangement" refers to information indicating the position and arrangement of furniture within a room obtained from the analyzed image.

[0962] "Wall color" refers to information indicating the color of the walls of a room obtained from the analyzed image.

[0963] "Amount of light" refers to information indicating the brightness of light in a room and the amount of light incident thereon, obtained from the analyzed image.

[0964] "Biometric data" refers to physiological data such as a user's facial expression, voice tone, heart rate, etc.

[0965] "Emotional state" refers to the emotion the user is feeling (such as joy, surprise, sadness, anger, etc.).

[0966] "Interior style" refers to interior design styles such as natural, modern, and classic.

[0967] "Furniture arrangement suggestions" refers to specific suggestions showing the optimal furniture arrangement and interior style for the user's room.

[0968] "External Database" refers to a database for obtaining product information from online shopping sites and other external resources.

[0969] "Detailed Information" refers to information about the selected product, including price, images, and purchase links.

[0970] "User interface" refers to the screen and operation method that allows a user to interact with a system.

[0971] The present invention is a system that allows users to upload image data of a room and receives interior design suggestions based on the user's preferences, budget, and real-time emotional state. Specific hardware and software used to implement the invention, as well as methods for processing and calculating data, and specific examples are described below.

[0972] Hardware and Software

[0973] A terminal is a computer or mobile device that provides a user interface and accepts data input from a user. The terminal has image processing software and an emotion engine installed.

[0974] The server is a computer system that processes and analyzes the received data. The following software is used for this analysis:

[0975] Use an image processing library (e.g., OpenCV, Pillow) to adjust image resolution and convert image formats.

[0976] A multimodal AI model (e.g., CNN) analyzes room characteristics (size, furniture arrangement, wall color, amount of light, etc.).

[0977] Recognize the user's emotional state with an emotion recognition library (e.g., OpenFace or other facial expression recognition libraries).

[0978] Data processing and calculation

[0979] When a user launches the system, an interface appears. The user uploads a photo of the room (e.g., a photo of the living room) and a floor plan, inputs the interior style (e.g., natural style) and budget (e.g., 100,000 yen), and provides facial expressions and voice data to detect emotional states.

[0980] The device sends the received data to the server, which then preprocesses the received image data, adjusting the resolution and converting the format, and analyzes the room size, furniture layout, wall color, amount of light, etc.

[0981] Based on this data, the server uses a generative AI model to generate optimal interior style and furniture arrangement suggestions tailored to the user's preferences, budget, and emotional state, with example prompts such as:

[0982] Prompt: "Picture of a living room, natural style, budget of ¥100,000, user smiling (happy)"

[0983] The server searches an external database to select interior products that meet the user's criteria. It selects the most suitable product by comprehensively considering factors such as price, design, and user reviews. It also obtains detailed information about the selected product (price, images, purchase links, etc.).

[0984] Finally, the server sends the generated recommendations to the terminal, which displays them on the user interface. The user can then check the recommendations, view detailed information and purchase links, and purchase the products they are interested in directly from the online shopping site.

[0985] For example, suppose a user uploads a photo of their living room, specifies a preference for a natural style, and sets a budget of 100,000 yen. Furthermore, suppose the emotion engine recognizes the user's smile and detects a happy emotional state. The server analyzes the characteristics of the living room and suggests natural-style furniture and accessories (e.g., a wooden sofa or a beige rug). These suggestions are chosen to create a bright and cheerful atmosphere. Detailed information and purchase links for the selected products are also included, allowing the user to easily purchase them.

[0986] In this way, by using the system of the present invention, users can receive interior design suggestions that take into consideration their own emotional state, and easily find and purchase their ideal interior design items.

[0987] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0988] Step 1: System startup and data entry

[0989] When the user boots the system, the interface is displayed.

[0990] Users upload photos of their rooms (e.g., photos of their living rooms) and floor plans.

[0991] The user inputs the interior style (e.g., natural style) and budget (e.g., 100,000 yen) through the interface.

[0992] The user provides facial expression and voice data to detect the emotional state, which then provides the emotional data.

[0993] Step 2: Receiving data and sending it to the server

[0994] The terminal receives input data from the user (room image data, preferences, budget, and emotional data).

[0995] The terminal sends the received data to the server. Input is the user's input data, which the terminal sends to the server.

[0996] Step 3: Preprocessing the image data

[0997] The server preprocesses the image data it receives, specifically adjusting the image resolution and converting the format.

[0998] The server uses the OpenCV and Pillow libraries for this process.

[0999] For example, downscaling a high-resolution image and converting it to an appropriate image format (JPEG or PNG). The input is the original image data and the output is the preprocessed image data.

[1000] Step 4: Analyzing the image data

[1001] The server uses the pre-processed image data to analyze the characteristics of the room.

[1002] The server uses a multimodal AI model (e.g., CNN) to determine the size of the room, furniture arrangement, wall color, amount of light, etc.

[1003] The analysis results are stored in an internal database. The input is preprocessed image data, and the output is the analysis results.

[1004] Step 5: Recognizing your emotional state

[1005] The device uses an emotion engine to recognize the user's emotional state, using data such as facial expressions, voice tone, and heart rate.

[1006] The device uses OpenFace and other facial recognition libraries for emotion recognition.

[1007] The emotional state is recognized and sent to the server. The input is the user's biometric data and the output is the emotional state data.

[1008] Step 6: Generate proposals

[1009] The server generates appropriate interior style and furniture arrangement suggestions taking into account the room's characteristics, the user's emotional state, preferences, and budget.

[1010] The server uses a generative AI model to create suggestions based on the input (prompt).

[1011] Prompt: "Picture of a living room, natural style, budget of ¥100,000, user smiling (happy)"

[1012] A proposal is generated. The inputs are the room characteristic data, the emotional state data, the user's preferences and budget, and the output is the generated proposal.

[1013] Step 7: Product Selection

[1014] The server searches an external database and selects interior products that meet the user's requirements.

[1015] The server selects the most suitable product by comprehensively considering price, design, user reviews, etc. Detailed information about the selected product (price, image, purchase link, etc.) is obtained.

[1016] The input is the user's conditions and proposal details, and the output is the selected product information.

[1017] Step 8: View the results

[1018] The server sends the generated proposal to the terminal.

[1019] The terminal displays the received proposal on a user interface.

[1020] Users can review the suggestions displayed, view detailed information and purchase links, and select the appropriate product.

[1021] The input is the proposal content and product information, and the output is the proposal and product information displayed on the user interface.

[1022] (Application example 2)

[1023] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1024] Conventional interior design suggestion systems can select interior products based on the user's preferences and budget, but they do not consider the user's emotional state when making suggestions. This makes it difficult to provide the optimal interior design that matches the user's psychological state. Furthermore, there is a lack of a way to easily access detailed information and purchase links on the spot.

[1025] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1026] In this invention, the server includes means for a user to upload image data of a room, means for receiving the uploaded image data and performing image analysis, means for selecting products that match the user's preferences, budget, and emotional state based on the image analysis results, means for generating proposals for the selected products and providing them to the user, means for analyzing the user's facial expression images to detect the user's emotional state, means including an emotion engine with the function of analyzing the emotional state in real time, and means for recommending interior styles according to the emotional state. This makes it possible to propose an optimal interior design that matches the user's real-time emotional state, and further allows the user to access detailed information and a purchase link on the spot.

[1027] "User" refers to an individual or organization that uses the system to upload image data of a room and receive interior design suggestions.

[1028] "Image data" refers to digital image files such as room photos and floor plans uploaded by users.

[1029] "Upload" refers to the act of a user sending image data from their device to a server.

[1030] "Image analysis" refers to the process by which the server preprocesses the image data it receives and analyzes the characteristics of the room.

[1031] "Preferences" refer to personal preferences regarding the interior style and design desired by the user.

[1032] "Budget" refers to the upper limit of costs set by the user when selecting interior products.

[1033] "Emotional state" refers to the psychological state of the user, which is analyzed using the emotion engine.

[1034] An "emotion engine" refers to an algorithm or software that analyzes a user's facial expression images and biometric data to detect their emotional state.

[1035] "Server" refers to a computer system that processes image data and other information received from users and generates and provides interior design proposals.

[1036] "Suggestions" refers to the interior design and product recommendations generated by the system.

[1037] "External databases" refer to online shopping sites and other information sources on the Internet that provide reference information for product selection.

[1038] "Product selection" refers to the process by which the system selects the most suitable interior products based on the user's preferences, budget, and emotional state.

[1039] "Real-time" refers to the process by which a system processes user input or detected information immediately and generates and delivers results.

[1040] This invention provides a system that allows a user to upload image data of a room and proposes interior design ideas based on the user's preferences, budget, and emotional state. The specific system design and operation are described below.

[1041] server

[1042] The server is a computer system that processes image data and other information received from users and generates and provides interior design proposals. The server includes the following means:

[1043] 1. A function to receive image data and perform preprocessing (resolution adjustment, format conversion, etc.).

[1044] 2. Room characteristic analysis function using multimodal AI (analyzing room size, furniture arrangement, wall color, amount of light, etc.).

[1045] 3. A function that uses an emotion engine to analyze the user's emotional state (analyzing facial expressions, voice, heart rate, etc. to detect joy, surprise, sadness, anger, etc.).

[1046] 4. A feature that suggests the best interior style and furniture arrangement based on the user's preferences, budget, and emotional state.

[1047] 5. The ability to search external databases (e.g., online shopping sites) and select appropriate interior products.

[1048] Terminal

[1049] A terminal is a device that allows users to access the system and input information. This terminal can be a smartphone, tablet, or PC. The specific operation steps are as follows:

[1050] 1. The user takes a photo of the room using the device and uploads the image data.

[1051] 2. The user inputs their preferred interior style and budget through the interface.

[1052] 3. The device captures the user's facial expressions and sends the data to the emotion engine in real time.

[1053] Processing Flow

[1054] Hardware and software configuration

[1055] Hardware: Smartphone (camera function, network connection), server (high performance computer).

[1056] Software: Python, PIL (image preprocessing), Emotion Engine (sentiment analysis), Interior Style Recommender (interior suggestions).

[1057] The server first preprocesses the image data of the room received from the user (adjusting the resolution, converting the format, etc.). It then uses multimodal AI to analyze the room's size, furniture arrangement, wall color, amount of light, etc. The analysis results are stored in a database on the server. At the same time, the device uses an emotion engine to analyze the user's emotional state, and this information is also sent to the server.

[1058] The server takes into account the room's characteristics, the user's emotional state, preferences, and budget to generate suggestions for appropriate interior styles and furniture arrangements. These suggestions also include product information (price, images, purchase links, etc.) retrieved from an external database. Once the suggestions are generated, the server sends them to the device and provides them to the user.

[1059] Examples and prompts

[1060] For example, if a user uploads a photo of their living room with a natural style and a budget of 100,000 yen, the system works as follows: The emotion engine recognizes the user's smile and detects a happy emotional state. The server analyzes the characteristics of the living room and suggests natural-style furniture and accessories (e.g., a wooden sofa, a beige rug). The suggestions are selected to create a bright and cheerful atmosphere.

[1061] Prompt Sentence Examples

[1062] "Upload a picture of your room: room.jpg"

[1063] "Please enter your preferred interior style: Natural"

[1064] "Please enter your budget: 100,000 yen"

[1065] "Upload a photo of your face: user.jpg"

[1066] This allows users to easily find and purchase their ideal interior items through specific interior suggestions.

[1067] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1068] Step 1:

[1069] The user uses the terminal to take a picture of the room and upload it.

[1070] Input: A photo of the room.

[1071] Output: The uploaded image data.

[1072] Specific operation: The user takes a photo of the room using the camera function of their smartphone, and then presses the upload button in the application to send the image data to the server.

[1073] Step 2:

[1074] The server pre-processes the image data received from the user.

[1075] Input: Uploaded image data.

[1076] Output: Preprocessed image data.

[1077] What it does: It adjusts the image resolution on the server and converts it to the required format (e.g. resize to 256x256 pixels). It uses Python and the PIL library for preprocessing.

[1078] Step 3:

[1079] The server analyzes the pre-processed image data to understand the characteristics of the room.

[1080] Input: Preprocessed image data.

[1081] Output: Analysis results such as room size, furniture placement, wall color, and amount of light.

[1082] Specific operation: A multimodal AI system on the server analyzes image data and extracts room characteristics. The extracted data is stored in an internal database.

[1083] Step 4:

[1084] The user inputs their preferred interior style and budget through the interface.

[1085] Input: The user's interior style preferences and budget.

[1086] Output: Interior style preferences, budget data.

[1087] What it does: Enter style and budget through a user interface using drop-down menus and text input forms.

[1088] Step 5:

[1089] The terminal uses an emotion engine to detect the user's emotional state.

[1090] Input: An image of the user's facial expression.

[1091] Output: Detected emotional state data.

[1092] Specific operation: The smartphone camera captures the user's facial expression and sends it to the server in real time. The emotion engine analyzes the facial expression data and detects the user's emotional state (e.g., joy, sadness, surprise, etc.).

[1093] Step 6:

[1094] The server suggests interior styles and furniture arrangements based on the room's characteristics, the user's preferences, budget, and emotional state.

[1095] Input: Room characteristic data, interior style preference, budget data, emotional state data.

[1096] Output: Interior design proposals (style, furniture arrangement, specific products).

[1097] What it does: The algorithm in the server generates the optimal interior design based on all input data, and then retrieves and finalizes the proposal from internal and external databases.

[1098] Step 7:

[1099] The server sends the selected product details and a link to purchase the product to the device.

[1100] Input: Interior design proposal.

[1101] Output: The suggestions that will be displayed on the user's device.

[1102] Specific operation: The server sends the generated interior design proposal to the device and displays detailed information (price, image, purchase link) on the interface.

[1103] Step 8:

[1104] The user reviews the offer and clicks the purchase link to purchase the product.

[1105] Input: Proposed interior product.

[1106] Output: Transition to online shopping site and purchase process.

[1107] Specific actions: The user checks the suggestions displayed on the device and clicks on the purchase link for the product they are interested in. They then complete the purchase at the linked online shopping site.

[1108] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1109] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1110] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1111] [Fourth embodiment]

[1112] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1113] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1114] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1115] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1116] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1117] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1118] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1119] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1120] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1121] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1122] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1123] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1124] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1125] The present invention begins with the user uploading image data of a room. Using a terminal connected to the Internet, the user sends photos and floor plans of the rooms in their home to the system. The user also operates an interface to input their preferred interior style and budget.

[1126] The device sends the image data, preferences, and budget information entered by the user to the server. The server temporarily stores the received data and performs preprocessing on the image data, including adjusting the resolution and converting the file format.

[1127] The server then loads the pre-processed image data into a multimodal AI system for image analysis, which detects characteristics such as the room size, furniture arrangement, wall color, window position, and amount of light, which are then stored in an internal database.

[1128] Based on the analysis results, the server applies the user's preferences and budget information and performs calculations to suggest an appropriate interior style and furniture arrangement. At this stage, the server accesses an external database (e.g., an online shopping site) to search for products that match the user's preferences and budget. Specific keyword search and filtering techniques are used to select products.

[1129] The server selects appropriate products from the search results and obtains detailed information (price, images, and purchase links) for each product. Based on this information, the server generates interior design suggestions suitable for the user. Specifically, the suggestions include furniture placement, color combinations, and accessory selection.

[1130] The generated proposals are sent to the device in HTML or JSON format. The device displays the received proposals on a user interface. The user can check the proposed interior designs and click on a product they like to be taken to an online shopping site that offers that product and complete the purchase process.

[1131] For example, if a user uploads a photo of their living room and inputs a natural style and a budget of 100,000 yen, the server will analyze the living room's characteristics (size, lighting conditions, etc.) and select furniture (wooden sofa, beige rug, etc.) and accessories (green potted plants, etc.) that are suitable for the natural style and within the user's budget. It will then provide an interior design proposal including specific placements for these items and links to purchase them.

[1132] This allows users to receive interior design suggestions based on their preferences and budget, and easily find and purchase the ideal interior products. This system solves users' interior design concerns and significantly reduces time and effort.

[1133] The processing flow will be explained below.

[1134] Step 1:

[1135] The user uploads photos and floor plans of the room using the device, and the device displays an interface for selecting the image data and inputting the user's preferred interior style (modern, natural, Scandinavian, etc.) and budget (e.g., 100,000 yen).

[1136] Step 2:

[1137] The device sends the uploaded image data, along with information about the user's preferences and budget, to the server, where it is packaged in an appropriate format and transmitted using a secure communication protocol.

[1138] Step 3:

[1139] The server temporarily stores the received data. The server then performs pre-processing on the received image data. This pre-processing includes adjusting the image resolution and converting the file format.

[1140] Step 4:

[1141] The server inputs the preprocessed image data into the multimodal AI for image analysis. Here, the AI ​​identifies the characteristics of the room. Specifically, it analyzes the room's size, furniture layout, wall color, window position, amount of lighting, etc. The results of this analysis are stored in the server's internal database.

[1142] Step 5:

[1143] The server performs calculations to propose an appropriate interior style based on the analysis results and the user's preferences. The calculations include furniture placement and color selection according to the characteristics of the room. The server searches an external database (such as an API of an online shopping site) to select appropriate interior items based on the user's preferences and budget.

[1144] Step 6:

[1145] The server filters the list of products retrieved from an external database and selects the best product that matches the user's criteria, taking into account factors such as price, design, and user reviews.

[1146] Step 7:

[1147] The server generates recommendations based on the selected product details (price, images, purchase links, etc.), including specific furniture placement, color schemes, and accessory selections.

[1148] Step 8:

[1149] The server sends the generated proposal to the device. The sent data is in HTML or JSON format and is organized in a format that the device can display appropriately.

[1150] Step 9:

[1151] The device displays the received proposals on a user interface, allowing the user to review the proposed interior designs and click on the desired product to view more information and a link to purchase it.

[1152] Step 10:

[1153] When the user clicks on the purchase link for the suggested product, the terminal will be redirected to the page of the corresponding online shopping site, where the user can purchase the product directly.

[1154] This process allows users to easily find interior designs and products that fit their tastes and budget, and smoothly move on to actual purchases.

[1155] Example 1

[1156] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1157] Conventional interior design suggestion systems require users to manually select products, making it easy for them to become overwhelmed by the vast amount of information, and making it difficult to efficiently find an interior design that suits their tastes and budget. Furthermore, insufficient image analysis can lead to the inability to accurately grasp the characteristics of a room and make appropriate suggestions. This can make it difficult for users to obtain satisfactory interior design suggestions, requiring a great deal of time and effort.

[1158] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1159] In this invention, the server includes means for users to upload image data of a space, means for receiving the uploaded image data and performing data preprocessing, means for analyzing the preprocessed image data and extracting features of the space, means for searching for and selecting products that match the user's preferences and budget based on the analysis results using a generative AI model, and means for generating and providing proposals for the selected products to the user. This allows users to easily obtain interior proposals that match their preferences and budget, significantly reducing time and effort.

[1160] "User" refers to an individual or group that operates the system and inputs image data of a room, preferences regarding the interior, and budget.

[1161] "Spatial image data" refers to digital images, including photographs and floor plans, of a user's own room or other space.

[1162] "Means for uploading" refers to an interface or function that allows a user to send image data or text information to a server via a terminal.

[1163] "Means for performing data preprocessing" refers to a function for performing preprocessing such as adjusting the resolution and converting the file format on received image data.

[1164] "Means for analyzing and extracting spatial characteristics" refers to the function of recognizing and identifying characteristics such as the size of a room, furniture arrangement, wall color, and amount of light based on preprocessed image data.

[1165] A "generative AI model" refers to an artificial intelligence model that can generate and analyze information from a variety of data.

[1166] "Means for searching and selecting products" refers to a function that uses an external database to find and select products that match the user's preferences and budget.

[1167] "Means for generating proposals and providing them to users" refers to the interface and functions that create interior proposals based on the selected products and present them to users.

[1168] "External Database" means a digital database containing product information and service data accessible on the Internet.

[1169] This invention is a system that uses a generative AI model to make appropriate interior design suggestions by allowing users to upload image data of their room and input their interior style and budget. The system begins by users accessing the system's web application using a device connected to the Internet.

[1170] Users upload photos and floor plans of their rooms via their devices, and input their preferred interior style and budget. For example, consider a case where a user uploads a photo of their living room and inputs a natural style and a budget of 100,000 yen. The user clicks the "Upload Image" button on the web interface to select a photo from their local device, and inputs their preferred style and budget using drop-down menus and text boxes.

[1171] The device sends the uploaded image data and the entered information to the server using the HTTP POST method. The server temporarily stores the received image data and performs preprocessing such as adjusting the resolution and converting the file format.

[1172] The server then passes the preprocessed image data to the multimodal AI, which uses deep learning models to identify characteristics such as room size, furniture placement, wall color, window position, and light intensity, and stores this data in an internal database.

[1173] Based on the analysis results, the server runs a recommendation algorithm using the user's preferences and budget information to calculate the appropriate interior style and furniture layout. During this process, the server sends an API request to an external database to search for products that match the user's preferences and budget. An example of an external database is the API of an online shopping site. In this process, keyword searches and price filtering are used to retrieve appropriate product information. The retrieved information includes the product's image URL, price, and purchase link.

[1174] Based on this information, the server creates interior design proposals, including specific furniture placement locations, color combinations to be used, and accessory selections, and generates data to provide to the user in HTML or JSON format.

[1175] The server sends the generated interior design proposals to the user's device. The device analyzes the received HTML and JSON data and displays the proposals on a user interface. The user can check the displayed interior designs and click on a product they like to be taken to the online shopping site that provides the product and complete the purchase process.

[1176] For example, by inputting the following prompt sentence into a generative AI model, interior design suggestions based on the user's requirements can be obtained.

[1177] Prompt: Analyze an image of a living room and create a natural-style interior design proposal that fits within a budget of ¥100,000. Attach an image of the living room, select items that are available within your budget, and suggest placement, color combinations, and accessory choices. Also provide a link to purchase each item.

[1178] This method allows users to easily receive interior design suggestions that match their preferences and budget, and also simplifies the purchasing process, making it possible to efficiently create the ideal interior.

[1179] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1180] Step 1:

[1181] Users use an internet-connected device to upload photos and floor plans of their rooms to the system, and also operate the interface to input their preferred interior style and budget.

[1182] Input: room image data, preferred interior style, budget

[1183] Output: The HTTP request that the user input is sent to the server.

[1184] Step 2:

[1185] The device sends the image data, preferences, and budget information entered by the user to the server using the HTTP POST method.

[1186] Input: User-entered image data, preferences, and budget

[1187] Output: Image data and user information arrive at the server as an HTTP request

[1188] Step 3:

[1189] The server temporarily stores the received data and performs pre-processing on the image data, including adjusting the resolution and converting the file format to JPEG or PNG.

[1190] Input: Image data received from the user

[1191] Output: Preprocessed image data (resolution adjusted, format converted)

[1192] Step 4:

[1193] The server then passes the preprocessed image data to a multimodal AI system that begins analyzing the image, using deep learning models to recognize characteristics such as the room's size, furniture placement, wall color, window position, and amount of light.

[1194] Input: Preprocessed image data

[1195] Output: Room characteristics data (area, furniture layout, wall color, window position, amount of light, etc.)

[1196] Step 5:

[1197] The server stores the results of the image analysis in an internal database.

[1198] Input: Room characteristic data

[1199] Output: Room characteristics data stored in a database

[1200] Step 6:

[1201] The server runs a recommendation algorithm based on the user's preferences and budget information to calculate the appropriate interior style and furniture layout. This calculation process involves sending API requests to an external database to search for products that match the user's preferences and budget.

[1202] Input: User preferences, budget information, room characteristics data

[1203] Output: Suitable interior style, furniture layout information, product candidates

[1204] Step 7:

[1205] The server selects appropriate products from the search results and retrieves detailed information for each product (price, image, purchase link).

[1206] Input: Product information as search results

[1207] Output: Selected product details (price, image, purchase link)

[1208] Step 8:

[1209] The server creates an interior design proposal, including specific furniture placement, color combinations, and accessory selection, and generates the proposal in HTML or JSON format.

[1210] Input: Selected product details, user preferences, budget information, room characteristics data

[1211] Output: Interior proposals in HTML or JSON format

[1212] Step 9:

[1213] The server sends the generated proposal to the user's device using an HTTP response.

[1214] Input: Interior proposal in HTML or JSON format

[1215] Output: HTTP response containing the proposal

[1216] Step 10:

[1217] The device parses the received suggestions and displays them in a user interface, while the front end parses the HTML and JSON data and renders images and text appropriately.

[1218] Input: HTTP response from the server (proposal)

[1219] Output: Interior proposals displayed on the user interface

[1220] Step 11:

[1221] The user can check the proposed interior designs and click on the product they like. When they click, the device opens a browser and goes to the corresponding product page on the online shopping site.

[1222] Input: Interior proposal displayed on the user interface

[1223] Output: Transition to online shopping site and start of purchase process

[1224] (Application example 1)

[1225] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1226] Many conventional interior design support systems analyze image data and propose products for a specific room in a user's home. However, providing a similar service in a physical store presents challenges, such as the unique size and characteristics of the store, as well as the appropriate placement of display items. This makes it difficult for store owners and interior designers to quickly determine effective product placement and design. The present invention aims to solve these problems and provide a system that easily proposes interior designs and product placements for physical stores.

[1227] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1228] In this invention, the server includes means for allowing a user to upload image data of a room, means for receiving the uploaded image data and performing image analysis, means for selecting products that match the user's preferences and budget based on the image analysis results, means for generating and providing suggestions for the selected products to the user, means for capturing and analyzing image data of the store, and means for suggesting display items for the store based on the image analysis results. This makes it easy to arrange and design products in a physical store in accordance with the user's preferences and budget.

[1229] A "user" is a person or organization that uses the system to receive proposals for interior design of a room or store.

[1230] A "living room" is a space in a residence where a user actually lives, and includes a room, a living room, a bedroom, etc.

[1231] "Image data" refers to digitally stored still or video images that contain visual information about a room or store.

[1232] "Upload" refers to the operation of a user sending image data from their own device to the system's server via the Internet.

[1233] "Means" refers to a method or component that enables a system or device to achieve a specific function or purpose.

[1234] "Receiving" refers to the process in which the server takes in image data sent by the user.

[1235] "Image analysis" is the process of analyzing received image data to extract characteristics of a room or store, furniture layout, wall color, amount of light, etc.

[1236] "Results" refers to the data or information obtained after performing image analysis.

[1237] "Preferences" refers to personal preference information such as interior style, color, and theme that the user has input into the system in advance.

[1238] "Budget" refers to the range of amounts set by the user for interior design and product purchases.

[1239] "Products" refers to furniture, decorations, and display items necessary for the interior design of a user's room or store.

[1240] "Selection" refers to the process of choosing products that suit the user's preferences and budget based on the results of image analysis.

[1241] "Proposal content" refers to the interior design and product placement plan shown to the user, including specific placement proposals and detailed information about the products to be used.

[1242] "Providing" refers to the process by which the system presents the suggestions to the user through the user interface.

[1243] The present invention is a system that allows users to upload image data of a room or a brick-and-mortar store, and then proposes interior designs and product layouts based on that data. This system involves a series of processes: sending image data from the user's device to a server, analyzing it, generating proposals, and providing them to the user.

[1244] First, the user captures image data of a room in their home or a brick-and-mortar store using a device and uploads it to the system. The device is equipped with a camera, and can be, for example, a smartphone or a head-mounted display (HMD). In addition to the image data, the user also enters their preferred interior style and budget information. This information is then sent to the server by the program and temporarily stored.

[1245] The server preprocesses the received image data, which includes adjusting the image resolution and converting the file format. To perform this processing, the server uses the OpenCV library.

[1246] The server then loads the preprocessed image data into a multimodal AI model for image analysis. The analysis results are converted into a dataset containing room or store characteristics (size, furniture arrangement, wall color, amount of light, etc.). The AI ​​model used here is a pre-trained generative AI model.

[1247] Based on the analysis results, the server accesses internal and external databases to search for products and display items that match the user's preferences and budget. Product selection uses specific keyword search and filtering techniques, which are implemented by the request library. For example, if a user enters a prompt such as "Please suggest items in a natural style that cost less than 100,000 yen," the server will perform a search based on this information.

[1248] The server selects appropriate products and display items from the search results and retrieves their detailed information (price, images, purchase links, etc.). Based on this, the server generates a suitable interior design proposal for the user, including specific furniture and display item placement, color combinations, accessory selection, etc.

[1249] The generated proposals are sent to the device in HTML or JSON format. The proposals are visually displayed on the user's device based on the received data, allowing the user to check the proposed interior design. By clicking on a product they like, they can be taken to the online shopping site that offers that product and complete the purchase process.

[1250] For example, if a user uploads a photo of their living room and enters "natural style with a budget of 100,000 yen," the server analyzes the characteristics of the living room and selects furniture (e.g., a wooden sofa, a beige rug) and accessories (e.g., green potted plants) suitable for the natural style within the user's budget. It then provides an interior design proposal, including specific placements for these items and links to purchase them.

[1251] This system allows users to easily receive store interior designs using their smartphones or HMDs, and quickly find and purchase the ideal interior products. This process is an effective way to resolve users' concerns about interior design and significantly reduce time and effort.

[1252] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1253] Step 1:

[1254] The user captures image data of a room or store using a terminal and uploads it to the system.

[1255] Inputs are image data captured by a camera, preferred interior style, and budget information.

[1256] In operation, the user takes an image using a smartphone or head-mounted display (HMD) and sends it to the server using the application's upload function.

[1257] The output is image data and user preference data sent to the server.

[1258] Step 2:

[1259] The server preprocesses the received image data.

[1260] The input is image data and preference information submitted by the user.

[1261] In operation, the server uses the OpenCV library to adjust the image resolution and convert it to the appropriate file format. This preprocessing makes the image suitable for analysis.

[1262] The output is pre-processed image data.

[1263] Step 3:

[1264] The server loads the preprocessed image data into a multimodal AI model and performs image analysis.

[1265] The input is the preprocessed image data.

[1266] In operation, the server inputs data into a multimodal AI model and performs image analysis, which extracts characteristics of the room or store (size, furniture arrangement, wall color, amount of light, etc.).

[1267] The output is characteristic data based on the analysis results.

[1268] Step 4:

[1269] Based on the image analysis results, the server selects products that suit the user's preferences and budget.

[1270] The inputs are the image analysis results data, user preferences, and budget information.

[1271] In operation, the server accesses internal and external databases to search for products and display items that meet the user's preferences and budget, using specific keyword search and filtering techniques.

[1272] The output is a list of selected products with their details (price, image, and purchase link).

[1273] Step 5:

[1274] The server generates and provides selected product and display item suggestions to the user.

[1275] The input is a product list and detailed information.

[1276] The server then sends the generated proposals in HTML or JSON format to the device, and creates an interior design including specific layouts and color combinations.

[1277] The output is the proposal sent to the user terminal.

[1278] Step 6:

[1279] The user checks the proposed content on the terminal, clicks on the product they like, and is taken to an online shopping site where they can complete the purchase.

[1280] The input is the proposal sent by the server.

[1281] The user visually checks the recommendations and browses detailed information about the recommended products. When the user clicks on a product they like, they are taken to a shopping site where they can complete the purchase process.

[1282] The output is to complete the purchase of the product.

[1283] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1284] The present invention provides a system that allows users to upload image data of their rooms and proposes interior design suggestions based on their preferences, budget, and emotional state. The image data uploaded by the user includes photos and floor plans of the rooms, which the user sends to the system through an interface. The user also inputs or has the system detect their preferred interior style, budget, and real-time emotional state.

[1285] When the device receives user input data, it sends it to the server. The server preprocesses the received image data (adjusting resolution, converting formats, etc.), and then uses multimodal AI to analyze the room's characteristics. This analysis includes the room's size, furniture layout, wall color, amount of light, etc. The analysis results are stored in an internal database.

[1286] Next, the device uses an emotion engine to recognize the user's emotional state. The emotion engine analyzes the user's input data and biometric data (facial expressions, voice tone, heart rate, etc.) to detect whether the user is feeling joy, surprise, sadness, anger, etc. The detected emotional state is also sent to the server.

[1287] The server generates appropriate interior style and furniture layout suggestions by taking into consideration the room's characteristics, the user's emotional state, as well as their preferences and budget. In this process, optimization is performed according to the user's emotional state; for example, if the user's emotional state is positive, suggestions for a bright and lively style are prioritized, whereas if the user's emotional state is negative, suggestions for a more subdued style are prioritized.

[1288] The server also searches external databases (such as the API of an online shopping site) to select interior products that meet the user's requirements, taking into consideration factors such as price, design, and user reviews. Detailed information about the selected products (price, images, purchase links, etc.) is also obtained.

[1289] Once the proposals are generated, the server sends them to the device. The device then displays the received proposals on its user interface. The user can review the displayed interior designs and view detailed information and purchase links for products they like. By clicking on a product, they can be taken directly to an online shopping site and complete the purchase process.

[1290] For example, suppose a user uploads a photo of their living room, specifies a preference for a natural style, and sets a budget of 100,000 yen. Furthermore, suppose the emotion engine recognizes the user's smile and detects a happy emotional state. The server analyzes the characteristics of the living room and suggests natural-style furniture and accessories (e.g., a wooden sofa or a beige rug). These suggestions are chosen to create a bright and cheerful atmosphere. Detailed information and purchase links for the selected products are also included, allowing the user to easily purchase them.

[1291] In this way, by using the system of the present invention, users can receive interior design suggestions that take into consideration their own emotional state, and easily find and purchase their ideal interior design items.

[1292] The processing flow will be explained below.

[1293] Step 1:

[1294] The user uploads image data of the room (e.g., a photo of the living room) to the system using a terminal. In addition, the user inputs their preferred interior style (e.g., natural) and budget (e.g., 100,000 yen) through the interface.

[1295] Step 2:

[1296] The device sends the uploaded image data, user preferences, and budget information to a server, where the data is packaged in a suitable format and transmitted using a secure communication protocol.

[1297] Step 3:

[1298] The server temporarily stores the received data. The server then preprocesses the received image data, adjusting the resolution and converting the format.

[1299] Step 4:

[1300] The server inputs the preprocessed image data into the multimodal AI for image analysis, which detects the room's characteristics (size, furniture arrangement, wall color, window position, amount of light, etc.). The analyzed characteristic data is stored in an internal database.

[1301] Step 5:

[1302] The device uses an emotion engine to recognize the user's emotional state. The emotion engine analyzes the user's biometric data (facial expressions, tone of voice, heart rate, etc.) to detect emotional states such as joy, surprise, sadness, and anger. The detected emotional data is sent to the server.

[1303] Step 6:

[1304] The server combines and analyzes the room's characteristic data, the user's preferences, budget, and emotional data to generate appropriate interior design suggestions. If the emotional state is positive, a bright and colorful interior style is prioritized, while if the emotional state is negative, a calm style is prioritized.

[1305] Step 7:

[1306] The server searches an external database (e.g., an API for an online shopping site) to select interior products that match the user's tastes and budget. The external database contains information such as prices, designs, and user reviews. Detailed information about the selected products (price, images, purchase links, etc.) is also retrieved.

[1307] Step 8:

[1308] The server generates interior design suggestions, including furniture placement, color schemes, and accessory selections based on the user's preferences, budget, and emotional state. The suggestions are formatted in HTML or JSON.

[1309] Step 9:

[1310] The server sends the generated proposal to the terminal, which displays the received data on the user interface, and the user checks the interior design displayed on the interface.

[1311] Step 10:

[1312] When the user clicks on the suggested product details or purchase link, the device will be redirected to the corresponding page on the online shopping site, where the user can purchase the product directly.

[1313] For example, if a user uploads a photo of their living room, selects a natural style and a budget of 100,000 yen, and the emotion engine detects the user's smile and recognizes their emotional state of joy, the server will suggest natural-style interior items that create a bright and cheerful atmosphere (e.g., a wooden sofa, a beige rug, etc.). These suggestions also include specific placement locations, color schemes, and accessories (e.g., green potted plants), allowing users to easily find and purchase the ideal interior items.

[1314] Example 2

[1315] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1316] Conventional interior design suggestion systems generally suggest interior styles based on the user's preferences and budget, but they are unable to take into account the user's real-time emotional state. As a result, they are unable to provide optimal interior suggestions that reflect the user's momentary psychological state. This makes it difficult to meet the diverse needs of users, and more personalized suggestions are needed.

[1317] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1318] In this invention, the server includes means for a user to upload image data of a room, means for adjusting the resolution of the image and converting the format, means for analyzing the room size, furniture layout, wall color, amount of light, etc., means for analyzing the user's biometric data to recognize the user's emotional state, means for generating suggestions for interior styles and furniture layouts based on the analysis results and the user's preferences, budget, and emotional state, means for searching an external database to obtain detailed information and purchase links for products that meet the user's requirements, and means for displaying the generated suggestions on a user interface. This enables personalized interior suggestions that take the user's real-time emotional state into consideration.

[1319] "User" refers to an individual who uses the system to receive room interior suggestions.

[1320] "Room image data" refers to image files such as room photos and floor plans that users upload to the system.

[1321] "Resolution adjustment" refers to the process of changing the resolution of an input image to an appropriate size.

[1322] "Format conversion" refers to the process of converting the file format of input image data into a format that can be used by the system.

[1323] "Room size" refers to information indicating the physical size of the room obtained from the analyzed image.

[1324] "Furniture arrangement" refers to information indicating the position and arrangement of furniture within a room obtained from the analyzed image.

[1325] "Wall color" refers to information indicating the color of the walls of a room obtained from the analyzed image.

[1326] "Amount of light" refers to information indicating the brightness of light in a room and the amount of light incident thereon, obtained from the analyzed image.

[1327] "Biometric data" refers to physiological data such as a user's facial expression, voice tone, heart rate, etc.

[1328] "Emotional state" refers to the emotion the user is feeling (such as joy, surprise, sadness, anger, etc.).

[1329] "Interior style" refers to interior design styles such as natural, modern, and classic.

[1330] "Furniture arrangement suggestions" refers to specific suggestions showing the optimal furniture arrangement and interior style for the user's room.

[1331] "External Database" refers to a database for obtaining product information from online shopping sites and other external resources.

[1332] "Detailed Information" refers to information about the selected product, including price, images, and purchase links.

[1333] "User interface" refers to the screen and operation method that allows a user to interact with a system.

[1334] The present invention is a system that allows users to upload image data of a room and receives interior design suggestions based on the user's preferences, budget, and real-time emotional state. Specific hardware and software used to implement the invention, as well as methods for processing and calculating data, and specific examples are described below.

[1335] Hardware and Software

[1336] A terminal is a computer or mobile device that provides a user interface and accepts data input from a user. The terminal has image processing software and an emotion engine installed.

[1337] The server is a computer system that processes and analyzes the received data. The following software is used for this analysis:

[1338] Use an image processing library (e.g., OpenCV, Pillow) to adjust image resolution and convert image formats.

[1339] A multimodal AI model (e.g., CNN) analyzes room characteristics (size, furniture arrangement, wall color, amount of light, etc.).

[1340] Recognize the user's emotional state with an emotion recognition library (e.g., OpenFace or other facial expression recognition libraries).

[1341] Data processing and calculation

[1342] When a user launches the system, an interface appears. The user uploads a photo of the room (e.g., a photo of the living room) and a floor plan, inputs the interior style (e.g., natural style) and budget (e.g., 100,000 yen), and provides facial expressions and voice data to detect emotional states.

[1343] The device sends the received data to the server, which then preprocesses the received image data, adjusting the resolution and converting the format, and analyzes the room size, furniture layout, wall color, amount of light, etc.

[1344] Based on this data, the server uses a generative AI model to generate optimal interior style and furniture arrangement suggestions tailored to the user's preferences, budget, and emotional state, with example prompts such as:

[1345] Prompt: "Picture of a living room, natural style, budget of ¥100,000, user smiling (happy)"

[1346] The server searches an external database to select interior products that meet the user's criteria. It selects the most suitable product by comprehensively considering factors such as price, design, and user reviews. It also obtains detailed information about the selected product (price, images, purchase links, etc.).

[1347] Finally, the server sends the generated recommendations to the terminal, which displays them on the user interface. The user can then check the recommendations, view detailed information and purchase links, and purchase the products they are interested in directly from the online shopping site.

[1348] For example, suppose a user uploads a photo of their living room, specifies a preference for a natural style, and sets a budget of 100,000 yen. Furthermore, suppose the emotion engine recognizes the user's smile and detects a happy emotional state. The server analyzes the characteristics of the living room and suggests natural-style furniture and accessories (e.g., a wooden sofa or a beige rug). These suggestions are chosen to create a bright and cheerful atmosphere. Detailed information and purchase links for the selected products are also included, allowing the user to easily purchase them.

[1349] In this way, by using the system of the present invention, users can receive interior design suggestions that take into consideration their own emotional state, and easily find and purchase their ideal interior design items.

[1350] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1351] Step 1: System startup and data entry

[1352] When the user boots the system, the interface is displayed.

[1353] Users upload photos of their rooms (e.g., photos of their living rooms) and floor plans.

[1354] The user inputs the interior style (e.g., natural style) and budget (e.g., 100,000 yen) through the interface.

[1355] The user provides facial expression and voice data to detect the emotional state, which then provides the emotional data.

[1356] Step 2: Receiving data and sending it to the server

[1357] The terminal receives input data from the user (room image data, preferences, budget, and emotional data).

[1358] The terminal sends the received data to the server. Input is the user's input data, which the terminal sends to the server.

[1359] Step 3: Preprocessing the image data

[1360] The server preprocesses the image data it receives, specifically adjusting the image resolution and converting the format.

[1361] The server uses the OpenCV and Pillow libraries for this process.

[1362] For example, downscaling a high-resolution image and converting it to an appropriate image format (JPEG or PNG). The input is the original image data and the output is the preprocessed image data.

[1363] Step 4: Analyzing the image data

[1364] The server uses the pre-processed image data to analyze the characteristics of the room.

[1365] The server uses a multimodal AI model (e.g., CNN) to determine the size of the room, furniture arrangement, wall color, amount of light, etc.

[1366] The analysis results are stored in an internal database. The input is preprocessed image data, and the output is the analysis results.

[1367] Step 5: Recognizing your emotional state

[1368] The device uses an emotion engine to recognize the user's emotional state, using data such as facial expressions, voice tone, and heart rate.

[1369] The device uses OpenFace and other facial recognition libraries for emotion recognition.

[1370] The emotional state is recognized and sent to the server. The input is the user's biometric data and the output is the emotional state data.

[1371] Step 6: Generate proposals

[1372] The server generates appropriate interior style and furniture arrangement suggestions taking into account the room's characteristics, the user's emotional state, preferences, and budget.

[1373] The server uses a generative AI model to create suggestions based on the input (prompt).

[1374] Prompt: "Picture of a living room, natural style, budget of ¥100,000, user smiling (happy)"

[1375] A proposal is generated. The inputs are the room characteristic data, the emotional state data, the user's preferences and budget, and the output is the generated proposal.

[1376] Step 7: Product Selection

[1377] The server searches an external database and selects interior products that meet the user's requirements.

[1378] The server selects the most suitable product by comprehensively considering price, design, user reviews, etc. Detailed information about the selected product (price, image, purchase link, etc.) is obtained.

[1379] The input is the user's conditions and proposal details, and the output is the selected product information.

[1380] Step 8: View the results

[1381] The server sends the generated proposal to the terminal.

[1382] The terminal displays the received proposal on a user interface.

[1383] Users can review the suggestions displayed, view detailed information and purchase links, and select the appropriate product.

[1384] The input is the proposal content and product information, and the output is the proposal and product information displayed on the user interface.

[1385] (Application example 2)

[1386] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1387] Conventional interior design suggestion systems can select interior products based on the user's preferences and budget, but they do not consider the user's emotional state when making suggestions. This makes it difficult to provide the optimal interior design that matches the user's psychological state. Furthermore, there is a lack of a way to easily access detailed information and purchase links on the spot.

[1388] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1389] In this invention, the server includes means for a user to upload image data of a room, means for receiving the uploaded image data and performing image analysis, means for selecting products that match the user's preferences, budget, and emotional state based on the image analysis results, means for generating proposals for the selected products and providing them to the user, means for analyzing the user's facial expression images to detect the user's emotional state, means including an emotion engine with the function of analyzing the emotional state in real time, and means for recommending interior styles according to the emotional state. This makes it possible to propose an optimal interior design that matches the user's real-time emotional state, and further allows the user to access detailed information and a purchase link on the spot.

[1390] "User" refers to an individual or organization that uses the system to upload image data of a room and receive interior design suggestions.

[1391] "Image data" refers to digital image files such as room photos and floor plans uploaded by users.

[1392] "Upload" refers to the act of a user sending image data from their device to a server.

[1393] "Image analysis" refers to the process by which the server preprocesses the image data it receives and analyzes the characteristics of the room.

[1394] "Preferences" refer to personal preferences regarding the interior style and design desired by the user.

[1395] "Budget" refers to the upper limit of costs set by the user when selecting interior products.

[1396] "Emotional state" refers to the psychological state of the user, which is analyzed using the emotion engine.

[1397] An "emotion engine" refers to an algorithm or software that analyzes a user's facial expression images and biometric data to detect their emotional state.

[1398] "Server" refers to a computer system that processes image data and other information received from users and generates and provides interior design proposals.

[1399] "Suggestions" refers to the interior design and product recommendations generated by the system.

[1400] "External databases" refer to online shopping sites and other information sources on the Internet that provide reference information for product selection.

[1401] "Product selection" refers to the process by which the system selects the most suitable interior products based on the user's preferences, budget, and emotional state.

[1402] "Real-time" refers to the process by which a system processes user input or detected information immediately and generates and delivers results.

[1403] This invention provides a system that allows a user to upload image data of a room and proposes interior design ideas based on the user's preferences, budget, and emotional state. The specific system design and operation are described below.

[1404] server

[1405] The server is a computer system that processes image data and other information received from users and generates and provides interior design proposals. The server includes the following means:

[1406] 1. A function to receive image data and perform preprocessing (resolution adjustment, format conversion, etc.).

[1407] 2. Room characteristic analysis function using multimodal AI (analyzing room size, furniture arrangement, wall color, amount of light, etc.).

[1408] 3. A function that uses an emotion engine to analyze the user's emotional state (analyzing facial expressions, voice, heart rate, etc. to detect joy, surprise, sadness, anger, etc.).

[1409] 4. A feature that suggests the best interior style and furniture arrangement based on the user's preferences, budget, and emotional state.

[1410] 5. The ability to search external databases (e.g., online shopping sites) and select appropriate interior products.

[1411] Terminal

[1412] A terminal is a device that allows users to access the system and input information. This terminal can be a smartphone, tablet, or PC. The specific operation steps are as follows:

[1413] 1. The user takes a photo of the room using the device and uploads the image data.

[1414] 2. The user inputs their preferred interior style and budget through the interface.

[1415] 3. The device captures the user's facial expressions and sends the data to the emotion engine in real time.

[1416] Processing Flow

[1417] Hardware and software configuration

[1418] Hardware: Smartphone (camera function, network connection), server (high performance computer).

[1419] Software: Python, PIL (image preprocessing), Emotion Engine (sentiment analysis), Interior Style Recommender (interior suggestions).

[1420] The server first preprocesses the image data of the room received from the user (adjusting the resolution, converting the format, etc.). It then uses multimodal AI to analyze the room's size, furniture arrangement, wall color, amount of light, etc. The analysis results are stored in a database on the server. At the same time, the device uses an emotion engine to analyze the user's emotional state, and this information is also sent to the server.

[1421] The server takes into account the room's characteristics, the user's emotional state, preferences, and budget to generate suggestions for appropriate interior styles and furniture arrangements. These suggestions also include product information (price, images, purchase links, etc.) retrieved from an external database. Once the suggestions are generated, the server sends them to the device and provides them to the user.

[1422] Examples and prompts

[1423] For example, if a user uploads a photo of their living room with a natural style and a budget of 100,000 yen, the system works as follows: The emotion engine recognizes the user's smile and detects a happy emotional state. The server analyzes the characteristics of the living room and suggests natural-style furniture and accessories (e.g., a wooden sofa, a beige rug). The suggestions are selected to create a bright and cheerful atmosphere.

[1424] Prompt Sentence Examples

[1425] "Upload a picture of your room: room.jpg"

[1426] "Please enter your preferred interior style: Natural"

[1427] "Please enter your budget: 100,000 yen"

[1428] "Upload a photo of your face: user.jpg"

[1429] This allows users to easily find and purchase their ideal interior items through specific interior suggestions.

[1430] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1431] Step 1:

[1432] The user uses the terminal to take a picture of the room and upload it.

[1433] Input: A photo of the room.

[1434] Output: The uploaded image data.

[1435] Specific operation: The user takes a photo of the room using the camera function of their smartphone, and then presses the upload button in the application to send the image data to the server.

[1436] Step 2:

[1437] The server pre-processes the image data received from the user.

[1438] Input: Uploaded image data.

[1439] Output: Preprocessed image data.

[1440] What it does: It adjusts the image resolution on the server and converts it to the required format (e.g. resize to 256x256 pixels). It uses Python and the PIL library for preprocessing.

[1441] Step 3:

[1442] The server analyzes the pre-processed image data to understand the characteristics of the room.

[1443] Input: Preprocessed image data.

[1444] Output: Analysis results such as room size, furniture placement, wall color, and amount of light.

[1445] Specific operation: A multimodal AI system on the server analyzes image data and extracts room characteristics. The extracted data is stored in an internal database.

[1446] Step 4:

[1447] The user inputs their preferred interior style and budget through the interface.

[1448] Input: The user's interior style preferences and budget.

[1449] Output: Interior style preferences, budget data.

[1450] What it does: Enter style and budget through a user interface using drop-down menus and text input forms.

[1451] Step 5:

[1452] The terminal uses an emotion engine to detect the user's emotional state.

[1453] Input: An image of the user's facial expression.

[1454] Output: Detected emotional state data.

[1455] Specific operation: The smartphone camera captures the user's facial expression and sends it to the server in real time. The emotion engine analyzes the facial expression data and detects the user's emotional state (e.g., joy, sadness, surprise, etc.).

[1456] Step 6:

[1457] The server suggests interior styles and furniture arrangements based on the room's characteristics, the user's preferences, budget, and emotional state.

[1458] Input: Room characteristic data, interior style preference, budget data, emotional state data.

[1459] Output: Interior design proposals (style, furniture arrangement, specific products).

[1460] What it does: The algorithm in the server generates the optimal interior design based on all input data, and then retrieves and finalizes the proposal from internal and external databases.

[1461] Step 7:

[1462] The server sends the selected product details and a link to purchase the product to the device.

[1463] Input: Interior design proposal.

[1464] Output: The suggestions that will be displayed on the user's device.

[1465] Specific operation: The server sends the generated interior design proposal to the device and displays detailed information (price, image, purchase link) on the interface.

[1466] Step 8:

[1467] The user reviews the offer and clicks the purchase link to purchase the product.

[1468] Input: Proposed interior product.

[1469] Output: Transition to online shopping site and purchase process.

[1470] Specific actions: The user checks the suggestions displayed on the device and clicks on the purchase link for the product they are interested in. They then complete the purchase at the linked online shopping site.

[1471] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1472] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1473] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1474] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1475] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1476] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1477] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1478] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1479] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1480] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1481] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1482] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1483] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1484] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1485] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1486] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1487] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1488] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1489] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1490] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1491] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1492] The following is further disclosed regarding the above embodiment.

[1493] (Claim 1)

[1494] A means for a user to upload image data of a room;

[1495] means for receiving the uploaded image data and performing image analysis;

[1496] A means for selecting a product that matches the user's preferences and budget based on the image analysis results;

[1497] means for generating and providing a recommendation for the selected product to the user;

[1498] A system including:

[1499] (Claim 2)

[1500] 2. The system according to claim 1, wherein the image analysis means has a function of grasping the characteristics of a room and detecting the arrangement of furniture, the color of the walls, and the amount of light.

[1501] (Claim 3)

[1502] 2. The system according to claim 1, wherein the product selection means has a function of searching an external database and selecting an appropriate product based on the user's preferences and budget.

[1503] "Example 1"

[1504] (Claim 1)

[1505] A means for a user to upload image data of the space;

[1506] means for receiving the uploaded image data and performing data pre-processing;

[1507] means for analyzing the preprocessed image data and extracting spatial features;

[1508] A means for searching and selecting products that match the user's preferences and budget based on the analysis results using the generative AI model;

[1509] means for generating and providing a recommendation for the selected product to the user;

[1510] A system including:

[1511] (Claim 2)

[1512] 2. The system according to claim 1, wherein the feature extraction means has a function of detecting the size of the space, the arrangement of furniture, the color of the walls, and the amount of light.

[1513] (Claim 3)

[1514] 2. The system according to claim 1, wherein the product selection means has a function of selecting an appropriate product based on the user's preferences and budget using an external database.

[1515] "Application Example 1"

[1516] (Claim 1)

[1517] A means for a user to upload image data of a room;

[1518] means for receiving the uploaded image data and performing image analysis;

[1519] A means for selecting a product that matches the user's preferences and budget based on the image analysis results;

[1520] means for generating and providing a recommendation for the selected product to the user;

[1521] A means of capturing and analyzing in-store image data;

[1522] A method to suggest in-store display items based on the results of image analysis,

[1523] A system including:

[1524] (Claim 2)

[1525] 2. The system according to claim 1, wherein the image analysis means has the function of grasping the characteristics of a room and the characteristics of a store, and detecting the arrangement of furniture, the color of walls, the amount of light, and the arrangement of display items.

[1526] (Claim 3)

[1527] 2. The system of claim 1, wherein the product selection means has the function of searching an external database and selecting appropriate products and display items based on the user's preferences and budget and the characteristics of the store.

[1528] "Example 2: Combining Emotion Engines"

[1529] (Claim 1)

[1530] A means for a user to upload image data of a room;

[1531] means for receiving uploaded image data and adjusting the resolution and converting the format of the image;

[1532] After preprocessing the image, a means of analyzing the size of the room, furniture arrangement, wall color, amount of light, etc.

[1533] means for analyzing biometric data of a user to recognize the emotional state of the user;

[1534] means for generating suitable interior style and furniture arrangement suggestions based on the analysis results and the user's preferences, budget, and emotional state;

[1535] A means for searching an external database and selecting products that meet the user's requirements;

[1536] A means to obtain detailed information and purchase links for selected products;

[1537] means for displaying the generated proposal content on a user interface;

[1538] A system including:

[1539] (Claim 2)

[1540] The system of claim 1, wherein the image analysis means has the function of using a multimodal AI model to understand the characteristics of a room from image data of the room uploaded by the user, and to detect furniture arrangement, wall color, and amount of light.

[1541] (Claim 3)

[1542] 2. The system according to claim 1, wherein the emotion recognition means has a function of detecting the emotional state of the user by analyzing biometric data such as facial expressions, voice tones, and heart rate.

[1543] "Application example 2 when combining emotion engines"

[1544] (Claim 1)

[1545] A means for a user to upload image data of a room;

[1546] means for receiving the uploaded image data and performing image analysis;

[1547] A means for selecting a product that matches the user's preferences, budget, and emotional state based on the image analysis results;

[1548] means for generating and providing a recommendation for the selected product to the user;

[1549] means for analyzing a facial expression image of a user to detect an emotional state;

[1550] means including an emotion engine capable of analyzing an emotional state in real time;

[1551] A means for recommending interior styles according to emotional states;

[1552] A system including:

[1553] (Claim 2)

[1554] 2. The system according to claim 1, wherein the image analysis means has a function of grasping the characteristics of a room and detecting the arrangement of furniture, the color of the walls, and the amount of light.

[1555] (Claim 3)

[1556] 2. The system according to claim 1, wherein the product selection means has a function of searching an external database and selecting an appropriate product based on the user's preferences, budget, and emotional state. [Explanation of symbols]

[1557] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. A means for a user to upload image data of a room; means for receiving the uploaded image data and performing image analysis; A means for selecting a product that matches the user's preferences and budget based on the image analysis results; means for generating and providing a recommendation for the selected product to the user; A system including:

2. 2. The system according to claim 1, wherein the image analysis means has a function of grasping the characteristics of a room and detecting the layout of furniture, the color of walls, and the amount of light.

3. 2. The system according to claim 1, wherein said product selection means has a function of searching an external database and selecting an appropriate product based on the user's preferences and budget.

Images