System

Abstract

A system is provided.SOLUTION: A system comprising: means for uploading image data of a room captured by a user terminal to a server; means for analyzing the image data and extracting information on furniture and layout in the room by the server; means for generating a rearrangement proposal based on the extracted information and creating a list of recommended products; means for linking to a sales platform that provides the recommended products; and means for transmitting the proposed rearrangement proposal and the list of recommended products to the user terminal.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] When it comes to interior coordination, many users lack confidence in their room layout and furniture selection, leaving them unsure of how to redecorate. As a result, they are unable to receive appropriate redecorating ideas or furniture suggestions, making it difficult to create the ideal room. Furthermore, promoting new products is a challenge for companies selling furniture and miscellaneous goods. The present invention aims to solve these problems by providing a system that allows users to redecorate easily and enjoyably, leading to increased sales for companies. [Means for solving the problem]

[0005] The present invention solves the above problem by providing a system that includes a means for uploading image data of a room taken by a user terminal to a server, a means for the server to analyze the image data and extract information about the furniture and layout in the room, a means for generating redecorating suggestions and creating a list of recommended products based on the extracted information, a means for linking to a sales platform that provides the recommended products, and a means for transmitting the proposed redecorating ideas and the list of recommended products to the user terminal.

[0006] A "user terminal" is an electronic device used by a user, such as a smartphone, tablet, or PC.

[0007] A "server" is a central system for processing, storing, and providing data, and is a computer that provides services in response to requests from user terminals.

[0008] "Image data" refers to photographs taken by users and their digital information.

[0009] "Upload" refers to the act of sending data from a user terminal to a server.

[0010] "Image analysis" is a technique for processing image data, recognizing its contents, and extracting information.

[0011] "Means of extraction" refers to the process of extracting information about the furniture and layout in the room as a result of image analysis.

[0012] "Redecoration suggestions" refer to ideas that offer new layouts or improvements to rooms.

[0013] "Recommended Products" is a list of furniture and miscellaneous items recommended to the user based on the proposed redecoration.

[0014] "Sales Platform" means an online marketplace for purchasing goods.

[0015] "Link" means a URL or hyperlink that allows a user to access a specified web page.

[0016] "Means of sending" refers to the process of sending data from the server to the user terminal. [Brief explanation of the drawings]

[0017] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12]FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0018] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0019] First, the terms used in the following description will be explained.

[0020] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0021] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0022] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0023] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0024] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0025] [First embodiment]

[0026] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0027] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0028] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0029] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0030] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0031] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0032] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0033] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0034] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0035] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0036] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0037] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0038] The present invention provides a system that allows users to easily rearrange the interior of their own rooms. This system is implemented based on the following specific operational flow, which operates via a user terminal, a server, and the Internet.

[0039] First, the user takes a photo of their own room. Using the user's device (e.g., a smartphone), the user positions the camera so that the entire room they want to redecorate is captured, and takes a photo. After taking the photo, the user uses the app's upload function to send the photo data to the server.

[0040] The server then receives the image data sent by the user and analyzes it to extract important information such as the room's furniture, layout, wall and floor locations, and even the location of windows and doors. This analysis is performed using advanced AI technology.

[0041] Based on the analyzed information, the server generates multiple redecorating ideas, including new furniture arrangements and suggestions for adding new furniture and accessories. For example, the server might suggest moving the sofa to the right wall and placing a modern coffee table in the center of the living room.

[0042] The server then generates a list of recommended products suitable for the user based on the home improvement ideas. This list includes products selected from online marketplaces such as Yahoo! Shopping. For example, the server generates a link to a product on a sales platform that corresponds to the suggested coffee table.

[0043] The server then sends these redecorating ideas and a list of recommended products to the user's device, which receives the data and displays it on the screen in a user-friendly format, including a diagram of the new layout, images of the recommended furniture, and a link to purchase the items.

[0044] Users can select the product they want to purchase from a list of recommended products displayed on their device. When the user clicks on the purchase link, they are redirected to the product page and can complete the purchase process. For example, the user selects a coffee table with a modern design and clicks on the link to complete the purchase process on the Yahoo! Shopping page.

[0045] In this way, the system of the present invention allows users to easily rearrange their interiors and helps them create their ideal rooms. It also provides companies with an effective means of promoting new products.

[0046] The processing flow will be explained below.

[0047] Step 1:

[0048] The user starts a photography app on their smartphone and takes a photo of the room they want to change the interior of.

[0049] Step 2:

[0050] The user selects a photo and clicks the upload button in the app. The user's device sends the captured image data to the server.

[0051] Step 3:

[0052] The server receives the image data. The server receives and stores the image data of the room sent from the user terminal.

[0053] Step 4:

[0054] The server performs image analysis and passes the received image data to the AI ​​platform to extract information about furniture, layout, walls, floors, windows, doors, etc.

[0055] Step 5:

[0056] The server generates redecorating ideas based on the extracted information. The server creates multiple redecorating plans and suggests new layouts and additional furniture that are suitable for the user's room.

[0057] Step 6:

[0058] The server creates a list of recommended products. Based on the generated redecorating ideas, the server searches online marketplaces for recommended products and creates a list.

[0059] Step 7:

[0060] The server sends a list of redecorating ideas and recommended products to the user's device. The server then sends this data to the user's device.

[0061] Step 8:

[0062] The user's device receives the data and displays the information on the display screen, allowing the user to see the new layout proposal and recommended product list on the device screen.

[0063] Step 9:

[0064] The user selects a recommended product and completes the purchase procedure. The user selects the product they wish to purchase from the list of recommended products, clicks on the link to the sales platform, and is taken to the purchase page to complete the purchase procedure.

[0065] Example 1

[0066] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0067] In the past, when users rearranged their room interiors, not only did it take a lot of time and effort to arrange furniture and select new items, but it was also difficult to imagine a specific layout plan. Furthermore, purchasing new furniture required searching blindly, which was not an efficient process. As a result, there was insufficient support for creating the ideal room.

[0068] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0069] In this invention, the server includes means for uploading image data of a room taken by a user terminal to the server, means for the server to analyze the image data and use advanced AI technology to extract the furniture, layout, wall and floor positions, and window and door positions within the room, means for generating redecorating proposals including new furniture arrangements and suggestions for adding new products based on the extracted information and creating a list of recommended products from an online marketplace, means for linking to a sales platform that provides the recommended products, and means for transmitting the proposed redecorating proposals and the list of recommended products to the user terminal, thereby enabling users to easily rearrange their interiors and create their ideal room.

[0070] A "user terminal" is an electronic device used by a user to take photos of a room and upload data, such as a smartphone or tablet.

[0071] A "server" is a computer system that functions as a central control device, receives and analyzes image data sent from user terminals, and generates and sends lists of redecorating suggestions and recommended products.

[0072] "Image data" is photographic data of the room taken by the user terminal, and is a digital image file that is uploaded to the server.

[0073] "Advanced AI technology" refers to technology that uses artificial intelligence, particularly algorithms and models for image analysis, data extraction, and layout proposals.

[0074] "Furniture" and "layout" refer to the items arranged in a room and their arrangement. Specifically, this includes interior items such as sofas, tables, shelves, and their arrangement.

[0075] "Redecoration proposal" refers to a proposal for a new interior layout or product addition to a room, generated by the server. For example, it includes a proposal for rearranging furniture or introducing new items.

[0076] A "recommended product list" is a list of products selected based on the redecorating suggestions and suggested to the user, including products selected from online marketplaces and their purchase links.

[0077] "Online marketplace" refers to a platform for selling goods over the Internet, including, for example, general e-commerce sites.

[0078] "Means of linking to a sales platform" refers to a method of providing a link to the product purchase page of an online marketplace that sells recommended products.

[0079] "Transmission means" refers to a method for sending data from one party to another, and in particular refers to a method for sending a list of redecorating suggestions or recommended products from a server to a user terminal.

[0080] The present invention provides a system that allows users to easily redecorate the interior of their own rooms. This system operates via a user terminal, a server, and the Internet. Specific embodiments are described below.

[0081] First, the user uses a user device (e.g., a smartphone) to take a photo of their room. The user positions the camera so that the entire room they want to redecorate is captured, and takes a photo. After taking the photo, the user uses the upload function in the app on their device to send the photo data to the server.

[0082] The server then receives the photo data sent from the user's device. The server incorporates advanced AI technology, which is used to analyze the image data. The analysis extracts important information, such as the room's furniture, layout, wall and floor positions, and even the location of windows and doors. This allows the current state of the room to be accurately determined.

[0083] The server then generates multiple redecorating ideas based on the extracted information. This process uses a generative AI model. Ideas include new furniture arrangements and suggestions for adding new furniture and accessories. For example, the server might automatically generate a layout that moves the sofa along the right wall and places a modern coffee table in the center of the living room.

[0084] Furthermore, the server generates a list of recommended products suitable for the user based on these home improvement ideas. This list is composed of products selected from online marketplaces (e.g., general e-commerce sites). The server generates links to furniture and miscellaneous items suitable for each suggestion to complete the list of recommended products.

[0085] The server then sends the list of redecorating ideas and recommended products to the user's device, which receives the data and displays it on a screen in an easy-to-understand format. The displayed content includes a diagram of the new layout plan, images of the recommended furniture, and a link to purchase the items.

[0086] The user can select the product they want to purchase from the list of recommended products displayed on the device. When the user clicks on the purchase link, they are redirected to the product page and can complete the purchase. For example, the user selects a coffee table with a modern design and clicks on the link to complete the purchase on the e-commerce site page.

[0087] An example prompt is:

[0088] "Users take photos of their rooms with their smartphones and upload them to the server. The server uses AI technology to analyze the images and automatically recognize furniture placement and layout. The server then generates multiple redecorating ideas and creates a list with links to suitable products. This list is then sent to the user's device, where the user can view and purchase the products."

[0089] "A user takes a picture of their living room with their smartphone and uploads it to the server through the app. The server analyzes the image and recognizes the furniture layout. The server then generates an idea to move the sofa along the right wall and place a modern coffee table in the center. The server creates a list of coffee tables, including links to popular e-commerce sites, and sends it to the user. The user reviews the list and purchases the coffee table."

[0090] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0091] Step 1:

[0092] The user takes a photo of the room.

[0093] Input: The user's room status

[0094] How it works: A user uses their smartphone to take a photo of the room they want to redecorate.

[0095] Output: Image data of the photographed room

[0096] Step 2:

[0097] The user uploads image data from the terminal to the server.

[0098] Input: Image data of the room

[0099] Operation: The user uses the application on the device to tap the upload button to send the captured image data to the server.

[0100] Output: Image data uploaded to the server

[0101] Step 3:

[0102] The server receives and analyzes the image data.

[0103] Input: Image data uploaded by the user

[0104] How it works: The server uses advanced AI technology to analyze the received image data and extract information such as the furniture in the room, layout, the position of walls and floors, and the position of windows and doors.

[0105] Output: Information about furniture and layout in the room

[0106] Step 4:

[0107] The server generates redecorating ideas based on the extracted information.

[0108] Input: Information about the furniture and layout in the room

[0109] How it works: The server uses a generative AI model to generate multiple redecorating ideas, including new furniture arrangements and suggestions for adding new products.

[0110] Output: Redecorating ideas

[0111] Step 5:

[0112] The server creates a list of recommended products based on the redecorating ideas.

[0113] Enter: Redecorating Ideas

[0114] How it works: The server searches online marketplaces for suitable product recommendations and creates a list by generating product links for each suggestion.

[0115] Output: A list of recommended products

[0116] Step 6:

[0117] The server sends a list of redecorating ideas and recommended products to the user's terminal.

[0118] Input: A list of home improvement ideas and recommended products

[0119] Operation: The server sends the generated redecorating ideas and a list of recommended products to the user's device.

[0120] Output: A list of home improvement ideas and recommended products sent to the user's device

[0121] Step 7:

[0122] The user terminal displays the received data.

[0123] Input: A list of redecorating ideas and recommended products sent from the server

[0124] Operation: The user's device displays the received data on the screen and provides the user with new layout proposals, images of recommended products, and purchase links.

[0125] Output: A list of redecorating ideas and recommended products

[0126] Step 8:

[0127] The user selects and purchases a product from a list of recommended products.

[0128] Input: List of displayed recommended products

[0129] How it works: A user selects the product they want to purchase from a list of recommended products and clicks a link to go to the product page on the online marketplace and complete the purchase.

[0130] Output: Purchased items and completed checkout

[0131] (Application example 1)

[0132] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0133] Conventional interior redecorating systems lacked a means for users to virtually visualize the new layout, making it difficult to visualize how the room would change. They also lacked an environment in which users could smoothly purchase recommended products. Furthermore, they lacked a means to compare the real room with the virtual layout in real time, leaving them with insufficient functionality to support users in making redecorating decisions.

[0134] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0135] In this invention, the server includes means for uploading image data of a room taken by a user terminal to the server, means for analyzing the image data by the server and extracting information about the furniture and layout in the room, means for generating redecorating proposals and creating a list of recommended products based on the extracted information, means for linking to a sales platform that provides the recommended products, means for transmitting the proposed redecorating proposals and the list of recommended products to the user terminal, and means for the user to confirm and purchase the proposed redecorating proposals and recommended products displayed on the user terminal in a virtual store. This makes it easier for the user to virtually visualize a new layout and to easily purchase recommended products, thereby making it possible to smoothly proceed with redecorating decisions.

[0136] A "user terminal" is a computer device, smartphone, tablet, or other portable electronic device used by a user.

[0137] "Image data" refers to photos of the room taken by the user terminal or image files.

[0138] A "server" is a computer system that processes requests and provides data from multiple client devices within a network.

[0139] "Analysis" refers to the process performed by the server to extract and analyze necessary information from image data.

[0140] "Furniture" refers to interior items such as tables, chairs, and sofas that are placed in a room.

[0141] "Layout" refers to the arrangement and design of furniture and equipment within a room.

[0142] "Suggestion" refers to a recommendation to the user of a new layout and arrangement for redecorating.

[0143] "Recommended products" are interior products that are candidates for purchase and are recommended by the server based on the new layout plan.

[0144] "Sales Platform" refers to a website or service that allows you to sell products online.

[0145] "Link" means a hypertext link that navigates from a user terminal to the sales platform.

[0146] A "virtual store" refers to an online shopping space that uses virtual reality and augmented reality.

[0147] "Virtual visualization" is a process that allows users to visually see in real time how a new layout proposal would look in a real room in a virtual environment.

[0148] The system of the present invention is designed to allow users to easily rearrange their rooms, virtually visualize new layouts, and purchase recommended products.

[0149] Hardware and software used

[0150] User devices: Smartphones (e.g., iPhones, Android devices), tablets

[0151] Server: A powerful computer system (e.g., an EC2 instance from Amazon Web Services)

[0152] Image analysis: TensorFlow, Keras

[0153] Database: MySQL

[0154] Server framework: Django (Python)

[0155] Frontend: React Native

[0156] API: Online marketplace API (e.g. Yahoo! Shopping API)

[0157] System processing flow

[0158] Uploading image data from a user device

[0159] Users can take photos of their rooms using their smartphones or tablets and upload the image data to the server using a dedicated application. By taking a photo of the entire room using the camera on the user's device, all the necessary information is captured.

[0160] Image data analysis by the server

[0161] The server analyzes the received image data using TensorFlow and Keras. Image analysis extracts information about the layout of furniture in the room and the positions of windows, doors, etc. This analysis makes it possible to grasp the components of the room in detail.

[0162] Generate redecorating suggestions

[0163] Based on the analyzed information, the server generates multiple redecorating ideas, including new arrangements for existing furniture and suggestions for adding new interior items.

[0164] Creating and providing a list of recommended products

[0165] Based on the generated redecorating ideas, the server uses an online marketplace API to create a recommended product list, which includes links to products that correspond to the suggested furniture and interior items.

[0166] Check and purchase in the virtual store

[0167] The proposed redecorating plan and the list of recommended products are sent to the user's device, where the user can use the application installed on the device to virtually visualize the new layout and purchase the recommended products.

[0168] Specific examples

[0169] Users take photos of their living room with their smartphones and upload them to the application. The server analyzes the images using TensorFlow and Keras to identify the positions of sofas and tables, as well as the layout of walls and windows. Based on this information, the server generates modern interior design proposals and lists recommended products (new sofas, tables, and accessories) retrieved from online marketplaces. Users can then virtually visualize the new layout through the application and purchase the recommended products on the spot.

[0170] Prompt Sentence Examples

[0171] "Room Layout Recommendations"

[0172] User Scenario:

[0173] The user uploads a photo of their living room to the app. The AI analyzes the photo to suggest new furniture arrangements and recommends matching products available for purchase.

[0174] Prompt Input:

[0175] Room Photo: [Upload Photo]

[0176] Current Furniture Layout: [Provide AI analysis]

[0177] Suggested Furniture Rearrangement: [Generate layout suggestions]

[0178] Product Recommendations: [Fetch from Online Marketplace API]

[0179] Desired Output:

[0180] 1. New furniture layout suggestions based on the uploaded photo.

[0181] 2. List of recommended products matching the new layout.

[0182] 3. Direct links to purchase recommended products.

[0183] Action:

[0184] Process the uploaded photo to analyze the current furniture layout and generate new arrangement suggestions. Fetch and display recommended products with purchase links."

[0185] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0186] Step 1:

[0187] Taking and uploading images

[0188] The user takes a photo of the room using a smartphone or tablet. The user opens the application and selects an image of the room or takes a new one and uploads it. The input is the image data of the room, and the output is the completion of uploading the image data to the server.

[0189] Step 2:

[0190] Receiving image data

[0191] The server receives image data sent from the user terminal. The input is the image data uploaded by the user, and the output is the image data saved on the server. The server saves the image data in a specified directory.

[0192] Step 3:

[0193] Image data analysis

[0194] The server analyzes the stored image data using TensorFlow and Keras. The input is the stored image data, and the output is the analysis results. Through the analysis, information such as the position of furniture in the room, walls, windows, and doors is extracted. Specifically, the image is input into a model, and data processing is performed to identify the position of furniture and layout information.

[0195] Step 4:

[0196] Generate redecorating suggestions

[0197] The server generates multiple redecorating plans based on the analysis results. The input is the analysis results, and the output is multiple redecorating plans. The server uses a generative AI model to suggest new furniture placements and interior items to add. Each plan includes the specific placement method and the reasons for it.

[0198] Step 5:

[0199] Creating a list of recommended products

[0200] The server uses an online marketplace API to create a list of recommended products based on the redecorating plan. The input is the redecorating plan, and the output is a list of recommended products. The server calls the API to obtain product information (price, links, images, etc.) corresponding to the suggested furniture.

[0201] Step 6:

[0202] Sending data

[0203] The server sends the generated list of redecorating ideas and recommended products to the user's device. The input is the list of redecorating ideas and recommended products, and the output is the completion of sending the data to the user's device. The server then sends data to the user's device to display this information.

[0204] Step 7:

[0205] Virtual store visualization

[0206] The user uses an application installed on their device to virtually visualize the new layout. The input is a list of redecorating ideas and recommended products sent from the server, and the output is the user's reaction to the visualized layout and their selection. The user can view the proposed layout using 3D models and AR functions to visualize the changes concretely.

[0207] Step 8:

[0208] Purchase recommended products

[0209] The user clicks on a link in the application to purchase the recommended product in the virtual store. The input is the link to the recommended product, and the output is the completion of the product purchase procedure. The user opens the link on their device and proceeds with the purchase procedure on the virtual store's details page.

[0210] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0211] The present invention is a system that allows users to easily rearrange the interior of their own rooms. This system is realized based on the following specific operational flow, which operates via a user terminal, a server, an emotion engine, and the Internet.

[0212] First, the user takes a photo of their own room. Using a user device (such as a smartphone), the user positions the camera so that the entire room in which they wish to change the interior is captured, and takes a photo. After taking the photo, the user uses the upload function in the app to send the photo data to the server.

[0213] The server then receives the image data sent by the user and analyzes it to extract important information such as the room's furniture, layout, wall and floor locations, and even the location of windows and doors. This analysis is performed using advanced AI technology.

[0214] Based on the analyzed information, the server generates multiple redecorating ideas, including new furniture arrangements and suggestions for adding new furniture and accessories. For example, the server might suggest moving the sofa to the right wall and placing a modern coffee table in the center of the living room.

[0215] Next, the emotion engine analyzes the user's facial expressions and voice to recognize their emotions. The emotion engine analyzes how the user feels about the proposed redecorating ideas and adjusts the ideas accordingly. For example, the server may make suggestions specific to the redecorating ideas the user is interested in, providing layout proposals that match the user's preferences.

[0216] After using the emotion engine to select the best idea that matches the user's emotions, the server creates a list of recommended products. This list includes products selected from online marketplaces such as Yahoo! Shopping. For example, the server generates a link to a product on a sales platform that corresponds to the suggested coffee table. This list is customized based on the user's emotions, so it recommends products that are more suitable for the user.

[0217] The server then sends these redecorating ideas and a list of recommended products to the user's device, which receives the data and displays it on the screen in a user-friendly format, including a diagram of the new layout, images of the recommended furniture, and a link to purchase the items.

[0218] Users can select the product they want to purchase from a list of recommended products displayed on their device. When the user clicks on the purchase link, they are redirected to the product page and can complete the purchase process. For example, the user selects a coffee table with a modern design and clicks on the link to complete the purchase process on the Yahoo! Shopping page.

[0219] In this way, the system of the present invention allows users to easily rearrange their interiors and support the creation of their ideal room. Furthermore, by taking the user's emotions into consideration, more personalized suggestions become possible. Furthermore, it can provide an effective means for companies to promote new products.

[0220] The processing flow will be explained below.

[0221] Step 1:

[0222] The user launches the camera app on their smartphone and takes a photo of the room they want to redecorate.

[0223] Step 2:

[0224] The user sends the photos they have taken to the server using the upload function within the app.

[0225] Step 3:

[0226] The server receives and stores the image data sent from the user terminal.

[0227] Step 4:

[0228] The server uses image analysis algorithms to analyze the image data and identify the furniture, layout, walls, floors, windows, doors, etc. within the room.

[0229] Step 5:

[0230] Based on the extracted information, the server generates multiple redecorating ideas, including suggestions for new furniture arrangements and additional furniture.

[0231] Step 6:

[0232] The emotion engine collects the user's facial and voice data and analyzes their emotions. For example, if the user smiles into the smartphone camera, the emotion engine will detect a positive emotion.

[0233] Step 7:

[0234] The emotion engine evaluates the redecorating ideas based on the user's emotions and adjusts the suggestions according to the user's emotions. The emotion engine reevaluates the suggestions and selects ideas that are likely to satisfy the user.

[0235] Step 8:

[0236] The server generates a customized list of recommended products based on the results of the emotion engine, which includes products that match the user's emotions.

[0237] Step 9:

[0238] The server sends a list of redecorating ideas and recommended products to the user's terminal.

[0239] Step 10:

[0240] The user's device receives the data from the server and displays it in a user-friendly format, including diagrams of new layout plans, images of recommended products, and links to purchase them.

[0241] Step 11:

[0242] Users select the product they want to purchase from the recommended product list and click the link to proceed to the sales platform and complete the purchase. For example, a user can purchase a modern coffee table on Yahoo! Shopping.

[0243] In this way, the system of the present invention allows users to easily rearrange their interiors and supports them in making optimal suggestions and purchasing products that suit their own emotions.

[0244] Example 2

[0245] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0246] Conventional interior redecorating systems simply analyze the layout information of a user's room and make suggestions. As a result, the suggested redecorating ideas often do not match the user's preferences, and suggestions do not take the user's emotions into consideration. Furthermore, the convenience of purchasing the recommended products is insufficient.

[0247] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0248] In this invention, the server includes means for uploading image data of a room taken by a user terminal to the server, means for the server to analyze the image data and extract information about the furniture and layout of the room, means for generating redecorating proposals and creating a list of recommended products based on the extracted information, means for linking to a sales platform that provides the recommended products, means for transmitting the proposed redecorating proposals and the list of recommended products to the user terminal, means for analyzing the user's facial expressions and voice and recognizing their emotions, and means for adjusting the redecorating proposals based on the recognized emotions. This enables more personalized redecorating proposals based on the user's emotions, thereby increasing user satisfaction. It also simplifies the process of purchasing recommended products, improving the user experience.

[0249] A "user terminal" is an information processing device that can be operated by a user and that can input and display data.

[0250] A "server" is a central information processing device that is connected to user terminals via a network and performs data recording, management, analysis, and the like.

[0251] "Image data" is digital data containing visual information captured and recorded by a user terminal.

[0252] "Upload" refers to the operation and process of sending image data from a user terminal to a server.

[0253] "Analysis" is the process of automatically processing the information contained in the image data and extracting specific information (e.g., furniture or layout).

[0254] "Furniture" refers to items such as desks, chairs, sofas, and shelves that are placed in a room.

[0255] "Layout" refers to the arrangement and placement of various furniture and decorative items in a room.

[0256] "Redecoration proposals" involve providing specific proposals and ideas for changing the existing interior layout.

[0257] "Recommended products" are commercial items related to the redecorating suggestions and recommended to the user.

[0258] A "sales platform" is an online marketplace or shopping site that sells recommended products.

[0259] "Linking" refers to the process of making the relevant product page on the sales platform accessible from the display screen of the user's terminal.

[0260] "Facial expressions" refer to emotions and reactions expressed through changes in the user's face.

[0261] "Speech" refers to information produced by the user's voice or words.

[0262] "Emotion recognition" means analyzing facial expressions and voice data to determine the user's psychological state and reactions.

[0263] "Adjusting" means optimizing redecorating suggestions and product recommendations based on perceived emotions.

[0264] This invention is a system that allows users to easily redecorate the interior of their own room. This system operates via a user terminal, a server, an emotion engine, and the Internet. The following describes in detail the mode for carrying out the invention.

[0265] User device operation

[0266] Users use their smartphone or other device to take photos of their room so that the entire room is visible. Taking photos from multiple angles and under appropriate lighting improves the accuracy of analysis. The photos are then sent to the server using the upload function of the dedicated app. The data sent is encrypted to protect privacy.

[0267] Server Processing

[0268] The server receives image data sent from the user's device. It decodes the image data and analyzes it using AI technology (e.g., TensorFlow or OpenCV). During the analysis process, the furniture in the room, its layout, the location of walls and floors, and the location of windows and doors are identified. For example, the contours and positions of furniture such as sofas and tables can be identified and mapped.

[0269] Once the analysis is complete, the server generates multiple redecorating ideas using automated design algorithms (e.g., generative design). These ideas include new furniture arrangements and suggestions for adding new furniture and accessories. For example, a layout suggestion might be to move the sofa along the right wall and place a modern coffee table in the center of the living room.

[0270] Emotion engine processing

[0271] The emotion engine analyzes the user's facial expressions and voice to recognize their emotions. It uses facial recognition software (e.g., Microsoft Azure Face API) and voice analysis software (e.g., Google Cloud Speech-to-Text). Based on the analysis results, it evaluates how the user feels about the proposed redecorating ideas. This allows it to make adjustments based on the ideas that the user likes.

[0272] Product recommendations and delivery

[0273] The server creates a list of recommended products based on the results of the emotion engine. This list is generated using APIs of various online marketplaces. The list includes links to purchase suggested new furniture and accessories. For example, a link to a sales page for a modern coffee table is generated.

[0274] The generated list of redecorating ideas and recommended products is sent to the user's device, which receives it and displays it on the screen. The displayed content includes a diagram of the new layout plan, images of the recommended products, and a link to purchase them.

[0275] Product purchase

[0276] The user selects the product they want to purchase from the list of recommended products displayed on their device. Clicking the purchase link takes them to the sales page for that product and allows them to complete the purchase process. For example, the user selects a coffee table with a modern design and clicks the link to complete the purchase process.

[0277] Specific examples

[0278] 1. Examples

[0279] A user wants to redecorate their living room and takes a photo of the room with their smartphone.

[0280] Upload the photo to the server via the app.

[0281] The server analyzes the image data and generates multiple layout proposals (e.g., placing the sofa against the wall and a new coffee table in the center).

[0282] The emotion engine recognizes happy reactions from the user's facial expressions.

[0283] The server selects and proposes the optimal layout plan based on the emotions.

[0284] Provides a list of recommended coffee tables with links to online marketplaces.

[0285] 2. Prompt sentence for generative AI model

[0286] I'd like to redecorate my living room, so I'll upload a photo of the entire room. Please analyze it and suggest new furniture arrangements and new furniture and accessories that go with it. Also, if you sense anything from my facial expressions or voice about these suggestions, please further adjust your recommendations based on that. And please provide me with the adjusted product list.

[0287] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0288] Step 1:

[0289] Users take photos of their rooms with their smartphones (user devices). They take photos under appropriate lighting from multiple angles so that the entire room is visible, and then upload the photos using a dedicated app. At this time, the input data is image data of the room, and encrypted image data is generated as output and sent to the server.

[0290] Step 2:

[0291] The server receives encrypted image data sent from the user terminal. It decodes the received image data and prepares it for analysis. The input data is encrypted image data, and the output data is image data that can be analyzed.

[0292] Step 3:

[0293] The server analyzes the received image data using AI technology (e.g., TensorFlow or OpenCV). During the analysis process, the furniture in the room, the layout, the position of the walls and floors, and even the position of the windows and doors are identified. The input data is analyzable image data, and the output generates room attribute data (furniture position, layout information, etc.).

[0294] Step 4:

[0295] The server generates multiple redecorating ideas based on the analysis results. Using an automated design algorithm (generative design), it makes suggestions for furniture rearrangement and adding new furniture and accessories. The input data is the room's attribute data, and the generated redecorating ideas are obtained as output. For example, it generates a suggestion to move the sofa along the right wall and place a modern coffee table in the center of the living room.

[0296] Step 5:

[0297] The emotion engine analyzes the user's facial expressions and voice to recognize their emotions. Using facial recognition software and voice analysis software, it evaluates the user's emotional response to the proposed redecorating ideas. The input data is the user's facial expressions and voice data, and the output is recognized emotion data.

[0298] Step 6:

[0299] The server adjusts the redecorating ideas based on the emotion analysis results and re-suggests ideas based on the ones that the user finds favorable. The input data are the recognized emotion data and the initial redecorating ideas, and the output is the adjusted redecorating ideas.

[0300] Step 7:

[0301] The server creates a list of recommended products related to the tailored redecorating plan. It uses the sales platform's API to obtain purchase links for the relevant products. The input data is the tailored redecorating plan, and the output is a list of recommended products. For example, a sales page link for the proposed coffee table is added to the list of recommended products.

[0302] Step 8:

[0303] The server transmits the generated redecorating ideas and the list of recommended products to the user terminal. The input data are the adjusted redecorating ideas and the list of recommended products, and the display data transmitted to the user terminal is obtained as the output.

[0304] Step 9:

[0305] The user device receives the sent redecorating ideas and recommended product list and displays them on the screen. The user can check these contents and select the products they want to purchase. The input data is the display data sent from the server, and is displayed on the user's screen as output.

[0306] Step 10:

[0307] The user selects the product they wish to purchase from the list of recommended products displayed on their device. Clicking on the purchase link will take them to the sales page for that product, where they can complete the purchase procedure. The input data is the selected recommended product link, and the output is the display of the sales page and the completion of the purchase procedure.

[0308] (Application example 2)

[0309] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0310] Conventional interior redecorating systems have the problem that it is difficult to select optimal products when users select and purchase interior products in a physical store because they lack suggestions based on actual layout images and emotions.In addition, there is also the problem of low user satisfaction because suggestions are made without considering the user's feelings about the proposed redecorating ideas.

[0311] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for uploading image data of a room taken by a user terminal to the server, means for analyzing the image data by the server and extracting information about the furniture and layout in the room, means for generating redecorating suggestions and creating a list of recommended products based on the extracted information, means for analyzing emotions from the user's facial expressions and voice using emotion analysis means and optimizing the redecorating suggestions based on the analyzed emotions, means for linking to a sales platform that provides the recommended products, means for transmitting the proposed redecorating suggestions and the list of recommended products to the user terminal, and means for displaying the proposed redecorating suggestions in a physical store and supporting the purchase of related products. This allows users to visually check the actual room layout even in a physical store and select and purchase the most suitable interior products based on their emotions.

[0312] A "user terminal" is a computing device for taking images and transmitting data, and includes smartphones, tablets, computers, etc.

[0313] A "server" is a central processing unit that receives, analyzes, and processes data sent from user terminals.

[0314] "Image data" refers to a photograph of a room taken and uploaded from a user terminal, based on which information about the furniture and layout within the room is extracted.

[0315] "Information about furniture and layout in the room" refers to data such as the position, arrangement, and shape of furniture and interior items arranged in the room.

[0316] A "redecoration suggestion" is a new interior layout proposal generated based on the room data analyzed by the server and the user's emotional information.

[0317] The "list of recommended products" is a list of interior products and furniture recommended to the user, selected by the server based on the redecorating suggestions.

[0318] "Emotion analysis means" is a technology for analyzing a user's facial expressions and voice to recognize their emotional state.

[0319] "Sales Platform" means an online or offline marketplace that offers recommended products and provides links to purchase the products.

[0320] The "means for transmitting to the user terminal" refers to a communication technology for transmitting the redecorating suggestions and recommended product list generated by the server to the user terminal.

[0321] A "physical store" is a physical store that a user can actually visit and where they purchase interior goods.

[0322] "Analyzing emotions and optimizing redecorating suggestions based on that" is a process of adjusting the suggestions based on the user's emotional data to increase user satisfaction.

[0323] "Supporting the purchase of related products" refers to providing information and systems that allow users to easily consider and purchase suggested products in physical stores.

[0324] The system for implementing this invention includes a user terminal, a server, emotion analysis means, and a link to a sales platform. A specific method for allowing a user to select interior products, redecorate, and increase satisfaction will be described below.

[0325] System program configuration

[0326] 1. User Device:

[0327] A smartphone is used as a user terminal. The user takes pictures of the room using the smartphone and uploads them to the server via the application.

[0328] 2. Server:

[0329] The server receives image data sent from the user's device and uses image analysis software (e.g., OpenCV) to extract information about the furniture and layout of the room. Based on the analysis results, the AI ​​model generates multiple redecorating suggestions.

[0330] 3. Emotion analysis means:

[0331] The emotion analysis method uses a smart device equipped with a camera and microphone. This collects the facial expressions and voices the user makes in response to the suggestions, which are then analyzed using emotion analysis software (e.g., the EmotionRecognition library). Based on the results of this analysis, the server generates optimal redecorating suggestions that reflect the user's interests.

[0332] 4. Create and send a list of recommended products:

[0333] The system selects recommended products related to the proposed redecorating plan, generates a list with links to purchase these products, and sends the list and suggestions to the user's device so that the user can review them.

[0334] Details of data processing and calculation

[0335] The server processes the data in the following steps:

[0336] 1. Analyze the image data received from the user device using image analysis technology such as OpenCV.

[0337] 2. Extract the position information of furniture, walls, floors, windows, etc. in the room from the analysis results.

[0338] 3. Use an AI model to generate multiple redecorating suggestions based on this information.

[0339] 4. The EmotionRecognition library analyzes the user's facial expressions and voice data in response to the proposed content to determine their emotional state.

[0340] 5. The server generates further optimized redecorating suggestions that reflect the emotional data.

[0341] 6. Create a list of recommended products and generate links to the corresponding sales platforms.

[0342] 7. Send optimized redecorating suggestions and recommended product lists to the user device.

[0343] Specific examples

[0344] For example, in a physical store, a user can input a prompt such as, "I'm thinking of redecorating my living room. Please suggest new furniture layouts and recommend products based on the photos below." Then, by inputting a response to the proposed content, such as, "The user has a happy expression on their face in response to the proposal. Please provide a more optimized layout proposal," an emotion analysis is performed based on the happy expression, and optimal proposals that reflect this are presented.

[0345] In this way, users can visually check the actual room layout even in a physical store, and select and purchase the most suitable interior goods based on their emotions.

[0346] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0347] Step 1:

[0348] The user takes a picture of the room on their device and uploads it to the server via the application. The user uses the camera on their smartphone to take a picture of the entire room. After taking the picture, they select the image data in the application and press the upload button. This sends the image data to the server.

[0349] Input: A picture of the room taken by the user with their smartphone

[0350] Output: Image data uploaded to the server

[0351] Step 2:

[0352] The server analyzes the received image data and extracts positional information for the furniture, walls, floors, windows, etc. in the room. The server uses image analysis software (e.g., OpenCV) to detect objects in the image and extract their respective positions and shapes as data.

[0353] Input: Uploaded image data

[0354] Output: Information about furniture and layout in the room

[0355] Step 3:

[0356] The server uses the extracted information to generate multiple redecorating suggestions using an AI model, and then uses machine learning algorithms to generate several new furniture placement options.

[0357] Input: Information about the furniture and layout in the room

[0358] Output: Multiple redecorating suggestions

[0359] Step 4:

[0360] The camera and microphone on the user's device are used to collect the facial expressions and voices of the user in response to the suggestions. While the user is checking the suggestions, the camera on the smartphone or HMD captures the facial expressions and the microphone records the voice.

[0361] Input: User's facial expression data and voice data

[0362] Output: Captured user emotion data

[0363] Step 5:

[0364] The server uses emotion analysis software (e.g., EmotionRecognition library) to analyze the captured user emotion data and optimize the redecorating suggestions based on it. The server analyzes the user emotion data and tailors the suggestions to those that show interest.

[0365] Input: Captured user emotion data

[0366] Output: Optimized redecorating suggestions

[0367] Step 6:

[0368] The server creates a list of recommended products and generates links to the corresponding sales platforms. Based on the redecorating proposal, the server selects appropriate products from the online marketplace and compiles a list of their purchase links.

[0369] Input: Optimized redecorating suggestions

[0370] Output: A list of recommended products and a link to purchase them

[0371] Step 7:

[0372] The server sends the optimized redecorating suggestions and recommended product list to the user's device, which displays the received data so that the user can check the redecorating suggestions and product details.

[0373] Input: List of recommended products and purchase links

[0374] Output: Redecorating suggestions and recommended product list displayed on the user's device

[0375] Step 8:

[0376] The user selects the product they wish to purchase from the recommended product list and clicks the link to complete the purchase process. When the user clicks the link, they are redirected to the product purchase page on the sales platform, where they can complete the purchase process.

[0377] Input: User selected product link

[0378] Output: Transition to the purchase page on the sales platform and completion of purchase

[0379] This series of processing steps allows users to visually check redecorating suggestions even in a physical store and purchase the most suitable interior products based on their own feelings.

[0380] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0381] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0382] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0383] [Second embodiment]

[0384] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0385] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0386] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0387] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0388] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0389] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0390] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0391] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0392] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0393] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0394] In the smart glasses 214, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0395] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0396] The present invention provides a system that allows users to easily rearrange the interior of their own rooms. This system is implemented based on the following specific operational flow, which operates via a user terminal, a server, and the Internet.

[0397] First, the user takes a photo of their own room. Using the user's device (e.g., a smartphone), the user positions the camera so that the entire room they want to redecorate is captured, and takes a photo. After taking the photo, the user uses the app's upload function to send the photo data to the server.

[0398] The server then receives the image data sent by the user and analyzes it to extract important information such as the room's furniture, layout, wall and floor locations, and even the location of windows and doors. This analysis is performed using advanced AI technology.

[0399] Based on the analyzed information, the server generates multiple redecorating ideas, including new furniture arrangements and suggestions for adding new furniture and accessories. For example, the server might suggest moving the sofa to the right wall and placing a modern coffee table in the center of the living room.

[0400] The server then generates a list of recommended products suitable for the user based on the home improvement ideas. This list includes products selected from online marketplaces such as Yahoo! Shopping. For example, the server generates a link to a product on a sales platform that corresponds to the suggested coffee table.

[0401] The server then sends these redecorating ideas and a list of recommended products to the user's device, which receives the data and displays it on the screen in a user-friendly format, including a diagram of the new layout, images of the recommended furniture, and a link to purchase the items.

[0402] Users can select the product they want to purchase from a list of recommended products displayed on their device. When the user clicks on the purchase link, they are redirected to the product page and can complete the purchase process. For example, the user selects a coffee table with a modern design and clicks on the link to complete the purchase process on the Yahoo! Shopping page.

[0403] In this way, the system of the present invention allows users to easily rearrange their interiors and helps them create their ideal rooms. It also provides companies with an effective means of promoting new products.

[0404] The processing flow will be explained below.

[0405] Step 1:

[0406] The user starts a photography app on their smartphone and takes a photo of the room they want to change the interior of.

[0407] Step 2:

[0408] The user selects a photo and clicks the upload button in the app. The user's device sends the captured image data to the server.

[0409] Step 3:

[0410] The server receives the image data. The server receives and stores the image data of the room sent from the user terminal.

[0411] Step 4:

[0412] The server performs image analysis and passes the received image data to the AI ​​platform to extract information about furniture, layout, walls, floors, windows, doors, etc.

[0413] Step 5:

[0414] The server generates redecorating ideas based on the extracted information. The server creates multiple redecorating plans and suggests new layouts and additional furniture that are suitable for the user's room.

[0415] Step 6:

[0416] The server creates a list of recommended products. Based on the generated redecorating ideas, the server searches online marketplaces for recommended products and creates a list.

[0417] Step 7:

[0418] The server sends a list of redecorating ideas and recommended products to the user's device. The server then sends this data to the user's device.

[0419] Step 8:

[0420] The user's device receives the data and displays the information on the display screen, allowing the user to see the new layout proposal and recommended product list on the device screen.

[0421] Step 9:

[0422] The user selects a recommended product and completes the purchase procedure. The user selects the product they wish to purchase from the list of recommended products, clicks on the link to the sales platform, and is taken to the purchase page to complete the purchase procedure.

[0423] Example 1

[0424] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0425] In the past, when users rearranged their room interiors, not only did it take a lot of time and effort to arrange furniture and select new items, but it was also difficult to imagine a specific layout plan. Furthermore, purchasing new furniture required searching blindly, which was not an efficient process. As a result, there was insufficient support for creating the ideal room.

[0426] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0427] In this invention, the server includes means for uploading image data of a room taken by a user terminal to the server, means for the server to analyze the image data and use advanced AI technology to extract the furniture, layout, wall and floor positions, and window and door positions within the room, means for generating redecorating proposals including new furniture arrangements and suggestions for adding new products based on the extracted information and creating a list of recommended products from an online marketplace, means for linking to a sales platform that provides the recommended products, and means for transmitting the proposed redecorating proposals and the list of recommended products to the user terminal, thereby enabling users to easily rearrange their interiors and create their ideal room.

[0428] A "user terminal" is an electronic device used by a user to take photos of a room and upload data, such as a smartphone or tablet.

[0429] A "server" is a computer system that functions as a central control device, receives and analyzes image data sent from user terminals, and generates and sends lists of redecorating suggestions and recommended products.

[0430] "Image data" is photographic data of the room taken by the user terminal, and is a digital image file that is uploaded to the server.

[0431] "Advanced AI technology" refers to technology that uses artificial intelligence, particularly algorithms and models for image analysis, data extraction, and layout proposals.

[0432] "Furniture" and "layout" refer to the items arranged in a room and their arrangement. Specifically, this includes interior items such as sofas, tables, shelves, and their arrangement.

[0433] "Redecoration proposal" refers to a proposal for a new interior layout or product addition to a room, generated by the server. For example, it includes a proposal for rearranging furniture or introducing new items.

[0434] A "recommended product list" is a list of products selected based on the redecorating suggestions and suggested to the user, including products selected from online marketplaces and their purchase links.

[0435] "Online marketplace" refers to a platform for selling goods over the Internet, including, for example, general e-commerce sites.

[0436] "Means of linking to a sales platform" refers to a method of providing a link to the product purchase page of an online marketplace that sells recommended products.

[0437] "Transmission means" refers to a method for sending data from one party to another, and in particular refers to a method for sending a list of redecorating suggestions or recommended products from a server to a user terminal.

[0438] The present invention provides a system that allows users to easily redecorate the interior of their own rooms. This system operates via a user terminal, a server, and the Internet. Specific embodiments are described below.

[0439] First, the user uses a user device (e.g., a smartphone) to take a photo of their room. The user positions the camera so that the entire room they want to redecorate is captured, and takes a photo. After taking the photo, the user uses the upload function in the app on their device to send the photo data to the server.

[0440] The server then receives the photo data sent from the user's device. The server incorporates advanced AI technology, which is used to analyze the image data. The analysis extracts important information, such as the room's furniture, layout, wall and floor positions, and even the location of windows and doors. This allows the current state of the room to be accurately determined.

[0441] The server then generates multiple redecorating ideas based on the extracted information. This process uses a generative AI model. Ideas include new furniture arrangements and suggestions for adding new furniture and accessories. For example, the server might automatically generate a layout that moves the sofa along the right wall and places a modern coffee table in the center of the living room.

[0442] Furthermore, the server generates a list of recommended products suitable for the user based on these home improvement ideas. This list is composed of products selected from online marketplaces (e.g., general e-commerce sites). The server generates links to furniture and miscellaneous items suitable for each suggestion to complete the list of recommended products.

[0443] The server then sends the list of redecorating ideas and recommended products to the user's device, which receives the data and displays it on a screen in an easy-to-understand format. The displayed content includes a diagram of the new layout plan, images of the recommended furniture, and a link to purchase the items.

[0444] The user can select the product they want to purchase from the list of recommended products displayed on the device. When the user clicks on the purchase link, they are redirected to the product page and can complete the purchase. For example, the user selects a coffee table with a modern design and clicks on the link to complete the purchase on the e-commerce site page.

[0445] An example prompt is:

[0446] "Users take photos of their rooms with their smartphones and upload them to the server. The server uses AI technology to analyze the images and automatically recognize furniture placement and layout. The server then generates multiple redecorating ideas and creates a list with links to suitable products. This list is then sent to the user's device, where the user can view and purchase the products."

[0447] "A user takes a picture of their living room with their smartphone and uploads it to the server through the app. The server analyzes the image and recognizes the furniture layout. The server then generates an idea to move the sofa along the right wall and place a modern coffee table in the center. The server creates a list of coffee tables, including links to popular e-commerce sites, and sends it to the user. The user reviews the list and purchases the coffee table."

[0448] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0449] Step 1:

[0450] The user takes a photo of the room.

[0451] Input: The user's room status

[0452] How it works: A user uses their smartphone to take a photo of the room they want to redecorate.

[0453] Output: Image data of the photographed room

[0454] Step 2:

[0455] The user uploads image data from the terminal to the server.

[0456] Input: Image data of the room

[0457] Operation: The user uses the application on the device to tap the upload button to send the captured image data to the server.

[0458] Output: Image data uploaded to the server

[0459] Step 3:

[0460] The server receives and analyzes the image data.

[0461] Input: Image data uploaded by the user

[0462] How it works: The server uses advanced AI technology to analyze the received image data and extract information such as the furniture in the room, layout, the position of walls and floors, and the position of windows and doors.

[0463] Output: Information about furniture and layout in the room

[0464] Step 4:

[0465] The server generates redecorating ideas based on the extracted information.

[0466] Input: Information about the furniture and layout in the room

[0467] How it works: The server uses a generative AI model to generate multiple redecorating ideas, including new furniture arrangements and suggestions for adding new products.

[0468] Output: Redecorating ideas

[0469] Step 5:

[0470] The server creates a list of recommended products based on the redecorating ideas.

[0471] Enter: Redecorating Ideas

[0472] How it works: The server searches online marketplaces for suitable product recommendations and creates a list by generating product links for each suggestion.

[0473] Output: A list of recommended products

[0474] Step 6:

[0475] The server sends a list of redecorating ideas and recommended products to the user's terminal.

[0476] Input: A list of home improvement ideas and recommended products

[0477] Operation: The server sends the generated redecorating ideas and a list of recommended products to the user's device.

[0478] Output: A list of home improvement ideas and recommended products sent to the user's device

[0479] Step 7:

[0480] The user terminal displays the received data.

[0481] Input: A list of redecorating ideas and recommended products sent from the server

[0482] Operation: The user's device displays the received data on the screen and provides the user with new layout proposals, images of recommended products, and purchase links.

[0483] Output: A list of redecorating ideas and recommended products

[0484] Step 8:

[0485] The user selects and purchases a product from a list of recommended products.

[0486] Input: List of displayed recommended products

[0487] How it works: A user selects the product they want to purchase from a list of recommended products and clicks a link to go to the product page on the online marketplace and complete the purchase.

[0488] Output: Purchased items and completed checkout

[0489] (Application example 1)

[0490] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0491] Conventional interior redecorating systems lacked a means for users to virtually visualize the new layout, making it difficult to visualize how the room would change. They also lacked an environment in which users could smoothly purchase recommended products. Furthermore, they lacked a means to compare the real room with the virtual layout in real time, leaving them with insufficient functionality to support users in making redecorating decisions.

[0492] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0493] In this invention, the server includes means for uploading image data of a room taken by a user terminal to the server, means for analyzing the image data by the server and extracting information about the furniture and layout in the room, means for generating redecorating proposals and creating a list of recommended products based on the extracted information, means for linking to a sales platform that provides the recommended products, means for transmitting the proposed redecorating proposals and the list of recommended products to the user terminal, and means for the user to confirm and purchase the proposed redecorating proposals and recommended products displayed on the user terminal in a virtual store. This makes it easier for the user to virtually visualize a new layout and to easily purchase recommended products, thereby making it possible to smoothly proceed with redecorating decisions.

[0494] A "user terminal" is a computer device, smartphone, tablet, or other portable electronic device used by a user.

[0495] "Image data" refers to photos of the room taken by the user terminal or image files.

[0496] A "server" is a computer system that processes requests and provides data from multiple client devices within a network.

[0497] "Analysis" refers to the process performed by the server to extract and analyze necessary information from image data.

[0498] "Furniture" refers to interior items such as tables, chairs, and sofas that are placed in a room.

[0499] "Layout" refers to the arrangement and design of furniture and equipment within a room.

[0500] "Suggestion" refers to a recommendation to the user of a new layout and arrangement for redecorating.

[0501] "Recommended products" are interior products that are candidates for purchase and are recommended by the server based on the new layout plan.

[0502] "Sales Platform" refers to a website or service that allows you to sell products online.

[0503] "Link" means a hypertext link that navigates from a user terminal to the sales platform.

[0504] A "virtual store" refers to an online shopping space that uses virtual reality and augmented reality.

[0505] "Virtual visualization" is a process that allows users to visually see in real time how a new layout proposal would look in a real room in a virtual environment.

[0506] The system of the present invention is designed to allow users to easily rearrange their rooms, virtually visualize new layouts, and purchase recommended products.

[0507] Hardware and software used

[0508] User devices: Smartphones (e.g., iPhones, Android devices), tablets

[0509] Server: A powerful computer system (e.g., an EC2 instance from Amazon Web Services)

[0510] Image analysis: TensorFlow, Keras

[0511] Database: MySQL

[0512] Server framework: Django (Python)

[0513] Frontend: React Native

[0514] API: Online marketplace API (e.g. Yahoo! Shopping API)

[0515] System processing flow

[0516] Uploading image data from a user device

[0517] Users can take photos of their rooms using their smartphones or tablets and upload the image data to the server using a dedicated application. By taking a photo of the entire room using the camera on the user's device, all the necessary information is captured.

[0518] Image data analysis by the server

[0519] The server analyzes the received image data using TensorFlow and Keras. Image analysis extracts information about the layout of furniture in the room and the positions of windows, doors, etc. This analysis makes it possible to grasp the components of the room in detail.

[0520] Generate redecorating suggestions

[0521] Based on the analyzed information, the server generates multiple redecorating ideas, including new arrangements for existing furniture and suggestions for adding new interior items.

[0522] Creating and providing a list of recommended products

[0523] Based on the generated redecorating ideas, the server uses an online marketplace API to create a recommended product list, which includes links to products that correspond to the suggested furniture and interior items.

[0524] Check and purchase in the virtual store

[0525] The proposed redecorating plan and the list of recommended products are sent to the user's device, where the user can use the application installed on the device to virtually visualize the new layout and purchase the recommended products.

[0526] Specific examples

[0527] Users take photos of their living room with their smartphones and upload them to the application. The server analyzes the images using TensorFlow and Keras to identify the positions of sofas and tables, as well as the layout of walls and windows. Based on this information, the server generates modern interior design proposals and lists recommended products (new sofas, tables, and accessories) retrieved from online marketplaces. Users can then virtually visualize the new layout through the application and purchase the recommended products on the spot.

[0528] Prompt Sentence Examples

[0529] "Room Layout Recommendations"

[0530] User Scenario:

[0531] The user uploads a photo of their living room to the app. The AI ​​nestles the photo to suggest new furniture arrangements and recommends matching products available for purchase.

[0532] Prompt Input:

[0533] Room Photo: [Upload Photo]

[0534] Current Furniture Layout: [Provide AI analysis]

[0535] Suggested Furniture Rearrangement: [Generate layout suggestions]

[0536] Product Recommendations: [Fetch from Online Marketplace API]

[0537] Desired Output:

[0538] 1. New furniture layout suggestions based on the uploaded photo.

[0539] 2. List of recommended products matching the new layout.

[0540] 3. Direct links to purchase recommended products.

[0541] Action:

[0542] Process the uploaded photo to analyze the current furniture layout and generate new arrangement suggestions. Fetch and display recommended products with purchase links."

[0543] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0544] Step 1:

[0545] Taking and uploading images

[0546] The user takes a photo of the room using a smartphone or tablet. The user opens the application and selects an image of the room or takes a new one and uploads it. The input is the image data of the room, and the output is the completion of uploading the image data to the server.

[0547] Step 2:

[0548] Receiving image data

[0549] The server receives image data sent from the user terminal. The input is the image data uploaded by the user, and the output is the image data saved on the server. The server saves the image data in a specified directory.

[0550] Step 3:

[0551] Image data analysis

[0552] The server analyzes the stored image data using TensorFlow and Keras. The input is the stored image data, and the output is the analysis results. Through the analysis, information such as the position of furniture in the room, walls, windows, and doors is extracted. Specifically, the image is input into a model, and data processing is performed to identify the position of furniture and layout information.

[0553] Step 4:

[0554] Generate redecorating suggestions

[0555] The server generates multiple redecorating plans based on the analysis results. The input is the analysis results, and the output is multiple redecorating plans. The server uses a generative AI model to suggest new furniture placements and interior items to add. Each plan includes the specific placement method and the reasons for it.

[0556] Step 5:

[0557] Creating a list of recommended products

[0558] The server uses an online marketplace API to create a list of recommended products based on the redecorating plan. The input is the redecorating plan, and the output is a list of recommended products. The server calls the API to obtain product information (price, links, images, etc.) corresponding to the suggested furniture.

[0559] Step 6:

[0560] Sending data

[0561] The server sends the generated list of redecorating ideas and recommended products to the user's device. The input is the list of redecorating ideas and recommended products, and the output is the completion of sending the data to the user's device. The server then sends data to the user's device to display this information.

[0562] Step 7:

[0563] Virtual store visualization

[0564] The user uses an application installed on their device to virtually visualize the new layout. The input is a list of redecorating ideas and recommended products sent from the server, and the output is the user's reaction to the visualized layout and their selection. The user can view the proposed layout using 3D models and AR functions to visualize the changes concretely.

[0565] Step 8:

[0566] Purchase recommended products

[0567] The user clicks on a link in the application to purchase the recommended product in the virtual store. The input is the link to the recommended product, and the output is the completion of the product purchase procedure. The user opens the link on their device and proceeds with the purchase procedure on the virtual store's details page.

[0568] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0569] The present invention is a system that allows users to easily rearrange the interior of their own rooms. This system is realized based on the following specific operational flow, which operates via a user terminal, a server, an emotion engine, and the Internet.

[0570] First, the user takes a photo of their own room. Using a user device (such as a smartphone), the user positions the camera so that the entire room in which they wish to change the interior is captured, and takes a photo. After taking the photo, the user uses the upload function in the app to send the photo data to the server.

[0571] The server then receives the image data sent by the user and analyzes it to extract important information such as the room's furniture, layout, wall and floor locations, and even the location of windows and doors. This analysis is performed using advanced AI technology.

[0572] Based on the analyzed information, the server generates multiple redecorating ideas, including new furniture arrangements and suggestions for adding new furniture and accessories. For example, the server might suggest moving the sofa to the right wall and placing a modern coffee table in the center of the living room.

[0573] Next, the emotion engine analyzes the user's facial expressions and voice to recognize their emotions. The emotion engine analyzes how the user feels about the proposed redecorating ideas and adjusts the ideas accordingly. For example, the server may make suggestions specific to the redecorating ideas the user is interested in, providing layout proposals that match the user's preferences.

[0574] After using the emotion engine to select the best idea that matches the user's emotions, the server creates a list of recommended products. This list includes products selected from online marketplaces such as Yahoo! Shopping. For example, the server generates a link to a product on a sales platform that corresponds to the suggested coffee table. This list is customized based on the user's emotions, so it recommends products that are more suitable for the user.

[0575] The server then sends these redecorating ideas and a list of recommended products to the user's device, which receives the data and displays it on the screen in a user-friendly format, including a diagram of the new layout, images of the recommended furniture, and a link to purchase the items.

[0576] Users can select the product they want to purchase from a list of recommended products displayed on their device. When the user clicks on the purchase link, they are redirected to the product page and can complete the purchase process. For example, the user selects a coffee table with a modern design and clicks on the link to complete the purchase process on the Yahoo! Shopping page.

[0577] In this way, the system of the present invention allows users to easily rearrange their interiors and support the creation of their ideal room. Furthermore, by taking the user's emotions into consideration, more personalized suggestions become possible. Furthermore, it can provide an effective means for companies to promote new products.

[0578] The processing flow will be explained below.

[0579] Step 1:

[0580] The user launches the camera app on their smartphone and takes a photo of the room they want to redecorate.

[0581] Step 2:

[0582] The user sends the photos they have taken to the server using the upload function within the app.

[0583] Step 3:

[0584] The server receives and stores the image data sent from the user terminal.

[0585] Step 4:

[0586] The server uses image analysis algorithms to analyze the image data and identify the furniture, layout, walls, floors, windows, doors, etc. within the room.

[0587] Step 5:

[0588] Based on the extracted information, the server generates multiple redecorating ideas, including suggestions for new furniture arrangements and additional furniture.

[0589] Step 6:

[0590] The emotion engine collects the user's facial and voice data and analyzes their emotions. For example, if the user smiles into the smartphone camera, the emotion engine will detect a positive emotion.

[0591] Step 7:

[0592] The emotion engine evaluates the redecorating ideas based on the user's emotions and adjusts the suggestions according to the user's emotions. The emotion engine reevaluates the suggestions and selects ideas that are likely to satisfy the user.

[0593] Step 8:

[0594] The server generates a customized list of recommended products based on the results of the emotion engine, which includes products that match the user's emotions.

[0595] Step 9:

[0596] The server sends a list of redecorating ideas and recommended products to the user's terminal.

[0597] Step 10:

[0598] The user's device receives the data from the server and displays it in a user-friendly format, including diagrams of new layout plans, images of recommended products, and links to purchase them.

[0599] Step 11:

[0600] Users select the product they want to purchase from the recommended product list and click the link to proceed to the sales platform and complete the purchase. For example, a user can purchase a modern coffee table on Yahoo! Shopping.

[0601] In this way, the system of the present invention allows users to easily rearrange their interiors and supports them in making optimal suggestions and purchasing products that suit their own emotions.

[0602] Example 2

[0603] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0604] Conventional interior redecorating systems simply analyze the layout information of a user's room and make suggestions. As a result, the suggested redecorating ideas often do not match the user's preferences, and suggestions do not take the user's emotions into consideration. Furthermore, the convenience of purchasing the recommended products is insufficient.

[0605] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0606] In this invention, the server includes means for uploading image data of a room taken by a user terminal to the server, means for the server to analyze the image data and extract information about the furniture and layout of the room, means for generating redecorating proposals and creating a list of recommended products based on the extracted information, means for linking to a sales platform that provides the recommended products, means for transmitting the proposed redecorating proposals and the list of recommended products to the user terminal, means for analyzing the user's facial expressions and voice and recognizing their emotions, and means for adjusting the redecorating proposals based on the recognized emotions. This enables more personalized redecorating proposals based on the user's emotions, thereby increasing user satisfaction. It also simplifies the process of purchasing recommended products, improving the user experience.

[0607] A "user terminal" is an information processing device that can be operated by a user and that can input and display data.

[0608] A "server" is a central information processing device that is connected to user terminals via a network and performs data recording, management, analysis, and the like.

[0609] "Image data" is digital data containing visual information captured and recorded by a user terminal.

[0610] "Upload" refers to the operation and process of sending image data from a user terminal to a server.

[0611] "Analysis" is the process of automatically processing the information contained in the image data and extracting specific information (e.g., furniture or layout).

[0612] "Furniture" refers to items such as desks, chairs, sofas, and shelves that are placed in a room.

[0613] "Layout" refers to the arrangement and placement of various furniture and decorative items in a room.

[0614] "Redecoration proposals" involve providing specific proposals and ideas for changing the existing interior layout.

[0615] "Recommended products" are commercial items related to the redecorating suggestions and recommended to the user.

[0616] A "sales platform" is an online marketplace or shopping site that sells recommended products.

[0617] "Linking" refers to the process of making the relevant product page on the sales platform accessible from the display screen of the user's terminal.

[0618] "Facial expressions" refer to emotions and reactions expressed through changes in the user's face.

[0619] "Speech" refers to information produced by the user's voice or words.

[0620] "Emotion recognition" means analyzing facial expressions and voice data to determine the user's psychological state and reactions.

[0621] "Adjusting" means optimizing redecorating suggestions and product recommendations based on perceived emotions.

[0622] This invention is a system that allows users to easily redecorate the interior of their own room. This system operates via a user terminal, a server, an emotion engine, and the Internet. The following describes in detail the mode for carrying out the invention.

[0623] User device operation

[0624] Users use their smartphone or other device to take photos of their room so that the entire room is visible. Taking photos from multiple angles and under appropriate lighting improves the accuracy of analysis. The photos are then sent to the server using the upload function of the dedicated app. The data sent is encrypted to protect privacy.

[0625] Server Processing

[0626] The server receives image data sent from the user's device. It decodes the image data and analyzes it using AI technology (e.g., TensorFlow or OpenCV). During the analysis process, the furniture in the room, its layout, the location of walls and floors, and the location of windows and doors are identified. For example, the contours and positions of furniture such as sofas and tables can be identified and mapped.

[0627] Once the analysis is complete, the server generates multiple redecorating ideas using automated design algorithms (e.g., generative design). These ideas include new furniture arrangements and suggestions for adding new furniture and accessories. For example, a layout suggestion might be to move the sofa along the right wall and place a modern coffee table in the center of the living room.

[0628] Emotion engine processing

[0629] The emotion engine analyzes the user's facial expressions and voice to recognize their emotions. It uses facial recognition software (e.g., Microsoft Azure Face API) and voice analysis software (e.g., Google Cloud Speech-to-Text). Based on the analysis results, it evaluates how the user feels about the proposed redecorating ideas. This allows it to make adjustments based on the ideas that the user likes.

[0630] Product recommendations and delivery

[0631] The server creates a list of recommended products based on the results of the emotion engine. This list is generated using APIs of various online marketplaces. The list includes links to purchase suggested new furniture and accessories. For example, a link to a sales page for a modern coffee table is generated.

[0632] The generated list of redecorating ideas and recommended products is sent to the user's device, which receives it and displays it on the screen. The displayed content includes a diagram of the new layout plan, images of the recommended products, and a link to purchase them.

[0633] Product purchase

[0634] The user selects the product they want to purchase from the list of recommended products displayed on their device. Clicking the purchase link takes them to the sales page for that product and allows them to complete the purchase process. For example, the user selects a coffee table with a modern design and clicks the link to complete the purchase process.

[0635] Specific examples

[0636] 1. Examples

[0637] A user wants to redecorate their living room and takes a photo of the room with their smartphone.

[0638] Upload the photo to the server via the app.

[0639] The server analyzes the image data and generates multiple layout proposals (e.g., placing the sofa against the wall and a new coffee table in the center).

[0640] The emotion engine recognizes happy reactions from the user's facial expressions.

[0641] The server selects and proposes the optimal layout plan based on the emotions.

[0642] Provides a list of recommended coffee tables with links to online marketplaces.

[0643] 2. Prompt sentence for generative AI model

[0644] I'd like to redecorate my living room, so I'll upload a photo of the entire room. Please analyze it and suggest new furniture arrangements and new furniture and accessories that go with it. Also, if you sense anything from my facial expressions or voice about these suggestions, please further adjust your recommendations based on that. And please provide me with the adjusted product list.

[0645] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0646] Step 1:

[0647] Users take photos of their rooms with their smartphones (user devices). They take photos under appropriate lighting from multiple angles so that the entire room is visible, and then upload the photos using a dedicated app. At this time, the input data is image data of the room, and encrypted image data is generated as output and sent to the server.

[0648] Step 2:

[0649] The server receives encrypted image data sent from the user terminal. It decodes the received image data and prepares it for analysis. The input data is encrypted image data, and the output data is image data that can be analyzed.

[0650] Step 3:

[0651] The server analyzes the received image data using AI technology (e.g., TensorFlow or OpenCV). During the analysis process, the furniture in the room, the layout, the position of the walls and floors, and even the position of the windows and doors are identified. The input data is analyzable image data, and the output generates room attribute data (furniture position, layout information, etc.).

[0652] Step 4:

[0653] The server generates multiple redecorating ideas based on the analysis results. Using an automated design algorithm (generative design), it makes suggestions for furniture rearrangement and adding new furniture and accessories. The input data is the room's attribute data, and the generated redecorating ideas are obtained as output. For example, it generates a suggestion to move the sofa along the right wall and place a modern coffee table in the center of the living room.

[0654] Step 5:

[0655] The emotion engine analyzes the user's facial expressions and voice to recognize their emotions. Using facial recognition software and voice analysis software, it evaluates the user's emotional response to the proposed redecorating ideas. The input data is the user's facial expressions and voice data, and the output is recognized emotion data.

[0656] Step 6:

[0657] The server adjusts the redecorating ideas based on the emotion analysis results and re-suggests ideas based on the ones that the user finds favorable. The input data are the recognized emotion data and the initial redecorating ideas, and the output is the adjusted redecorating ideas.

[0658] Step 7:

[0659] The server creates a list of recommended products related to the tailored redecorating plan. It uses the sales platform's API to obtain purchase links for the relevant products. The input data is the tailored redecorating plan, and the output is a list of recommended products. For example, a sales page link for the proposed coffee table is added to the list of recommended products.

[0660] Step 8:

[0661] The server transmits the generated redecorating ideas and the list of recommended products to the user terminal. The input data are the adjusted redecorating ideas and the list of recommended products, and the display data transmitted to the user terminal is obtained as the output.

[0662] Step 9:

[0663] The user device receives the sent redecorating ideas and recommended product list and displays them on the screen. The user can check these contents and select the products they want to purchase. The input data is the display data sent from the server, and is displayed on the user's screen as output.

[0664] Step 10:

[0665] The user selects the product they wish to purchase from the list of recommended products displayed on their device. Clicking on the purchase link will take them to the sales page for that product, where they can complete the purchase procedure. The input data is the selected recommended product link, and the output is the display of the sales page and the completion of the purchase procedure.

[0666] (Application example 2)

[0667] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0668] Conventional interior redecorating systems have the problem that it is difficult to select optimal products when users select and purchase interior products in a physical store because they lack suggestions based on actual layout images and emotions.In addition, there is also the problem of low user satisfaction because suggestions are made without considering the user's feelings about the proposed redecorating ideas.

[0669] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for uploading image data of a room taken by a user terminal to the server, means for analyzing the image data by the server and extracting information about the furniture and layout in the room, means for generating redecorating suggestions and creating a list of recommended products based on the extracted information, means for analyzing emotions from the user's facial expressions and voice using emotion analysis means and optimizing the redecorating suggestions based on the analyzed emotions, means for linking to a sales platform that provides the recommended products, means for transmitting the proposed redecorating suggestions and the list of recommended products to the user terminal, and means for displaying the proposed redecorating suggestions in a physical store and supporting the purchase of related products. This allows users to visually check the actual room layout even in a physical store and select and purchase the most suitable interior products based on their emotions.

[0670] A "user terminal" is a computing device for taking images and transmitting data, and includes smartphones, tablets, computers, etc.

[0671] A "server" is a central processing unit that receives, analyzes, and processes data sent from user terminals.

[0672] "Image data" refers to a photograph of a room taken and uploaded from a user terminal, based on which information about the furniture and layout within the room is extracted.

[0673] "Information about furniture and layout in the room" refers to data such as the position, arrangement, and shape of furniture and interior items arranged in the room.

[0674] A "redecoration suggestion" is a new interior layout proposal generated based on the room data analyzed by the server and the user's emotional information.

[0675] The "list of recommended products" is a list of interior products and furniture recommended to the user, selected by the server based on the redecorating suggestions.

[0676] "Emotion analysis means" is a technology for analyzing a user's facial expressions and voice to recognize their emotional state.

[0677] "Sales Platform" means an online or offline marketplace that offers recommended products and provides links to purchase the products.

[0678] The "means for transmitting to the user terminal" refers to a communication technology for transmitting the redecorating suggestions and recommended product list generated by the server to the user terminal.

[0679] A "physical store" is a physical store that a user can actually visit and where they purchase interior goods.

[0680] "Analyzing emotions and optimizing redecorating suggestions based on that" is a process of adjusting the suggestions based on the user's emotional data to increase user satisfaction.

[0681] "Supporting the purchase of related products" refers to providing information and systems that allow users to easily consider and purchase suggested products in physical stores.

[0682] The system for implementing this invention includes a user terminal, a server, emotion analysis means, and a link to a sales platform. A specific method for allowing a user to select interior products, redecorate, and increase satisfaction will be described below.

[0683] System program configuration

[0684] 1. User Device:

[0685] A smartphone is used as a user terminal. The user takes pictures of the room using the smartphone and uploads them to the server via the application.

[0686] 2. Server:

[0687] The server receives image data sent from the user's device and uses image analysis software (e.g., OpenCV) to extract information about the furniture and layout of the room. Based on the analysis results, the AI ​​model generates multiple redecorating suggestions.

[0688] 3. Emotion analysis means:

[0689] The emotion analysis method uses a smart device equipped with a camera and microphone. This collects the facial expressions and voices the user makes in response to the suggestions, which are then analyzed using emotion analysis software (e.g., the EmotionRecognition library). Based on the results of this analysis, the server generates optimal redecorating suggestions that reflect the user's interests.

[0690] 4. Create and send a list of recommended products:

[0691] The system selects recommended products related to the proposed redecorating plan, generates a list with links to purchase these products, and sends the list and suggestions to the user's device so that the user can review them.

[0692] Details of data processing and calculation

[0693] The server processes the data in the following steps:

[0694] 1. Analyze the image data received from the user device using image analysis technology such as OpenCV.

[0695] 2. Extract the position information of furniture, walls, floors, windows, etc. in the room from the analysis results.

[0696] 3. Use an AI model to generate multiple redecorating suggestions based on this information.

[0697] 4. The EmotionRecognition library analyzes the user's facial expressions and voice data in response to the proposed content to determine their emotional state.

[0698] 5. The server generates further optimized redecorating suggestions that reflect the emotional data.

[0699] 6. Create a list of recommended products and generate links to the corresponding sales platforms.

[0700] 7. Send optimized redecorating suggestions and recommended product lists to the user device.

[0701] Specific examples

[0702] For example, in a physical store, a user can input a prompt such as, "I'm thinking of redecorating my living room. Please suggest new furniture layouts and recommend products based on the photos below." Then, by inputting a response to the proposed content, such as, "The user has a happy expression on their face in response to the proposal. Please provide a more optimized layout proposal," an emotion analysis is performed based on the happy expression, and optimal proposals that reflect this are presented.

[0703] In this way, users can visually check the actual room layout even in a physical store, and select and purchase the most suitable interior goods based on their emotions.

[0704] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0705] Step 1:

[0706] The user takes a picture of the room on their device and uploads it to the server via the application. The user uses the camera on their smartphone to take a picture of the entire room. After taking the picture, they select the image data in the application and press the upload button. This sends the image data to the server.

[0707] Input: A picture of the room taken by the user with their smartphone

[0708] Output: Image data uploaded to the server

[0709] Step 2:

[0710] The server analyzes the received image data and extracts positional information for the furniture, walls, floors, windows, etc. in the room. The server uses image analysis software (e.g., OpenCV) to detect objects in the image and extract their respective positions and shapes as data.

[0711] Input: Uploaded image data

[0712] Output: Information about furniture and layout in the room

[0713] Step 3:

[0714] The server uses the extracted information to generate multiple redecorating suggestions using an AI model, and then uses machine learning algorithms to generate several new furniture placement options.

[0715] Input: Information about the furniture and layout in the room

[0716] Output: Multiple redecorating suggestions

[0717] Step 4:

[0718] The camera and microphone on the user's device are used to collect the facial expressions and voices of the user in response to the suggestions. While the user is checking the suggestions, the camera on the smartphone or HMD captures the facial expressions and the microphone records the voice.

[0719] Input: User's facial expression data and voice data

[0720] Output: Captured user emotion data

[0721] Step 5:

[0722] The server uses emotion analysis software (e.g., EmotionRecognition library) to analyze the captured user emotion data and optimize the redecorating suggestions based on it. The server analyzes the user emotion data and tailors the suggestions to those that show interest.

[0723] Input: Captured user emotion data

[0724] Output: Optimized redecorating suggestions

[0725] Step 6:

[0726] The server creates a list of recommended products and generates links to the corresponding sales platforms. Based on the redecorating proposal, the server selects appropriate products from the online marketplace and compiles a list of their purchase links.

[0727] Input: Optimized redecorating suggestions

[0728] Output: A list of recommended products and a link to purchase them

[0729] Step 7:

[0730] The server sends the optimized redecorating suggestions and recommended product list to the user's device, which displays the received data so that the user can check the redecorating suggestions and product details.

[0731] Input: List of recommended products and purchase links

[0732] Output: Redecorating suggestions and recommended product list displayed on the user's device

[0733] Step 8:

[0734] The user selects the product they wish to purchase from the recommended product list and clicks the link to complete the purchase process. When the user clicks the link, they are redirected to the product purchase page on the sales platform, where they can complete the purchase process.

[0735] Input: User selected product link

[0736] Output: Transition to the purchase page on the sales platform and completion of purchase

[0737] This series of processing steps allows users to visually check redecorating suggestions even in a physical store and purchase the most suitable interior products based on their own feelings.

[0738] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0739] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0740] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0741] [Third embodiment]

[0742] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0743] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0744] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0745] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0746] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0747] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0748] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0749] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0750] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0751] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0752] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0753] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0754] The present invention provides a system that allows users to easily rearrange the interior of their own rooms. This system is implemented based on the following specific operational flow, which operates via a user terminal, a server, and the Internet.

[0755] First, the user takes a photo of their own room. Using the user's device (e.g., a smartphone), the user positions the camera so that the entire room they want to redecorate is captured, and takes a photo. After taking the photo, the user uses the app's upload function to send the photo data to the server.

[0756] The server then receives the image data sent by the user and analyzes it to extract important information such as the room's furniture, layout, wall and floor locations, and even the location of windows and doors. This analysis is performed using advanced AI technology.

[0757] Based on the analyzed information, the server generates multiple redecorating ideas, including new furniture arrangements and suggestions for adding new furniture and accessories. For example, the server might suggest moving the sofa to the right wall and placing a modern coffee table in the center of the living room.

[0758] The server then generates a list of recommended products suitable for the user based on the home improvement ideas. This list includes products selected from online marketplaces such as Yahoo! Shopping. For example, the server generates a link to a product on a sales platform that corresponds to the suggested coffee table.

[0759] The server then sends these redecorating ideas and a list of recommended products to the user's device, which receives the data and displays it on the screen in a user-friendly format, including a diagram of the new layout, images of the recommended furniture, and a link to purchase the items.

[0760] Users can select the product they want to purchase from a list of recommended products displayed on their device. When the user clicks on the purchase link, they are redirected to the product page and can complete the purchase process. For example, the user selects a coffee table with a modern design and clicks on the link to complete the purchase process on the Yahoo! Shopping page.

[0761] In this way, the system of the present invention allows users to easily rearrange their interiors and helps them create their ideal rooms. It also provides companies with an effective means of promoting new products.

[0762] The processing flow will be explained below.

[0763] Step 1:

[0764] The user starts a photography app on their smartphone and takes a photo of the room they want to change the interior of.

[0765] Step 2:

[0766] The user selects a photo and clicks the upload button in the app. The user's device sends the captured image data to the server.

[0767] Step 3:

[0768] The server receives the image data. The server receives and stores the image data of the room sent from the user terminal.

[0769] Step 4:

[0770] The server performs image analysis and passes the received image data to the AI ​​platform to extract information about furniture, layout, walls, floors, windows, doors, etc.

[0771] Step 5:

[0772] The server generates redecorating ideas based on the extracted information. The server creates multiple redecorating plans and suggests new layouts and additional furniture that are suitable for the user's room.

[0773] Step 6:

[0774] The server creates a list of recommended products. Based on the generated redecorating ideas, the server searches online marketplaces for recommended products and creates a list.

[0775] Step 7:

[0776] The server sends a list of redecorating ideas and recommended products to the user's device. The server then sends this data to the user's device.

[0777] Step 8:

[0778] The user's device receives the data and displays the information on the display screen, allowing the user to see the new layout proposal and recommended product list on the device screen.

[0779] Step 9:

[0780] The user selects a recommended product and completes the purchase procedure. The user selects the product they wish to purchase from the list of recommended products, clicks on the link to the sales platform, and is taken to the purchase page to complete the purchase procedure.

[0781] Example 1

[0782] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0783] In the past, when users rearranged their room interiors, not only did it take a lot of time and effort to arrange furniture and select new items, but it was also difficult to imagine a specific layout plan. Furthermore, purchasing new furniture required searching blindly, which was not an efficient process. As a result, there was insufficient support for creating the ideal room.

[0784] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0785] In this invention, the server includes means for uploading image data of a room taken by a user terminal to the server, means for the server to analyze the image data and use advanced AI technology to extract the furniture, layout, wall and floor positions, and window and door positions within the room, means for generating redecorating proposals including new furniture arrangements and suggestions for adding new products based on the extracted information and creating a list of recommended products from an online marketplace, means for linking to a sales platform that provides the recommended products, and means for transmitting the proposed redecorating proposals and the list of recommended products to the user terminal, thereby enabling users to easily rearrange their interiors and create their ideal room.

[0786] A "user terminal" is an electronic device used by a user to take photos of a room and upload data, such as a smartphone or tablet.

[0787] A "server" is a computer system that functions as a central control device, receives and analyzes image data sent from user terminals, and generates and sends lists of redecorating suggestions and recommended products.

[0788] "Image data" is photographic data of the room taken by the user terminal, and is a digital image file that is uploaded to the server.

[0789] "Advanced AI technology" refers to technology that uses artificial intelligence, particularly algorithms and models for image analysis, data extraction, and layout proposals.

[0790] "Furniture" and "layout" refer to the items arranged in a room and their arrangement. Specifically, this includes interior items such as sofas, tables, shelves, and their arrangement.

[0791] "Redecoration proposal" refers to a proposal for a new interior layout or product addition to a room, generated by the server. For example, it includes a proposal for rearranging furniture or introducing new items.

[0792] A "recommended product list" is a list of products selected based on the redecorating suggestions and suggested to the user, including products selected from online marketplaces and their purchase links.

[0793] "Online marketplace" refers to a platform for selling goods over the Internet, including, for example, general e-commerce sites.

[0794] "Means of linking to a sales platform" refers to a method of providing a link to the product purchase page of an online marketplace that sells recommended products.

[0795] "Transmission means" refers to a method for sending data from one party to another, and in particular refers to a method for sending a list of redecorating suggestions or recommended products from a server to a user terminal.

[0796] The present invention provides a system that allows users to easily redecorate the interior of their own rooms. This system operates via a user terminal, a server, and the Internet. Specific embodiments are described below.

[0797] First, the user uses a user device (e.g., a smartphone) to take a photo of their room. The user positions the camera so that the entire room they want to redecorate is captured, and takes a photo. After taking the photo, the user uses the upload function in the app on their device to send the photo data to the server.

[0798] The server then receives the photo data sent from the user's device. The server incorporates advanced AI technology, which is used to analyze the image data. The analysis extracts important information, such as the room's furniture, layout, wall and floor positions, and even the location of windows and doors. This allows the current state of the room to be accurately determined.

[0799] The server then generates multiple redecorating ideas based on the extracted information. This process uses a generative AI model. Ideas include new furniture arrangements and suggestions for adding new furniture and accessories. For example, the server might automatically generate a layout that moves the sofa along the right wall and places a modern coffee table in the center of the living room.

[0800] Furthermore, the server generates a list of recommended products suitable for the user based on these home improvement ideas. This list is composed of products selected from online marketplaces (e.g., general e-commerce sites). The server generates links to furniture and miscellaneous items suitable for each suggestion to complete the list of recommended products.

[0801] The server then sends the list of redecorating ideas and recommended products to the user's device, which receives the data and displays it on a screen in an easy-to-understand format. The displayed content includes a diagram of the new layout plan, images of the recommended furniture, and a link to purchase the items.

[0802] The user can select the product they want to purchase from the list of recommended products displayed on the device. When the user clicks on the purchase link, they are redirected to the product page and can complete the purchase. For example, the user selects a coffee table with a modern design and clicks on the link to complete the purchase on the e-commerce site page.

[0803] An example prompt is:

[0804] "Users take photos of their rooms with their smartphones and upload them to the server. The server uses AI technology to analyze the images and automatically recognize furniture placement and layout. The server then generates multiple redecorating ideas and creates a list with links to suitable products. This list is then sent to the user's device, where the user can view and purchase the products."

[0805] "A user takes a picture of their living room with their smartphone and uploads it to the server through the app. The server analyzes the image and recognizes the furniture layout. The server then generates an idea to move the sofa along the right wall and place a modern coffee table in the center. The server creates a list of coffee tables, including links to popular e-commerce sites, and sends it to the user. The user reviews the list and purchases the coffee table."

[0806] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0807] Step 1:

[0808] The user takes a photo of the room.

[0809] Input: The user's room status

[0810] How it works: A user uses their smartphone to take a photo of the room they want to redecorate.

[0811] Output: Image data of the photographed room

[0812] Step 2:

[0813] The user uploads image data from the terminal to the server.

[0814] Input: Image data of the room

[0815] Operation: The user uses the application on the device to tap the upload button to send the captured image data to the server.

[0816] Output: Image data uploaded to the server

[0817] Step 3:

[0818] The server receives and analyzes the image data.

[0819] Input: Image data uploaded by the user

[0820] How it works: The server uses advanced AI technology to analyze the received image data and extract information such as the furniture in the room, layout, the position of walls and floors, and the position of windows and doors.

[0821] Output: Information about furniture and layout in the room

[0822] Step 4:

[0823] The server generates redecorating ideas based on the extracted information.

[0824] Input: Information about the furniture and layout in the room

[0825] How it works: The server uses a generative AI model to generate multiple redecorating ideas, including new furniture arrangements and suggestions for adding new products.

[0826] Output: Redecorating ideas

[0827] Step 5:

[0828] The server creates a list of recommended products based on the redecorating ideas.

[0829] Enter: Redecorating Ideas

[0830] How it works: The server searches online marketplaces for suitable product recommendations and creates a list by generating product links for each suggestion.

[0831] Output: A list of recommended products

[0832] Step 6:

[0833] The server sends a list of redecorating ideas and recommended products to the user's terminal.

[0834] Input: A list of home improvement ideas and recommended products

[0835] Operation: The server sends the generated redecorating ideas and a list of recommended products to the user's device.

[0836] Output: A list of home improvement ideas and recommended products sent to the user's device

[0837] Step 7:

[0838] The user terminal displays the received data.

[0839] Input: A list of redecorating ideas and recommended products sent from the server

[0840] Operation: The user's device displays the received data on the screen and provides the user with new layout proposals, images of recommended products, and purchase links.

[0841] Output: A list of redecorating ideas and recommended products

[0842] Step 8:

[0843] The user selects and purchases a product from a list of recommended products.

[0844] Input: List of displayed recommended products

[0845] How it works: A user selects the product they want to purchase from a list of recommended products and clicks a link to go to the product page on the online marketplace and complete the purchase.

[0846] Output: Purchased items and completed checkout

[0847] (Application example 1)

[0848] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0849] Conventional interior redecorating systems lacked a means for users to virtually visualize the new layout, making it difficult to visualize how the room would change. They also lacked an environment in which users could smoothly purchase recommended products. Furthermore, they lacked a means to compare the real room with the virtual layout in real time, leaving them with insufficient functionality to support users in making redecorating decisions.

[0850] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0851] In this invention, the server includes means for uploading image data of a room taken by a user terminal to the server, means for analyzing the image data by the server and extracting information about the furniture and layout in the room, means for generating redecorating proposals and creating a list of recommended products based on the extracted information, means for linking to a sales platform that provides the recommended products, means for transmitting the proposed redecorating proposals and the list of recommended products to the user terminal, and means for the user to confirm and purchase the proposed redecorating proposals and recommended products displayed on the user terminal in a virtual store. This makes it easier for the user to virtually visualize a new layout and to easily purchase recommended products, thereby making it possible to smoothly proceed with redecorating decisions.

[0852] A "user terminal" is a computer device, smartphone, tablet, or other portable electronic device used by a user.

[0853] "Image data" refers to photos of the room taken by the user terminal or image files.

[0854] A "server" is a computer system that processes requests and provides data from multiple client devices within a network.

[0855] "Analysis" refers to the process performed by the server to extract and analyze necessary information from image data.

[0856] "Furniture" refers to interior items such as tables, chairs, and sofas that are placed in a room.

[0857] "Layout" refers to the arrangement and design of furniture and equipment within a room.

[0858] "Suggestion" refers to a recommendation to the user of a new layout and arrangement for redecorating.

[0859] "Recommended products" are interior products that are candidates for purchase and are recommended by the server based on the new layout plan.

[0860] "Sales Platform" refers to a website or service that allows you to sell products online.

[0861] "Link" means a hypertext link that navigates from a user terminal to the sales platform.

[0862] A "virtual store" refers to an online shopping space that uses virtual reality and augmented reality.

[0863] "Virtual visualization" is a process that allows users to visually see in real time how a new layout proposal would look in a real room in a virtual environment.

[0864] The system of the present invention is designed to allow users to easily rearrange their rooms, virtually visualize new layouts, and purchase recommended products.

[0865] Hardware and software used

[0866] User devices: Smartphones (e.g., iPhones, Android devices), tablets

[0867] Server: A powerful computer system (e.g., an EC2 instance from Amazon Web Services)

[0868] Image analysis: TensorFlow, Keras

[0869] Database: MySQL

[0870] Server framework: Django (Python)

[0871] Frontend: React Native

[0872] API: Online marketplace API (e.g. Yahoo! Shopping API)

[0873] System processing flow

[0874] Uploading image data from a user device

[0875] Users can take photos of their rooms using their smartphones or tablets and upload the image data to the server using a dedicated application. By taking a photo of the entire room using the camera on the user's device, all the necessary information is captured.

[0876] Image data analysis by the server

[0877] The server analyzes the received image data using TensorFlow and Keras. Image analysis extracts information about the layout of furniture in the room and the positions of windows, doors, etc. This analysis makes it possible to grasp the components of the room in detail.

[0878] Generate redecorating suggestions

[0879] Based on the analyzed information, the server generates multiple redecorating ideas, including new arrangements for existing furniture and suggestions for adding new interior items.

[0880] Creating and providing a list of recommended products

[0881] Based on the generated redecorating ideas, the server uses an online marketplace API to create a recommended product list, which includes links to products that correspond to the suggested furniture and interior items.

[0882] Check and purchase in the virtual store

[0883] The proposed redecorating plan and the list of recommended products are sent to the user's device, where the user can use the application installed on the device to virtually visualize the new layout and purchase the recommended products.

[0884] Specific examples

[0885] Users take photos of their living room with their smartphones and upload them to the application. The server analyzes the images using TensorFlow and Keras to identify the positions of sofas and tables, as well as the layout of walls and windows. Based on this information, the server generates modern interior design proposals and lists recommended products (new sofas, tables, and accessories) retrieved from online marketplaces. Users can then virtually visualize the new layout through the application and purchase the recommended products on the spot.

[0886] Prompt Sentence Examples

[0887] "Room Layout Recommendations"

[0888] User Scenario:

[0889] The user uploads a photo of their living room to the app. The AI ​​nestles the photo to suggest new furniture arrangements and recommends matching products available for purchase.

[0890] Prompt Input:

[0891] Room Photo: [Upload Photo]

[0892] Current Furniture Layout: [Provide AI analysis]

[0893] Suggested Furniture Rearrangement: [Generate layout suggestions]

[0894] Product Recommendations: [Fetch from Online Marketplace API]

[0895] Desired Output:

[0896] 1. New furniture layout suggestions based on the uploaded photo.

[0897] 2. List of recommended products matching the new layout.

[0898] 3. Direct links to purchase recommended products.

[0899] Action:

[0900] Process the uploaded photo to analyze the current furniture layout and generate new arrangement suggestions. Fetch and display recommended products with purchase links."

[0901] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0902] Step 1:

[0903] Taking and uploading images

[0904] The user takes a photo of the room using a smartphone or tablet. The user opens the application and selects an image of the room or takes a new one and uploads it. The input is the image data of the room, and the output is the completion of uploading the image data to the server.

[0905] Step 2:

[0906] Receiving image data

[0907] The server receives image data sent from the user terminal. The input is the image data uploaded by the user, and the output is the image data saved on the server. The server saves the image data in a specified directory.

[0908] Step 3:

[0909] Image data analysis

[0910] The server analyzes the stored image data using TensorFlow and Keras. The input is the stored image data, and the output is the analysis results. Through the analysis, information such as the position of furniture in the room, walls, windows, and doors is extracted. Specifically, the image is input into a model, and data processing is performed to identify the position of furniture and layout information.

[0911] Step 4:

[0912] Generate redecorating suggestions

[0913] The server generates multiple redecorating plans based on the analysis results. The input is the analysis results, and the output is multiple redecorating plans. The server uses a generative AI model to suggest new furniture placements and interior items to add. Each plan includes the specific placement method and the reasons for it.

[0914] Step 5:

[0915] Creating a list of recommended products

[0916] The server uses an online marketplace API to create a list of recommended products based on the redecorating plan. The input is the redecorating plan, and the output is a list of recommended products. The server calls the API to obtain product information (price, links, images, etc.) corresponding to the suggested furniture.

[0917] Step 6:

[0918] Sending data

[0919] The server sends the generated list of redecorating ideas and recommended products to the user's device. The input is the list of redecorating ideas and recommended products, and the output is the completion of sending the data to the user's device. The server then sends data to the user's device to display this information.

[0920] Step 7:

[0921] Virtual store visualization

[0922] The user uses an application installed on their device to virtually visualize the new layout. The input is a list of redecorating ideas and recommended products sent from the server, and the output is the user's reaction to the visualized layout and their selection. The user can view the proposed layout using 3D models and AR functions to visualize the changes concretely.

[0923] Step 8:

[0924] Purchase recommended products

[0925] The user clicks on a link in the application to purchase the recommended product in the virtual store. The input is the link to the recommended product, and the output is the completion of the product purchase procedure. The user opens the link on their device and proceeds with the purchase procedure on the virtual store's details page.

[0926] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0927] The present invention is a system that allows users to easily rearrange the interior of their own rooms. This system is realized based on the following specific operational flow, which operates via a user terminal, a server, an emotion engine, and the Internet.

[0928] First, the user takes a photo of their own room. Using a user device (such as a smartphone), the user positions the camera so that the entire room in which they wish to change the interior is captured, and takes a photo. After taking the photo, the user uses the upload function in the app to send the photo data to the server.

[0929] The server then receives the image data sent by the user and analyzes it to extract important information such as the room's furniture, layout, wall and floor locations, and even the location of windows and doors. This analysis is performed using advanced AI technology.

[0930] Based on the analyzed information, the server generates multiple redecorating ideas, including new furniture arrangements and suggestions for adding new furniture and accessories. For example, the server might suggest moving the sofa to the right wall and placing a modern coffee table in the center of the living room.

[0931] Next, the emotion engine analyzes the user's facial expressions and voice to recognize their emotions. The emotion engine analyzes how the user feels about the proposed redecorating ideas and adjusts the ideas accordingly. For example, the server may make suggestions specific to the redecorating ideas the user is interested in, providing layout proposals that match the user's preferences.

[0932] After using the emotion engine to select the best idea that matches the user's emotions, the server creates a list of recommended products. This list includes products selected from online marketplaces such as Yahoo! Shopping. For example, the server generates a link to a product on a sales platform that corresponds to the suggested coffee table. This list is customized based on the user's emotions, so it recommends products that are more suitable for the user.

[0933] The server then sends these redecorating ideas and a list of recommended products to the user's device, which receives the data and displays it on the screen in a user-friendly format, including a diagram of the new layout, images of the recommended furniture, and a link to purchase the items.

[0934] Users can select the product they want to purchase from a list of recommended products displayed on their device. When the user clicks on the purchase link, they are redirected to the product page and can complete the purchase process. For example, the user selects a coffee table with a modern design and clicks on the link to complete the purchase process on the Yahoo! Shopping page.

[0935] In this way, the system of the present invention allows users to easily rearrange their interiors and support the creation of their ideal room. Furthermore, by taking the user's emotions into consideration, more personalized suggestions become possible. Furthermore, it can provide an effective means for companies to promote new products.

[0936] The processing flow will be explained below.

[0937] Step 1:

[0938] The user launches the camera app on their smartphone and takes a photo of the room they want to redecorate.

[0939] Step 2:

[0940] The user sends the photos they have taken to the server using the upload function within the app.

[0941] Step 3:

[0942] The server receives and stores the image data sent from the user terminal.

[0943] Step 4:

[0944] The server uses image analysis algorithms to analyze the image data and identify the furniture, layout, walls, floors, windows, doors, etc. within the room.

[0945] Step 5:

[0946] Based on the extracted information, the server generates multiple redecorating ideas, including suggestions for new furniture arrangements and additional furniture.

[0947] Step 6:

[0948] The emotion engine collects the user's facial and voice data and analyzes their emotions. For example, if the user smiles into the smartphone camera, the emotion engine will detect a positive emotion.

[0949] Step 7:

[0950] The emotion engine evaluates the redecorating ideas based on the user's emotions and adjusts the suggestions according to the user's emotions. The emotion engine reevaluates the suggestions and selects ideas that are likely to satisfy the user.

[0951] Step 8:

[0952] The server generates a customized list of recommended products based on the results of the emotion engine, which includes products that match the user's emotions.

[0953] Step 9:

[0954] The server sends a list of redecorating ideas and recommended products to the user's terminal.

[0955] Step 10:

[0956] The user's device receives the data from the server and displays it in a user-friendly format, including diagrams of new layout plans, images of recommended products, and links to purchase them.

[0957] Step 11:

[0958] Users select the product they want to purchase from the recommended product list and click the link to proceed to the sales platform and complete the purchase. For example, a user can purchase a modern coffee table on Yahoo! Shopping.

[0959] In this way, the system of the present invention allows users to easily rearrange their interiors and supports them in making optimal suggestions and purchasing products that suit their own emotions.

[0960] Example 2

[0961] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0962] Conventional interior redecorating systems simply analyze the layout information of a user's room and make suggestions. As a result, the suggested redecorating ideas often do not match the user's preferences, and suggestions do not take the user's emotions into consideration. Furthermore, the convenience of purchasing the recommended products is insufficient.

[0963] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0964] In this invention, the server includes means for uploading image data of a room taken by a user terminal to the server, means for the server to analyze the image data and extract information about the furniture and layout of the room, means for generating redecorating proposals and creating a list of recommended products based on the extracted information, means for linking to a sales platform that provides the recommended products, means for transmitting the proposed redecorating proposals and the list of recommended products to the user terminal, means for analyzing the user's facial expressions and voice and recognizing their emotions, and means for adjusting the redecorating proposals based on the recognized emotions. This enables more personalized redecorating proposals based on the user's emotions, thereby increasing user satisfaction. It also simplifies the process of purchasing recommended products, improving the user experience.

[0965] A "user terminal" is an information processing device that can be operated by a user and that can input and display data.

[0966] A "server" is a central information processing device that is connected to user terminals via a network and performs data recording, management, analysis, and the like.

[0967] "Image data" is digital data containing visual information captured and recorded by a user terminal.

[0968] "Upload" refers to the operation and process of sending image data from a user terminal to a server.

[0969] "Analysis" is the process of automatically processing the information contained in the image data and extracting specific information (e.g., furniture or layout).

[0970] "Furniture" refers to items such as desks, chairs, sofas, and shelves that are placed in a room.

[0971] "Layout" refers to the arrangement and placement of various furniture and decorative items in a room.

[0972] "Redecoration proposals" involve providing specific proposals and ideas for changing the existing interior layout.

[0973] "Recommended products" are commercial items related to the redecorating suggestions and recommended to the user.

[0974] A "sales platform" is an online marketplace or shopping site that sells recommended products.

[0975] "Linking" refers to the process of making the relevant product page on the sales platform accessible from the display screen of the user's terminal.

[0976] "Facial expressions" refer to emotions and reactions expressed through changes in the user's face.

[0977] "Speech" refers to information produced by the user's voice or words.

[0978] "Emotion recognition" means analyzing facial expressions and voice data to determine the user's psychological state and reactions.

[0979] "Adjusting" means optimizing redecorating suggestions and product recommendations based on perceived emotions.

[0980] This invention is a system that allows users to easily redecorate the interior of their own room. This system operates via a user terminal, a server, an emotion engine, and the Internet. The following describes in detail the mode for carrying out the invention.

[0981] User device operation

[0982] Users use their smartphone or other device to take photos of their room so that the entire room is visible. Taking photos from multiple angles and under appropriate lighting improves the accuracy of analysis. The photos are then sent to the server using the upload function of the dedicated app. The data sent is encrypted to protect privacy.

[0983] Server Processing

[0984] The server receives image data sent from the user's device. It decodes the image data and analyzes it using AI technology (e.g., TensorFlow or OpenCV). During the analysis process, the furniture in the room, its layout, the location of walls and floors, and the location of windows and doors are identified. For example, the contours and positions of furniture such as sofas and tables can be identified and mapped.

[0985] Once the analysis is complete, the server generates multiple redecorating ideas using automated design algorithms (e.g., generative design). These ideas include new furniture arrangements and suggestions for adding new furniture and accessories. For example, a layout suggestion might be to move the sofa along the right wall and place a modern coffee table in the center of the living room.

[0986] Emotion engine processing

[0987] The emotion engine analyzes the user's facial expressions and voice to recognize their emotions. It uses facial recognition software (e.g., Microsoft Azure Face API) and voice analysis software (e.g., Google Cloud Speech-to-Text). Based on the analysis results, it evaluates how the user feels about the proposed redecorating ideas. This allows it to make adjustments based on the ideas that the user likes.

[0988] Product recommendations and delivery

[0989] The server creates a list of recommended products based on the results of the emotion engine. This list is generated using APIs of various online marketplaces. The list includes links to purchase suggested new furniture and accessories. For example, a link to a sales page for a modern coffee table is generated.

[0990] The generated list of redecorating ideas and recommended products is sent to the user's device, which receives it and displays it on the screen. The displayed content includes a diagram of the new layout plan, images of the recommended products, and a link to purchase them.

[0991] Product purchase

[0992] The user selects the product they want to purchase from the list of recommended products displayed on their device. Clicking the purchase link takes them to the sales page for that product and allows them to complete the purchase process. For example, the user selects a coffee table with a modern design and clicks the link to complete the purchase process.

[0993] Specific examples

[0994] 1. Examples

[0995] A user wants to redecorate their living room and takes a photo of the room with their smartphone.

[0996] Upload the photo to the server via the app.

[0997] The server analyzes the image data and generates multiple layout proposals (e.g., placing the sofa against the wall and a new coffee table in the center).

[0998] The emotion engine recognizes happy reactions from the user's facial expressions.

[0999] The server selects and proposes the optimal layout plan based on the emotions.

[1000] Provides a list of recommended coffee tables with links to online marketplaces.

[1001] 2. Prompt sentence for generative AI model

[1002] I'd like to redecorate my living room, so I'll upload a photo of the entire room. Please analyze it and suggest new furniture arrangements and new furniture and accessories that go with it. Also, if you sense anything from my facial expressions or voice about these suggestions, please further adjust your recommendations based on that. And please provide me with the adjusted product list.

[1003] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1004] Step 1:

[1005] Users take photos of their rooms with their smartphones (user devices). They take photos under appropriate lighting from multiple angles so that the entire room is visible, and then upload the photos using a dedicated app. At this time, the input data is image data of the room, and encrypted image data is generated as output and sent to the server.

[1006] Step 2:

[1007] The server receives encrypted image data sent from the user terminal. It decodes the received image data and prepares it for analysis. The input data is encrypted image data, and the output data is image data that can be analyzed.

[1008] Step 3:

[1009] The server analyzes the received image data using AI technology (e.g., TensorFlow or OpenCV). During the analysis process, the furniture in the room, the layout, the position of the walls and floors, and even the position of the windows and doors are identified. The input data is analyzable image data, and the output generates room attribute data (furniture position, layout information, etc.).

[1010] Step 4:

[1011] The server generates multiple redecorating ideas based on the analysis results. Using an automated design algorithm (generative design), it makes suggestions for furniture rearrangement and adding new furniture and accessories. The input data is the room's attribute data, and the generated redecorating ideas are obtained as output. For example, it generates a suggestion to move the sofa along the right wall and place a modern coffee table in the center of the living room.

[1012] Step 5:

[1013] The emotion engine analyzes the user's facial expressions and voice to recognize their emotions. Using facial recognition software and voice analysis software, it evaluates the user's emotional response to the proposed redecorating ideas. The input data is the user's facial expressions and voice data, and the output is recognized emotion data.

[1014] Step 6:

[1015] The server adjusts the redecorating ideas based on the emotion analysis results and re-suggests ideas based on the ones that the user finds favorable. The input data are the recognized emotion data and the initial redecorating ideas, and the output is the adjusted redecorating ideas.

[1016] Step 7:

[1017] The server creates a list of recommended products related to the tailored redecorating plan. It uses the sales platform's API to obtain purchase links for the relevant products. The input data is the tailored redecorating plan, and the output is a list of recommended products. For example, a sales page link for the proposed coffee table is added to the list of recommended products.

[1018] Step 8:

[1019] The server transmits the generated redecorating ideas and the list of recommended products to the user terminal. The input data are the adjusted redecorating ideas and the list of recommended products, and the display data transmitted to the user terminal is obtained as the output.

[1020] Step 9:

[1021] The user device receives the sent redecorating ideas and recommended product list and displays them on the screen. The user can check these contents and select the products they want to purchase. The input data is the display data sent from the server, and is displayed on the user's screen as output.

[1022] Step 10:

[1023] The user selects the product they wish to purchase from the list of recommended products displayed on their device. Clicking on the purchase link will take them to the sales page for that product, where they can complete the purchase procedure. The input data is the selected recommended product link, and the output is the display of the sales page and the completion of the purchase procedure.

[1024] (Application example 2)

[1025] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1026] Conventional interior redecorating systems have the problem that it is difficult to select optimal products when users select and purchase interior products in a physical store because they lack suggestions based on actual layout images and emotions.In addition, there is also the problem of low user satisfaction because suggestions are made without considering the user's feelings about the proposed redecorating ideas.

[1027] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for uploading image data of a room taken by a user terminal to the server, means for analyzing the image data by the server and extracting information about the furniture and layout in the room, means for generating redecorating suggestions and creating a list of recommended products based on the extracted information, means for analyzing emotions from the user's facial expressions and voice using emotion analysis means and optimizing the redecorating suggestions based on the analyzed emotions, means for linking to a sales platform that provides the recommended products, means for transmitting the proposed redecorating suggestions and the list of recommended products to the user terminal, and means for displaying the proposed redecorating suggestions in a physical store and supporting the purchase of related products. This allows users to visually check the actual room layout even in a physical store and select and purchase the most suitable interior products based on their emotions.

[1028] A "user terminal" is a computing device for taking images and transmitting data, and includes smartphones, tablets, computers, etc.

[1029] A "server" is a central processing unit that receives, analyzes, and processes data sent from user terminals.

[1030] "Image data" refers to a photograph of a room taken and uploaded from a user terminal, based on which information about the furniture and layout within the room is extracted.

[1031] "Information about furniture and layout in the room" refers to data such as the position, arrangement, and shape of furniture and interior items arranged in the room.

[1032] A "redecoration suggestion" is a new interior layout proposal generated based on the room data analyzed by the server and the user's emotional information.

[1033] The "list of recommended products" is a list of interior products and furniture recommended to the user, selected by the server based on the redecorating suggestions.

[1034] "Emotion analysis means" is a technology for analyzing a user's facial expressions and voice to recognize their emotional state.

[1035] "Sales Platform" means an online or offline marketplace that offers recommended products and provides links to purchase the products.

[1036] The "means for transmitting to the user terminal" refers to a communication technology for transmitting the redecorating suggestions and recommended product list generated by the server to the user terminal.

[1037] A "physical store" is a physical store that a user can actually visit and where they purchase interior goods.

[1038] "Analyzing emotions and optimizing redecorating suggestions based on that" is a process of adjusting the suggestions based on the user's emotional data to increase user satisfaction.

[1039] "Supporting the purchase of related products" refers to providing information and systems that allow users to easily consider and purchase suggested products in physical stores.

[1040] The system for implementing this invention includes a user terminal, a server, emotion analysis means, and a link to a sales platform. A specific method for allowing a user to select interior products, redecorate, and increase satisfaction will be described below.

[1041] System program configuration

[1042] 1. User Device:

[1043] A smartphone is used as a user terminal. The user takes pictures of the room using the smartphone and uploads them to the server via the application.

[1044] 2. Server:

[1045] The server receives image data sent from the user's device and uses image analysis software (e.g., OpenCV) to extract information about the furniture and layout of the room. Based on the analysis results, the AI ​​model generates multiple redecorating suggestions.

[1046] 3. Emotion analysis means:

[1047] The emotion analysis method uses a smart device equipped with a camera and microphone. This collects the facial expressions and voices the user makes in response to the suggestions, which are then analyzed using emotion analysis software (e.g., the EmotionRecognition library). Based on the results of this analysis, the server generates optimal redecorating suggestions that reflect the user's interests.

[1048] 4. Create and send a list of recommended products:

[1049] The system selects recommended products related to the proposed redecorating plan, generates a list with links to purchase these products, and sends the list and suggestions to the user's device so that the user can review them.

[1050] Details of data processing and calculation

[1051] The server processes the data in the following steps:

[1052] 1. Analyze the image data received from the user device using image analysis technology such as OpenCV.

[1053] 2. Extract the position information of furniture, walls, floors, windows, etc. in the room from the analysis results.

[1054] 3. Use an AI model to generate multiple redecorating suggestions based on this information.

[1055] 4. The EmotionRecognition library analyzes the user's facial expressions and voice data in response to the proposed content to determine their emotional state.

[1056] 5. The server generates further optimized redecorating suggestions that reflect the emotional data.

[1057] 6. Create a list of recommended products and generate links to the corresponding sales platforms.

[1058] 7. Send optimized redecorating suggestions and recommended product lists to the user device.

[1059] Specific examples

[1060] For example, in a physical store, a user can input a prompt such as, "I'm thinking of redecorating my living room. Please suggest new furniture layouts and recommend products based on the photos below." Then, by inputting a response to the proposed content, such as, "The user has a happy expression on their face in response to the proposal. Please provide a more optimized layout proposal," an emotion analysis is performed based on the happy expression, and optimal proposals that reflect this are presented.

[1061] In this way, users can visually check the actual room layout even in a physical store, and select and purchase the most suitable interior goods based on their emotions.

[1062] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1063] Step 1:

[1064] The user takes a picture of the room on their device and uploads it to the server via the application. The user uses the camera on their smartphone to take a picture of the entire room. After taking the picture, they select the image data in the application and press the upload button. This sends the image data to the server.

[1065] Input: A picture of the room taken by the user with their smartphone

[1066] Output: Image data uploaded to the server

[1067] Step 2:

[1068] The server analyzes the received image data and extracts positional information for the furniture, walls, floors, windows, etc. in the room. The server uses image analysis software (e.g., OpenCV) to detect objects in the image and extract their respective positions and shapes as data.

[1069] Input: Uploaded image data

[1070] Output: Information about furniture and layout in the room

[1071] Step 3:

[1072] The server uses the extracted information to generate multiple redecorating suggestions using an AI model, and then uses machine learning algorithms to generate several new furniture placement options.

[1073] Input: Information about the furniture and layout in the room

[1074] Output: Multiple redecorating suggestions

[1075] Step 4:

[1076] The camera and microphone on the user's device are used to collect the facial expressions and voices of the user in response to the suggestions. While the user is checking the suggestions, the camera on the smartphone or HMD captures the facial expressions and the microphone records the voice.

[1077] Input: User's facial expression data and voice data

[1078] Output: Captured user emotion data

[1079] Step 5:

[1080] The server uses emotion analysis software (e.g., EmotionRecognition library) to analyze the captured user emotion data and optimize the redecorating suggestions based on it. The server analyzes the user emotion data and tailors the suggestions to those that show interest.

[1081] Input: Captured user emotion data

[1082] Output: Optimized redecorating suggestions

[1083] Step 6:

[1084] The server creates a list of recommended products and generates links to corresponding sales platforms. Based on the redecorating proposal, the server selects appropriate products from online marketplaces and compiles a list of their purchase links.

[1085] Input: Optimized redecorating suggestions

[1086] Output: A list of recommended products and a link to purchase them

[1087] Step 7:

[1088] The server sends the optimized redecorating suggestions and recommended product list to the user's device, which displays the received data so that the user can check the redecorating suggestions and product details.

[1089] Input: List of recommended products and purchase links

[1090] Output: Redecorating suggestions and recommended product list displayed on the user's device

[1091] Step 8:

[1092] The user selects the product they wish to purchase from the recommended product list and clicks the link to complete the purchase process. When the user clicks the link, they are redirected to the product purchase page on the sales platform, where they can complete the purchase process.

[1093] Input: User selected product link

[1094] Output: Transition to the purchase page on the sales platform and completion of purchase

[1095] This series of processing steps allows users to visually check redecorating suggestions even in a physical store and purchase the most suitable interior products based on their own feelings.

[1096] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1097] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1098] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1099] [Fourth embodiment]

[1100] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1101] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1102] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1103] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1104] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1105] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1106] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1107] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1108] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1109] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1110] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1111] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1112] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1113] The present invention provides a system that allows users to easily rearrange the interior of their own rooms. This system is implemented based on the following specific operational flow, which operates via a user terminal, a server, and the Internet.

[1114] First, the user takes a photo of their own room. Using the user's device (e.g., a smartphone), the user positions the camera so that the entire room they want to redecorate is captured, and takes a photo. After taking the photo, the user uses the app's upload function to send the photo data to the server.

[1115] The server then receives the image data sent by the user and analyzes it to extract important information such as the room's furniture, layout, wall and floor locations, and even the location of windows and doors. This analysis is performed using advanced AI technology.

[1116] Based on the analyzed information, the server generates multiple redecorating ideas, including new furniture arrangements and suggestions for adding new furniture and accessories. For example, the server might suggest moving the sofa to the right wall and placing a modern coffee table in the center of the living room.

[1117] The server then generates a list of recommended products suitable for the user based on the home improvement ideas. This list includes products selected from online marketplaces such as Yahoo! Shopping. For example, the server generates a link to a product on a sales platform that corresponds to the suggested coffee table.

[1118] The server then sends these redecorating ideas and a list of recommended products to the user's device, which receives the data and displays it on the screen in a user-friendly format, including a diagram of the new layout, images of the recommended furniture, and a link to purchase the items.

[1119] Users can select the product they want to purchase from a list of recommended products displayed on their device. When the user clicks on the purchase link, they are redirected to the product page and can complete the purchase process. For example, the user selects a coffee table with a modern design and clicks on the link to complete the purchase process on the Yahoo! Shopping page.

[1120] In this way, the system of the present invention allows users to easily rearrange their interiors and helps them create their ideal rooms. It also provides companies with an effective means of promoting new products.

[1121] The processing flow will be explained below.

[1122] Step 1:

[1123] The user starts a photography app on their smartphone and takes a photo of the room they want to change the interior of.

[1124] Step 2:

[1125] The user selects a photo and clicks the upload button in the app. The user's device sends the captured image data to the server.

[1126] Step 3:

[1127] The server receives the image data. The server receives and stores the image data of the room sent from the user terminal.

[1128] Step 4:

[1129] The server performs image analysis and passes the received image data to the AI ​​platform to extract information about furniture, layout, walls, floors, windows, doors, etc.

[1130] Step 5:

[1131] The server generates redecorating ideas based on the extracted information. The server creates multiple redecorating plans and suggests new layouts and additional furniture that are suitable for the user's room.

[1132] Step 6:

[1133] The server creates a list of recommended products. Based on the generated redecorating ideas, the server searches online marketplaces for recommended products and creates a list.

[1134] Step 7:

[1135] The server sends a list of redecorating ideas and recommended products to the user's device. The server then sends this data to the user's device.

[1136] Step 8:

[1137] The user's device receives the data and displays the information on the display screen, allowing the user to see the new layout proposal and recommended product list on the device screen.

[1138] Step 9:

[1139] The user selects a recommended product and completes the purchase procedure. The user selects the product they wish to purchase from the list of recommended products, clicks on the link to the sales platform, and is taken to the purchase page to complete the purchase procedure.

[1140] Example 1

[1141] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1142] In the past, when users rearranged their room interiors, not only did it take a lot of time and effort to arrange furniture and select new items, but it was also difficult to imagine a specific layout plan. Furthermore, purchasing new furniture required searching blindly, which was not an efficient process. As a result, there was insufficient support for creating the ideal room.

[1143] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1144] In this invention, the server includes means for uploading image data of a room taken by a user terminal to the server, means for the server to analyze the image data and use advanced AI technology to extract the furniture, layout, wall and floor positions, and window and door positions within the room, means for generating redecorating proposals including new furniture arrangements and suggestions for adding new products based on the extracted information and creating a list of recommended products from an online marketplace, means for linking to a sales platform that provides the recommended products, and means for transmitting the proposed redecorating proposals and the list of recommended products to the user terminal, thereby enabling users to easily rearrange their interiors and create their ideal room.

[1145] A "user terminal" is an electronic device used by a user to take photos of a room and upload data, such as a smartphone or tablet.

[1146] A "server" is a computer system that functions as a central control device, receives and analyzes image data sent from user terminals, and generates and sends lists of redecorating suggestions and recommended products.

[1147] "Image data" is photographic data of the room taken by the user terminal, and is a digital image file that is uploaded to the server.

[1148] "Advanced AI technology" refers to technology that uses artificial intelligence, particularly algorithms and models for image analysis, data extraction, and layout proposals.

[1149] "Furniture" and "layout" refer to the items arranged in a room and their arrangement. Specifically, this includes interior items such as sofas, tables, shelves, and their arrangement.

[1150] "Redecoration proposal" refers to a proposal for a new interior layout or product addition to a room, generated by the server. For example, it includes a proposal for rearranging furniture or introducing new items.

[1151] A "recommended product list" is a list of products selected based on the redecorating suggestions and suggested to the user, including products selected from online marketplaces and their purchase links.

[1152] "Online marketplace" refers to a platform for selling goods over the Internet, including, for example, general e-commerce sites.

[1153] "Means of linking to a sales platform" refers to a method of providing a link to the product purchase page of an online marketplace that sells recommended products.

[1154] "Transmission means" refers to a method for sending data from one party to another, and in particular refers to a method for sending a list of redecorating suggestions or recommended products from a server to a user terminal.

[1155] The present invention provides a system that allows users to easily redecorate the interior of their own rooms. This system operates via a user terminal, a server, and the Internet. Specific embodiments are described below.

[1156] First, the user uses a user device (e.g., a smartphone) to take a photo of their room. The user positions the camera so that the entire room they want to redecorate is captured, and takes a photo. After taking the photo, the user uses the upload function in the app on their device to send the photo data to the server.

[1157] The server then receives the photo data sent from the user's device. The server incorporates advanced AI technology, which is used to analyze the image data. The analysis extracts important information, such as the room's furniture, layout, wall and floor positions, and even the location of windows and doors. This allows the current state of the room to be accurately determined.

[1158] The server then generates multiple redecorating ideas based on the extracted information. This process uses a generative AI model. Ideas include new furniture arrangements and suggestions for adding new furniture and accessories. For example, the server might automatically generate a layout that moves the sofa along the right wall and places a modern coffee table in the center of the living room.

[1159] Furthermore, the server generates a list of recommended products suitable for the user based on these home improvement ideas. This list is composed of products selected from online marketplaces (e.g., general e-commerce sites). The server generates links to furniture and miscellaneous items suitable for each suggestion to complete the list of recommended products.

[1160] The server then sends the list of redecorating ideas and recommended products to the user's device, which receives the data and displays it on a screen in an easy-to-understand format. The displayed content includes a diagram of the new layout plan, images of the recommended furniture, and a link to purchase the items.

[1161] The user can select the product they want to purchase from the list of recommended products displayed on the device. When the user clicks on the purchase link, they are redirected to the product page and can complete the purchase. For example, the user selects a coffee table with a modern design and clicks on the link to complete the purchase on the e-commerce site page.

[1162] An example prompt is:

[1163] "Users take photos of their rooms with their smartphones and upload them to the server. The server uses AI technology to analyze the images and automatically recognize furniture placement and layout. The server then generates multiple redecorating ideas and creates a list with links to suitable products. This list is then sent to the user's device, where the user can view and purchase the products."

[1164] "A user takes a picture of their living room with their smartphone and uploads it to the server through the app. The server analyzes the image and recognizes the furniture layout. The server then generates an idea to move the sofa along the right wall and place a modern coffee table in the center. The server creates a list of coffee tables, including links to popular e-commerce sites, and sends it to the user. The user reviews the list and purchases the coffee table."

[1165] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1166] Step 1:

[1167] The user takes a photo of the room.

[1168] Input: The user's room status

[1169] How it works: A user uses their smartphone to take a photo of the room they want to redecorate.

[1170] Output: Image data of the photographed room

[1171] Step 2:

[1172] The user uploads image data from the terminal to the server.

[1173] Input: Image data of the room

[1174] Operation: The user uses the application on the device to tap the upload button to send the captured image data to the server.

[1175] Output: Image data uploaded to the server

[1176] Step 3:

[1177] The server receives and analyzes the image data.

[1178] Input: Image data uploaded by the user

[1179] How it works: The server uses advanced AI technology to analyze the received image data and extract information such as the furniture in the room, layout, the position of walls and floors, and the position of windows and doors.

[1180] Output: Information about furniture and layout in the room

[1181] Step 4:

[1182] The server generates redecorating ideas based on the extracted information.

[1183] Input: Information about the furniture and layout in the room

[1184] How it works: The server uses a generative AI model to generate multiple redecorating ideas, including new furniture arrangements and suggestions for adding new products.

[1185] Output: Redecorating ideas

[1186] Step 5:

[1187] The server creates a list of recommended products based on the redecorating ideas.

[1188] Enter: Redecorating Ideas

[1189] How it works: The server searches online marketplaces for suitable product recommendations and creates a list by generating product links for each suggestion.

[1190] Output: A list of recommended products

[1191] Step 6:

[1192] The server sends a list of redecorating ideas and recommended products to the user's terminal.

[1193] Input: A list of home improvement ideas and recommended products

[1194] Operation: The server sends the generated redecorating ideas and a list of recommended products to the user's device.

[1195] Output: A list of home improvement ideas and recommended products sent to the user's device

[1196] Step 7:

[1197] The user terminal displays the received data.

[1198] Input: A list of redecorating ideas and recommended products sent from the server

[1199] Operation: The user's device displays the received data on the screen and provides the user with new layout proposals, images of recommended products, and purchase links.

[1200] Output: A list of redecorating ideas and recommended products

[1201] Step 8:

[1202] The user selects and purchases a product from a list of recommended products.

[1203] Input: List of displayed recommended products

[1204] How it works: A user selects the product they want to purchase from a list of recommended products and clicks a link to go to the product page on the online marketplace and complete the purchase.

[1205] Output: Purchased items and completed checkout

[1206] (Application example 1)

[1207] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1208] Conventional interior redecorating systems lacked a means for users to virtually visualize the new layout, making it difficult to visualize how the room would change. They also lacked an environment in which users could smoothly purchase recommended products. Furthermore, they lacked a means to compare the real room with the virtual layout in real time, leaving them with insufficient functionality to support users in making redecorating decisions.

[1209] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1210] In this invention, the server includes means for uploading image data of a room taken by a user terminal to the server, means for analyzing the image data by the server and extracting information about the furniture and layout in the room, means for generating redecorating proposals and creating a list of recommended products based on the extracted information, means for linking to a sales platform that provides the recommended products, means for transmitting the proposed redecorating proposals and the list of recommended products to the user terminal, and means for the user to confirm and purchase the proposed redecorating proposals and recommended products displayed on the user terminal in a virtual store. This makes it easier for the user to virtually visualize a new layout and to easily purchase recommended products, thereby making it possible to smoothly proceed with redecorating decisions.

[1211] A "user terminal" is a computer device, smartphone, tablet, or other portable electronic device used by a user.

[1212] "Image data" refers to photos of the room taken by the user terminal or image files.

[1213] A "server" is a computer system that processes requests and provides data from multiple client devices within a network.

[1214] "Analysis" refers to the process performed by the server to extract and analyze necessary information from image data.

[1215] "Furniture" refers to interior items such as tables, chairs, and sofas that are placed in a room.

[1216] "Layout" refers to the arrangement and design of furniture and equipment within a room.

[1217] "Suggestion" refers to a recommendation to the user of a new layout and arrangement for redecorating.

[1218] "Recommended products" are interior products that are candidates for purchase and are recommended by the server based on the new layout plan.

[1219] "Sales Platform" refers to a website or service that allows you to sell products online.

[1220] "Link" means a hypertext link that navigates from a user terminal to the sales platform.

[1221] A "virtual store" refers to an online shopping space that uses virtual reality and augmented reality.

[1222] "Virtual visualization" is a process that allows users to visually see in real time how a new layout proposal would look in a real room in a virtual environment.

[1223] The system of the present invention is designed to allow users to easily rearrange their rooms, virtually visualize new layouts, and purchase recommended products.

[1224] Hardware and software used

[1225] User devices: Smartphones (e.g., iPhones, Android devices), tablets

[1226] Server: A powerful computer system (e.g., an EC2 instance from Amazon Web Services)

[1227] Image analysis: TensorFlow, Keras

[1228] Database: MySQL

[1229] Server framework: Django (Python)

[1230] Frontend: React Native

[1231] API: Online marketplace API (e.g. Yahoo! Shopping API)

[1232] System processing flow

[1233] Uploading image data from a user device

[1234] Users can take photos of their rooms using their smartphones or tablets and upload the image data to the server using a dedicated application. By taking a photo of the entire room using the camera on the user's device, all the necessary information is captured.

[1235] Image data analysis by the server

[1236] The server analyzes the received image data using TensorFlow and Keras. Image analysis extracts information about the layout of furniture in the room and the positions of windows, doors, etc. This analysis makes it possible to grasp the components of the room in detail.

[1237] Generate redecorating suggestions

[1238] Based on the analyzed information, the server generates multiple redecorating ideas, including new arrangements for existing furniture and suggestions for adding new interior items.

[1239] Creating and providing a list of recommended products

[1240] Based on the generated redecorating ideas, the server uses an online marketplace API to create a recommended product list, which includes links to products that correspond to the suggested furniture and interior items.

[1241] Check and purchase in the virtual store

[1242] The proposed redecorating plan and the list of recommended products are sent to the user's device, where the user can use the application installed on the device to virtually visualize the new layout and purchase the recommended products.

[1243] Specific examples

[1244] Users take photos of their living room with their smartphones and upload them to the application. The server analyzes the images using TensorFlow and Keras to identify the positions of sofas and tables, as well as the layout of walls and windows. Based on this information, the server generates modern interior design proposals and lists recommended products (new sofas, tables, and accessories) retrieved from online marketplaces. Users can then virtually visualize the new layout through the application and purchase the recommended products on the spot.

[1245] Prompt Sentence Examples

[1246] "Room Layout Recommendations"

[1247] User Scenario:

[1248] The user uploads a photo of their living room to the app. The AI ​​nestles the photo to suggest new furniture arrangements and recommends matching products available for purchase.

[1249] Prompt Input:

[1250] Room Photo: [Upload Photo]

[1251] Current Furniture Layout: [Provide AI analysis]

[1252] Suggested Furniture Rearrangement: [Generate layout suggestions]

[1253] Product Recommendations: [Fetch from Online Marketplace API]

[1254] Desired Output:

[1255] 1. New furniture layout suggestions based on the uploaded photo.

[1256] 2. List of recommended products matching the new layout.

[1257] 3. Direct links to purchase recommended products.

[1258] Action:

[1259] Process the uploaded photo to analyze the current furniture layout and generate new arrangement suggestions. Fetch and display recommended products with purchase links."

[1260] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1261] Step 1:

[1262] Taking and uploading images

[1263] The user takes a photo of the room using a smartphone or tablet. The user opens the application and selects an image of the room or takes a new one and uploads it. The input is the image data of the room, and the output is the completion of uploading the image data to the server.

[1264] Step 2:

[1265] Receiving image data

[1266] The server receives image data sent from the user terminal. The input is the image data uploaded by the user, and the output is the image data saved on the server. The server saves the image data in a specified directory.

[1267] Step 3:

[1268] Image data analysis

[1269] The server analyzes the stored image data using TensorFlow and Keras. The input is the stored image data, and the output is the analysis results. Through the analysis, information such as the position of furniture in the room, walls, windows, and doors is extracted. Specifically, the image is input into a model, and data processing is performed to identify the position of furniture and layout information.

[1270] Step 4:

[1271] Generate redecorating suggestions

[1272] The server generates multiple redecorating plans based on the analysis results. The input is the analysis results, and the output is multiple redecorating plans. The server uses a generative AI model to suggest new furniture placements and interior items to add. Each plan includes the specific placement method and the reasons for it.

[1273] Step 5:

[1274] Creating a list of recommended products

[1275] The server uses an online marketplace API to create a list of recommended products based on the redecorating plan. The input is the redecorating plan, and the output is a list of recommended products. The server calls the API to obtain product information (price, links, images, etc.) corresponding to the suggested furniture.

[1276] Step 6:

[1277] Sending data

[1278] The server sends the generated list of redecorating ideas and recommended products to the user's device. The input is the list of redecorating ideas and recommended products, and the output is the completion of sending the data to the user's device. The server then sends data to the user's device to display this information.

[1279] Step 7:

[1280] Virtual store visualization

[1281] The user uses an application installed on their device to virtually visualize the new layout. The input is a list of redecorating ideas and recommended products sent from the server, and the output is the user's reaction to the visualized layout and their selection. The user can view the proposed layout using 3D models and AR functions to visualize the changes concretely.

[1282] Step 8:

[1283] Purchase recommended products

[1284] The user clicks on a link in the application to purchase the recommended product in the virtual store. The input is the link to the recommended product, and the output is the completion of the product purchase procedure. The user opens the link on their device and proceeds with the purchase procedure on the virtual store's details page.

[1285] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1286] The present invention is a system that allows users to easily rearrange the interior of their own rooms. This system is realized based on the following specific operational flow, which operates via a user terminal, a server, an emotion engine, and the Internet.

[1287] First, the user takes a photo of their own room. Using a user device (such as a smartphone), the user positions the camera so that the entire room in which they wish to change the interior is captured, and takes a photo. After taking the photo, the user uses the upload function in the app to send the photo data to the server.

[1288] The server then receives the image data sent by the user and analyzes it to extract important information such as the room's furniture, layout, wall and floor locations, and even the location of windows and doors. This analysis is performed using advanced AI technology.

[1289] Based on the analyzed information, the server generates multiple redecorating ideas, including new furniture arrangements and suggestions for adding new furniture and accessories. For example, the server might suggest moving the sofa to the right wall and placing a modern coffee table in the center of the living room.

[1290] Next, the emotion engine analyzes the user's facial expressions and voice to recognize their emotions. The emotion engine analyzes how the user feels about the proposed redecorating ideas and adjusts the ideas accordingly. For example, the server may make suggestions specific to the redecorating ideas the user is interested in, providing layout proposals that match the user's preferences.

[1291] After using the emotion engine to select the best idea that matches the user's emotions, the server creates a list of recommended products. This list includes products selected from online marketplaces such as Yahoo! Shopping. For example, the server generates a link to a product on a sales platform that corresponds to the suggested coffee table. This list is customized based on the user's emotions, so it recommends products that are more suitable for the user.

[1292] The server then sends these redecorating ideas and a list of recommended products to the user's device, which receives the data and displays it on the screen in a user-friendly format, including a diagram of the new layout, images of the recommended furniture, and a link to purchase the items.

[1293] Users can select the product they want to purchase from a list of recommended products displayed on their device. When the user clicks on the purchase link, they are redirected to the product page and can complete the purchase process. For example, the user selects a coffee table with a modern design and clicks on the link to complete the purchase process on the Yahoo! Shopping page.

[1294] In this way, the system of the present invention allows users to easily rearrange their interiors and support the creation of their ideal room. Furthermore, by taking the user's emotions into consideration, more personalized suggestions become possible. Furthermore, it can provide an effective means for companies to promote new products.

[1295] The processing flow will be explained below.

[1296] Step 1:

[1297] The user launches the camera app on their smartphone and takes a photo of the room they want to redecorate.

[1298] Step 2:

[1299] The user sends the photos they have taken to the server using the upload function within the app.

[1300] Step 3:

[1301] The server receives and stores the image data sent from the user terminal.

[1302] Step 4:

[1303] The server uses image analysis algorithms to analyze the image data and identify the furniture, layout, walls, floors, windows, doors, etc. within the room.

[1304] Step 5:

[1305] Based on the extracted information, the server generates multiple redecorating ideas, including suggestions for new furniture arrangements and additional furniture.

[1306] Step 6:

[1307] The emotion engine collects the user's facial and voice data and analyzes their emotions. For example, if the user smiles into the smartphone camera, the emotion engine will detect a positive emotion.

[1308] Step 7:

[1309] The emotion engine evaluates the redecorating ideas based on the user's emotions and adjusts the suggestions according to the user's emotions. The emotion engine reevaluates the suggestions and selects ideas that are likely to satisfy the user.

[1310] Step 8:

[1311] The server generates a customized list of recommended products based on the results of the emotion engine, which includes products that match the user's emotions.

[1312] Step 9:

[1313] The server sends a list of redecorating ideas and recommended products to the user's terminal.

[1314] Step 10:

[1315] The user's device receives the data from the server and displays it in a user-friendly format, including diagrams of new layout plans, images of recommended products, and links to purchase them.

[1316] Step 11:

[1317] Users select the product they want to purchase from the recommended product list and click the link to proceed to the sales platform and complete the purchase. For example, a user can purchase a modern coffee table on Yahoo! Shopping.

[1318] In this way, the system of the present invention allows users to easily rearrange their interiors and supports them in making optimal suggestions and purchasing products that suit their own emotions.

[1319] Example 2

[1320] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1321] Conventional interior redecorating systems simply analyze the layout information of a user's room and make suggestions. As a result, the suggested redecorating ideas often do not match the user's preferences, and suggestions do not take the user's emotions into consideration. Furthermore, the convenience of purchasing the recommended products is insufficient.

[1322] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1323] In this invention, the server includes means for uploading image data of a room taken by a user terminal to the server, means for the server to analyze the image data and extract information about the furniture and layout of the room, means for generating redecorating proposals and creating a list of recommended products based on the extracted information, means for linking to a sales platform that provides the recommended products, means for transmitting the proposed redecorating proposals and the list of recommended products to the user terminal, means for analyzing the user's facial expressions and voice and recognizing their emotions, and means for adjusting the redecorating proposals based on the recognized emotions. This enables more personalized redecorating proposals based on the user's emotions, thereby increasing user satisfaction. It also simplifies the process of purchasing recommended products, improving the user experience.

[1324] A "user terminal" is an information processing device that can be operated by a user and that can input and display data.

[1325] A "server" is a central information processing device that is connected to user terminals via a network and performs data recording, management, analysis, and the like.

[1326] "Image data" is digital data containing visual information captured and recorded by a user terminal.

[1327] "Upload" refers to the operation and process of sending image data from a user terminal to a server.

[1328] "Analysis" is the process of automatically processing the information contained in the image data and extracting specific information (e.g., furniture or layout).

[1329] "Furniture" refers to items such as desks, chairs, sofas, and shelves that are placed in a room.

[1330] "Layout" refers to the arrangement and placement of various furniture and decorative items in a room.

[1331] "Redecoration proposals" involve providing specific proposals and ideas for changing the existing interior layout.

[1332] "Recommended products" are commercial items related to the redecorating suggestions and recommended to the user.

[1333] A "sales platform" is an online marketplace or shopping site that sells recommended products.

[1334] "Linking" refers to the process of making the relevant product page on the sales platform accessible from the display screen of the user's terminal.

[1335] "Facial expressions" refer to emotions and reactions expressed through changes in the user's face.

[1336] "Speech" refers to information produced by the user's voice or words.

[1337] "Emotion recognition" means analyzing facial expressions and voice data to determine the user's psychological state and reactions.

[1338] "Adjusting" means optimizing redecorating suggestions and product recommendations based on perceived emotions.

[1339] This invention is a system that allows users to easily redecorate the interior of their own room. This system operates via a user terminal, a server, an emotion engine, and the Internet. The following describes in detail the mode for carrying out the invention.

[1340] User device operation

[1341] Users use their smartphone or other device to take photos of their room so that the entire room is visible. Taking photos from multiple angles and under appropriate lighting improves the accuracy of analysis. The photos are then sent to the server using the upload function of the dedicated app. The data sent is encrypted to protect privacy.

[1342] Server Processing

[1343] The server receives image data sent from the user's device. It decodes the image data and analyzes it using AI technology (e.g., TensorFlow or OpenCV). During the analysis process, the furniture in the room, its layout, the location of walls and floors, and the location of windows and doors are identified. For example, the contours and positions of furniture such as sofas and tables can be identified and mapped.

[1344] Once the analysis is complete, the server generates multiple redecorating ideas using automated design algorithms (e.g., generative design). These ideas include new furniture arrangements and suggestions for adding new furniture and accessories. For example, a layout suggestion might be to move the sofa along the right wall and place a modern coffee table in the center of the living room.

[1345] Emotion engine processing

[1346] The emotion engine analyzes the user's facial expressions and voice to recognize their emotions. It uses facial recognition software (e.g., Microsoft Azure Face API) and voice analysis software (e.g., Google Cloud Speech-to-Text). Based on the analysis results, it evaluates how the user feels about the proposed redecorating ideas. This allows it to make adjustments based on the ideas that the user likes.

[1347] Product recommendations and delivery

[1348] The server creates a list of recommended products based on the results of the emotion engine. This list is generated using APIs of various online marketplaces. The list includes links to purchase suggested new furniture and accessories. For example, a link to a sales page for a modern coffee table is generated.

[1349] The generated list of redecorating ideas and recommended products is sent to the user's device, which receives it and displays it on the screen. The displayed content includes a diagram of the new layout plan, images of the recommended products, and a link to purchase them.

[1350] Product purchase

[1351] The user selects the product they want to purchase from the list of recommended products displayed on their device. Clicking the purchase link takes them to the sales page for that product and allows them to complete the purchase process. For example, the user selects a coffee table with a modern design and clicks the link to complete the purchase process.

[1352] Specific examples

[1353] 1. Examples

[1354] A user wants to redecorate their living room and takes a photo of the room with their smartphone.

[1355] Upload the photo to the server via the app.

[1356] The server analyzes the image data and generates multiple layout proposals (e.g., placing the sofa against the wall and a new coffee table in the center).

[1357] The emotion engine recognizes happy reactions from the user's facial expressions.

[1358] The server selects and proposes the optimal layout plan based on the emotions.

[1359] Provides a list of recommended coffee tables with links to online marketplaces.

[1360] 2. Prompt sentence for generative AI model

[1361] I'd like to redecorate my living room, so I'll upload a photo of the entire room. Please analyze it and suggest new furniture arrangements and new furniture and accessories that go with it. Also, if you sense anything from my facial expressions or voice about these suggestions, please further adjust your recommendations based on that. And please provide me with the adjusted product list.

[1362] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1363] Step 1:

[1364] Users take photos of their rooms with their smartphones (user devices). They take photos under appropriate lighting from multiple angles so that the entire room is visible, and then upload the photos using a dedicated app. At this time, the input data is image data of the room, and encrypted image data is generated as output and sent to the server.

[1365] Step 2:

[1366] The server receives encrypted image data sent from the user terminal. It decodes the received image data and prepares it for analysis. The input data is encrypted image data, and the output data is image data that can be analyzed.

[1367] Step 3:

[1368] The server analyzes the received image data using AI technology (e.g., TensorFlow or OpenCV). During the analysis process, the furniture in the room, the layout, the position of the walls and floors, and even the position of the windows and doors are identified. The input data is analyzable image data, and the output generates room attribute data (furniture position, layout information, etc.).

[1369] Step 4:

[1370] The server generates multiple redecorating ideas based on the analysis results. Using an automated design algorithm (generative design), it makes suggestions for furniture rearrangement and adding new furniture and accessories. The input data is the room's attribute data, and the generated redecorating ideas are obtained as output. For example, it generates a suggestion to move the sofa along the right wall and place a modern coffee table in the center of the living room.

[1371] Step 5:

[1372] The emotion engine analyzes the user's facial expressions and voice to recognize their emotions. Using facial recognition software and voice analysis software, it evaluates the user's emotional response to the proposed redecorating ideas. The input data is the user's facial expressions and voice data, and the output is recognized emotion data.

[1373] Step 6:

[1374] The server adjusts the redecorating ideas based on the emotion analysis results and re-suggests ideas based on the ones that the user finds favorable. The input data are the recognized emotion data and the initial redecorating ideas, and the output is the adjusted redecorating ideas.

[1375] Step 7:

[1376] The server creates a list of recommended products related to the tailored redecorating plan. It uses the sales platform's API to obtain purchase links for the relevant products. The input data is the tailored redecorating plan, and the output is a list of recommended products. For example, a sales page link for the proposed coffee table is added to the list of recommended products.

[1377] Step 8:

[1378] The server transmits the generated redecorating ideas and the list of recommended products to the user terminal. The input data are the adjusted redecorating ideas and the list of recommended products, and the display data transmitted to the user terminal is obtained as the output.

[1379] Step 9:

[1380] The user device receives the sent redecorating ideas and recommended product list and displays them on the screen. The user can check these contents and select the products they want to purchase. The input data is the display data sent from the server, and is displayed on the user's screen as output.

[1381] Step 10:

[1382] The user selects the product they wish to purchase from the list of recommended products displayed on their device. Clicking on the purchase link will take them to the sales page for that product, where they can complete the purchase procedure. The input data is the selected recommended product link, and the output is the display of the sales page and the completion of the purchase procedure.

[1383] (Application example 2)

[1384] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1385] Conventional interior redecorating systems have the problem that it is difficult to select optimal products when users select and purchase interior products in a physical store because they lack suggestions based on actual layout images and emotions.In addition, there is also the problem of low user satisfaction because suggestions are made without considering the user's feelings about the proposed redecorating ideas.

[1386] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for uploading image data of a room taken by a user terminal to the server, means for analyzing the image data by the server and extracting information about the furniture and layout in the room, means for generating redecorating suggestions and creating a list of recommended products based on the extracted information, means for analyzing emotions from the user's facial expressions and voice using emotion analysis means and optimizing the redecorating suggestions based on the analyzed emotions, means for linking to a sales platform that provides the recommended products, means for transmitting the proposed redecorating suggestions and the list of recommended products to the user terminal, and means for displaying the proposed redecorating suggestions in a physical store and supporting the purchase of related products. This allows users to visually check the actual room layout even in a physical store and select and purchase the most suitable interior products based on their emotions.

[1387] A "user terminal" is a computing device for taking images and transmitting data, and includes smartphones, tablets, computers, etc.

[1388] A "server" is a central processing unit that receives, analyzes, and processes data sent from user terminals.

[1389] "Image data" refers to a photograph of a room taken and uploaded from a user terminal, based on which information about the furniture and layout within the room is extracted.

[1390] "Information about furniture and layout in the room" refers to data such as the position, arrangement, and shape of furniture and interior items arranged in the room.

[1391] A "redecoration suggestion" is a new interior layout proposal generated based on the room data analyzed by the server and the user's emotional information.

[1392] The "list of recommended products" is a list of interior products and furniture recommended to the user, selected by the server based on the redecorating suggestions.

[1393] "Emotion analysis means" is a technology for analyzing a user's facial expressions and voice to recognize their emotional state.

[1394] "Sales Platform" means an online or offline marketplace that offers recommended products and provides links to purchase the products.

[1395] The "means for transmitting to the user terminal" refers to a communication technology for transmitting the redecorating suggestions and recommended product list generated by the server to the user terminal.

[1396] A "physical store" is a physical store that a user can actually visit and where they purchase interior goods.

[1397] "Analyzing emotions and optimizing redecorating suggestions based on that" is a process of adjusting the suggestions based on the user's emotional data to increase user satisfaction.

[1398] "Supporting the purchase of related products" refers to providing information and systems that allow users to easily consider and purchase suggested products in physical stores.

[1399] The system for implementing this invention includes a user terminal, a server, emotion analysis means, and a link to a sales platform. A specific method for allowing a user to select interior products, redecorate, and increase satisfaction will be described below.

[1400] System program configuration

[1401] 1. User Device:

[1402] A smartphone is used as a user terminal. The user takes pictures of the room using the smartphone and uploads them to the server via the application.

[1403] 2. Server:

[1404] The server receives image data sent from the user's device and uses image analysis software (e.g., OpenCV) to extract information about the furniture and layout of the room. Based on the analysis results, the AI ​​model generates multiple redecorating suggestions.

[1405] 3. Emotion analysis means:

[1406] The emotion analysis method uses a smart device equipped with a camera and microphone. This collects the facial expressions and voices the user makes in response to the suggestions, which are then analyzed using emotion analysis software (e.g., the EmotionRecognition library). Based on the results of this analysis, the server generates optimal redecorating suggestions that reflect the user's interests.

[1407] 4. Create and send a list of recommended products:

[1408] The system selects recommended products related to the proposed redecorating plan, generates a list with links to purchase these products, and sends the list and suggestions to the user's device so that the user can review them.

[1409] Details of data processing and calculation

[1410] The server processes the data in the following steps:

[1411] 1. Analyze the image data received from the user device using image analysis technology such as OpenCV.

[1412] 2. Extract the position information of furniture, walls, floors, windows, etc. in the room from the analysis results.

[1413] 3. Use an AI model to generate multiple redecorating suggestions based on this information.

[1414] 4. The EmotionRecognition library analyzes the user's facial expressions and voice data in response to the proposed content to determine their emotional state.

[1415] 5. The server generates further optimized redecorating suggestions that reflect the emotional data.

[1416] 6. Create a list of recommended products and generate links to the corresponding sales platforms.

[1417] 7. Send optimized redecorating suggestions and recommended product lists to the user device.

[1418] Specific examples

[1419] For example, in a physical store, a user can input a prompt such as, "I'm thinking of redecorating my living room. Please suggest new furniture layouts and recommend products based on the photos below." Then, by inputting a response to the proposed content, such as, "The user has a happy expression on their face in response to the proposal. Please provide a more optimized layout proposal," an emotion analysis is performed based on the happy expression, and optimal proposals that reflect this are presented.

[1420] In this way, users can visually check the actual room layout even in a physical store, and select and purchase the most suitable interior goods based on their emotions.

[1421] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1422] Step 1:

[1423] The user takes a picture of the room on their device and uploads it to the server via the application. The user uses the camera on their smartphone to take a picture of the entire room. After taking the picture, they select the image data in the application and press the upload button. This sends the image data to the server.

[1424] Input: A picture of the room taken by the user with their smartphone

[1425] Output: Image data uploaded to the server

[1426] Step 2:

[1427] The server analyzes the received image data and extracts positional information for the furniture, walls, floors, windows, etc. in the room. The server uses image analysis software (e.g., OpenCV) to detect objects in the image and extract their respective positions and shapes as data.

[1428] Input: Uploaded image data

[1429] Output: Information about furniture and layout in the room

[1430] Step 3:

[1431] The server uses the extracted information to generate multiple redecorating suggestions using an AI model, and then uses machine learning algorithms to generate several new furniture placement options.

[1432] Input: Information about the furniture and layout in the room

[1433] Output: Multiple redecorating suggestions

[1434] Step 4:

[1435] The camera and microphone on the user's device are used to collect the facial expressions and voices of the user in response to the suggestions. While the user is checking the suggestions, the camera on the smartphone or HMD captures the facial expressions and the microphone records the voice.

[1436] Input: User's facial expression data and voice data

[1437] Output: Captured user emotion data

[1438] Step 5:

[1439] The server uses emotion analysis software (e.g., EmotionRecognition library) to analyze the captured user emotion data and optimize the redecorating suggestions based on it. The server analyzes the user emotion data and tailors the suggestions to those that show interest.

[1440] Input: Captured user emotion data

[1441] Output: Optimized redecorating suggestions

[1442] Step 6:

[1443] The server creates a list of recommended products and generates links to the corresponding sales platforms. Based on the redecorating proposal, the server selects appropriate products from the online marketplace and compiles a list of their purchase links.

[1444] Input: Optimized redecorating suggestions

[1445] Output: A list of recommended products and a link to purchase them

[1446] Step 7:

[1447] The server sends the optimized redecorating suggestions and recommended product list to the user's device, which displays the received data so that the user can check the redecorating suggestions and product details.

[1448] Input: List of recommended products and purchase links

[1449] Output: Redecorating suggestions and recommended product list displayed on the user's device

[1450] Step 8:

[1451] The user selects the product they wish to purchase from the recommended product list and clicks the link to complete the purchase process. When the user clicks the link, they are redirected to the product purchase page on the sales platform, where they can complete the purchase process.

[1452] Input: User selected product link

[1453] Output: Transition to the purchase page on the sales platform and completion of purchase

[1454] This series of processing steps allows users to visually check redecorating suggestions even in a physical store and purchase the most suitable interior products based on their own feelings.

[1455] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1456] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1457] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1458] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1459] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1460] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1461] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1462] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1463] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1464] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1465] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1466] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1467] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1468] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1469] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1470] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1471] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1472] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1473] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1474] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1475] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1476] The following is further disclosed regarding the above embodiment.

[1477] (Claim 1)

[1478] means for uploading image data of the room captured by the user terminal to a server;

[1479] A server analyzes the image data and extracts information about furniture and layout in the room.

[1480] A means for generating a redecorating proposal and a list of recommended products based on the extracted information;

[1481] a means for linking to a sales platform that provides the recommended products;

[1482] means for transmitting the proposed redecorating plan and a list of recommended products to a user terminal;

[1483] A system including:

[1484] (Claim 2)

[1485] 2. The system according to claim 1, further comprising means for purchasing a product selected by the user from the recommended product list displayed on the user terminal.

[1486] (Claim 3)

[1487] 2. The system of claim 1, wherein the redecorating suggestions generated by the server include a plurality of furniture rearrangement suggestions and additional products.

[1488] "Example 1"

[1489] (Claim 1)

[1490] means for uploading image data of the room captured by the user terminal to a server;

[1491] A means for analyzing the image data by a server and extracting the furniture, layout, positions of walls and floors, and positions of windows and doors in the room using advanced AI technology;

[1492] A means for generating, based on the extracted information, redecorating proposals including new furniture arrangements and suggestions for adding new products, and creating a list of recommended products from an online marketplace;

[1493] a means for linking to a sales platform that provides the recommended products;

[1494] means for transmitting the proposed redecorating plan and a list of recommended products to a user terminal;

[1495] A system including:

[1496] (Claim 2)

[1497] 2. The system according to claim 1, further comprising means for purchasing a product selected by the user from the recommended product list displayed on the user terminal.

[1498] (Claim 3)

[1499] 2. The system of claim 1, wherein the redecorating suggestions generated by the server include a plurality of furniture rearrangement suggestions and additional products.

[1500] "Application Example 1"

[1501] (Claim 1)

[1502] means for uploading image data of the room captured by the user terminal to a server;

[1503] A server analyzes the image data and extracts information about furniture and layout in the room.

[1504] A means for generating a redecorating proposal and a list of recommended products based on the extracted information;

[1505] a means for linking to a sales platform that provides the recommended products;

[1506] means for transmitting the proposed redecorating plan and a list of recommended products to a user terminal;

[1507] A means for a user to confirm and purchase the suggested redecorating ideas and recommended products displayed on the user terminal in the virtual store;

[1508] A system including:

[1509] (Claim 2)

[1510] The system of claim 1 further includes a means for allowing a user to purchase a product selected from the recommended product list displayed on the user terminal, and for virtually visualizing the proposed redecorating ideas.

[1511] (Claim 3)

[1512] The system of claim 1, wherein the redecorating suggestions generated by the server include multiple furniture rearrangement suggestions and additional products, and wherein a means is provided for providing a virtual visualization of the proposed redecorating suggestions using an application installed on the user terminal.

[1513] "Example 2: Combining Emotion Engines"

[1514] (Claim 1)

[1515] means for uploading image data of the room captured by the user terminal to a server;

[1516] A server analyzes the image data and extracts information about furniture and layout in the room.

[1517] A means for generating a redecorating proposal and a list of recommended products based on the extracted information;

[1518] a means for linking to a sales platform that provides the recommended products;

[1519] means for transmitting the proposed redecorating plan and a list of recommended products to a user terminal;

[1520] A means for analyzing a user's facial expressions and voice and recognizing their emotions;

[1521] means for adjusting the redecorating suggestions based on the recognized emotions;

[1522] A system including:

[1523] (Claim 2)

[1524] 2. The system according to claim 1, further comprising means for purchasing a product selected by the user from the recommended product list displayed on the user terminal.

[1525] (Claim 3)

[1526] 2. The system of claim 1, wherein the redecorating suggestions generated by the server include a plurality of furniture rearrangement suggestions and additional products.

[1527] "Application example 2 when combining emotion engines"

[1528] (Claim 1)

[1529] means for uploading image data of the room captured by the user terminal to a server;

[1530] A server analyzes the image data and extracts information about furniture and layout in the room.

[1531] A means for generating a redecorating proposal and a list of recommended products based on the extracted information;

[1532] A means for analyzing emotions from the user's facial expressions and voice using an emotion analysis means and optimizing suggestions for redecorating based on the emotions;

[1533] a means for linking to a sales platform that provides the recommended products;

[1534] means for transmitting the proposed redecorating plan and a list of recommended products to a user terminal;

[1535] A means for displaying the proposed redecorating plan in a physical store and supporting the purchase of related products;

[1536] A system including:

[1537] (Claim 2)

[1538] 2. The system according to claim 1, further comprising means for purchasing a product selected by the user from the recommended product list displayed on the user terminal.

[1539] (Claim 3)

[1540] 2. The system of claim 1, wherein the redecorating suggestions generated by the server include a plurality of furniture rearrangement suggestions and additional products. [Explanation of symbols]

[1541] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. means for uploading image data of the room captured by the user terminal to a server; A server analyzes the image data and extracts information about furniture and layout in the room. A means for generating a redecorating proposal and a list of recommended products based on the extracted information; a means for linking to a sales platform that provides the recommended products; means for transmitting the proposed redecorating plan and a list of recommended products to a user terminal; A system including:

2. 2. The system according to claim 1, further comprising means for purchasing a product selected by the user from the recommended product list displayed on the user terminal.

3. 2. The system of claim 1, wherein the redecorating suggestions generated by the server include a plurality of furniture rearrangement suggestions and additional products.

Images