System

Abstract

A system is provided.SOLUTION: A system comprising: means for uploading a picture of an interior taken by a user; means for automatically erasing unnecessary elements included in the uploaded picture by a AI; means for generating a plurality of design options by an interior design AI based on the erased picture; means for providing the generated plurality of design options to a user device; and means for receiving a feedback of the user and modifying an interior design based on the feedback.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Traditionally, home staging in real estate sales is used to enhance the appeal of vacant properties, but arranging the furniture physically takes time and money. Virtual home staging has been proposed to solve this problem, but current technology requires a lot of trial and error by the user, making it difficult to customize the interior design. Therefore, there is a need for technology that allows users to customize interiors easily and interactively. [Means for solving the problem]

[0005] To solve the above-mentioned problems, the present invention provides the following means: a system including a means for uploading interior photos taken by a user, a means for an AI to automatically delete unnecessary elements contained in the uploaded photos, a means for an interior design AI to generate multiple design options based on the deleted images, a means for providing the generated multiple design options to the user's device, and a means for accepting user feedback and modifying the interior design based on the feedback. Furthermore, by including a means for the interior design AI to regenerate new design options based on user feedback and a means for generating interior designs based on preset styles such as "modern," "classic," and "casual" for preprocessed images, users can easily customize their interior designs and find the optimal design.

[0006] A "user" is someone who uses the system to upload photos of their interior and receive interior design suggestions and modifications.

[0007] A "terminal" is a device operated by a user to send and receive data to and from the system, and refers to devices such as smartphones and tablets.

[0008] The "server" is a central computing device that processes images, generates interior designs, and processes user feedback.

[0009] The "means for uploading photos" refers to a method or device that provides a function for a user to send indoor photos taken by the user to a server.

[0010] "Means for removing unwanted elements" refers to methods or devices that allow AI to automatically remove unwanted objects from uploaded photos.

[0011] "Interior Design AI" refers to an artificial intelligence program that makes interior design suggestions based on photos uploaded by users.

[0012] "Means for generating design options" refers to a method or device that allows the interior design AI to create designs with multiple different styles and layouts and provide the results to the user.

[0013] The term "means for receiving feedback" refers to a method or device for a user to input requests for corrections or changes to design options into the system and transmit those requests to the server.

[0014] The "means for modifying the design" refers to a method or device for regenerating a new interior design based on user feedback and providing it to the user.

[0015] "Preset styles" refer to predefined design themes such as "modern," "classic," and "casual" that are the basis for the interior design AI. [Brief explanation of the drawings]

[0016] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8]FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0017] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0018] First, the terms used in the following description will be explained.

[0019] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0020] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0021] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0022] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0023] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0024] [First embodiment]

[0025] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0026] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0027] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0028] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0029] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0030] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0031] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0032] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0033] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0034] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0035] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0036] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0037] ---

[0038] This invention relates to a virtual home staging system for increasing the desire to purchase vacant properties in real estate sales. This system uses AI to generate virtual interior designs based on photos of the interior taken by the user, and the user can interactively modify and adjust them.

[0039] Overall system overview

[0040] This system mainly consists of a user's device, a server, and an AI module. Users take photos of the interior of a room using a device such as a smartphone or tablet and send them to the server via an application. The server analyzes the received photos, automatically removes unnecessary elements, and then generates multiple design options using the interior design AI module. The generated design options are returned to the user's device, and the user provides feedback based on the displayed designs. The server passes this feedback back to the AI, which then generates a new design and provides it to the user.

[0041] Program processing flow (example)

[0042] 1. User photo uploads

[0043] User: Takes a photo of the living room with a smartphone and uploads it to a dedicated application.

[0044] Device: Sends uploaded photos to the server.

[0045] 2. Image Preprocessing

[0046] Server: Saves the received images in temporary storage.

[0047] Server: Uses AI to automatically detect and remove unwanted elements (e.g., cardboard boxes, old furniture, etc.) from images.

[0048] 3. Generate multiple interior design options

[0049] Server: Passes the preprocessed images to the interior design AI module.

[0050] Server: The module generates multiple design options based on preset styles (e.g., modern, classic, casual).

[0051] Server: Stores the generated design options in temporary storage.

[0052] 4. Providing design options

[0053] Server: Sends the generated design options to the user's device.

[0054] Terminal: Displays the received design options to the user.

[0055] 5. User Feedback

[0056] User: Review the displayed design and enter feedback into the application, such as "I'd like the wall colors to be brighter" or "I'd like the furniture to be a little more modern."

[0057] Terminal: Sends user feedback to the server.

[0058] 6. Interior design revisions

[0059] Server: Receives feedback and uses it to generate new design options for the interior design AI module.

[0060] Server: Saves the modified design options in temporary storage and sends them back to the user's device.

[0061] Terminal: Redisplays new design options to the user.

[0062] 7. Finalize the design

[0063] User: If the new design is satisfactory, the user confirms it as the final design, which is then saved in the system and can be reviewed later.

[0064] This series of processes allows users to easily and interactively experiment with interior designs to find the optimal design. This system is expected to promote real estate sales.

[0065] The processing flow will be explained below.

[0066] ---

[0067] Step 1:

[0068] User: Take a photo of the living room with a smartphone, launch the dedicated application and log in.

[0069] Device: Tap the "Upload Photo" button on the application screen and select the photo you have taken.

[0070] On the device: Send the selected photo to the backend server.

[0071] Step 2:

[0072] Server: Stores the received photo data in temporary storage.

[0073] Server: Passes the saved image data to the AI ​​image analysis module.

[0074] Server: The image analysis module automatically detects unwanted elements in the image (e.g., cardboard boxes, old furniture) and digitally removes them.

[0075] Step 3:

[0076] Server: Obtains clean image data with unnecessary elements removed and passes it to the interior design AI module.

[0077] Server: The interior design AI module generates multiple design options based on preset styles such as "modern," "classic," and "casual."

[0078] Server: Saves each generated design option to temporary storage.

[0079] Step 4:

[0080] Server: Creates a response for sending image data from multiple saved design options to the user's device.

[0081] Terminal: Receives image data of design options sent from the server and displays it to the user.

[0082] Step 5:

[0083] User: Review the design options presented and provide specific feedback if necessary (e.g., "I'd like the walls to be a lighter color" or "I'd like the furniture to be a little more modern").

[0084] Terminal: Makes a request to send the user's input feedback to the server.

[0085] Step 6:

[0086] Server: Analyzes the received feedback and passes it to the interior design AI module.

[0087] Server: The interior design AI module regenerates new design options based on feedback.

[0088] Server: Saves the newly generated design options to temporary storage.

[0089] Step 7:

[0090] Server: Creates a response to send the new design options to the user's device.

[0091] Terminal: Re-display new design options received to the user.

[0092] User: Checks the new design and, if satisfied, provides input to confirm the final design.

[0093] Terminal: Sends final design confirmation information to the server.

[0094] ---

[0095] These are the specific processing steps of the program for the entire system, which allow users to interactively customize the interior design and realize a virtual home staging that reflects their ideals.

[0096] Example 1

[0097] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0098] In real estate sales, there is a demand for methods to visually enhance the appeal of vacant properties. Installing realistic interior designs is costly and time-consuming, and potential buyers have difficulty forming a concrete image of the property. Conventional methods make it difficult for users to select the design they want through trial and error, which discourages them from purchasing.

[0099] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0100] In this invention, the server includes means for uploading interior photos taken by a user, means for automatically erasing unnecessary elements included in the uploaded photos by an artificial intelligence, means for the interior design artificial intelligence to generate multiple design options based on the erased images, means for providing the generated multiple design options to the user's information processing device, means for receiving user feedback and modifying the interior design based on the feedback, means for regenerating new design options based on the user's feedback, means for saving the generated design options in online storage, and means for providing the regenerated design options again to the user's information processing device, thereby enabling the user to interactively experiment with interior designs and find the optimal design.

[0101] "User" refers to a person who uses this system to take photos of an interior and receive interior design suggestions.

[0102] "Server" refers to a computing device that receives users' photos and uses artificial intelligence to process the images and generate designs.

[0103] "Information processing device" refers to an electronic device (e.g., smartphone, tablet, PC) that a user uses to take photos and receive design suggestions from a server.

[0104] "Unwanted elements" refer to objects or backgrounds (e.g., cardboard boxes, old furniture) that are included in the interior photos taken by the user but should be removed when proposing interior designs.

[0105] "Artificial intelligence" refers to technology that automates image processing and design generation using large amounts of data and machine learning algorithms.

[0106] "Interior design artificial intelligence" refers to a specific artificial intelligence model used to automatically generate interior design options from a user's photos.

[0107] "Design options" refers to multiple different interior design proposals generated by the interior design AI.

[0108] "Online storage" refers to a data storage service that allows you to store data via the Internet and access it as needed.

[0109] "Feedback" refers to specific opinions and change requests that users give regarding design options.

[0110] This invention is a virtual home staging system for visually enhancing the appeal of vacant properties in real estate sales. This system uses artificial intelligence (AI) to generate virtual interior designs based on interior photos taken by the user, and allows the user to interactively modify and adjust them.

[0111] Overall system overview

[0112] This system primarily consists of the user's information processing device (smartphone, tablet, PC, etc.), a server, and an interior design AI module. The user uses the information processing device to take photos of the interior and sends them to the server via a dedicated application. The server analyzes the received photos, automatically removes unnecessary elements, and then uses the interior design AI module to generate multiple design options. The generated design options are provided to the user's information processing device, and the user provides feedback based on the displayed designs. The server passes this feedback back to the AI ​​module, which then generates a new design and provides it to the user.

[0113] Hardware and software used

[0114] Hardware:

[0115] Information processing devices (smartphones, tablets, PCs, etc.)

[0116] Server (for data processing and storage)

[0117] software:

[0118] Dedicated application (installed on the user's information processing device)

[0119] Server-side software (online storage such as Amazon S3, Python scripts, OpenCV, TensorFlow-based interior design AI module)

[0120] Specific examples of processing

[0121] Example 1: User uploads a photo

[0122] A user takes a photo of their living room with their smartphone and uploads it to the server using a dedicated application. The user presses the upload button in the application to start sending the photo. The server temporarily stores the received photo in an Amazon S3 bucket.

[0123] Example 2: Removing unnecessary elements

[0124] The server uses the OpenCV library to analyze the stored photos, automatically detecting unwanted elements in the photo, such as cardboard boxes or old furniture, and identifying them through masking. It then applies an inpainting technique to remove the unwanted elements from the image.

[0125] Example 3: Generating interior designs

[0126] The preprocessed images are input into a TensorFlow-based interior design AI module, which generates multiple design options based on pre-trained design patterns (e.g., modern, classic, casual).

[0127] Example prompt sentence:

[0128] For example, if a user "wants a modern style living room," they would enter the following as the prompt:

[0129] For example: "Please add a modern style interior design to the living room in the photo. I would like light wall colors and simple furniture."

[0130] This system allows users to interactively experiment with interior designs to find the best fit, and is expected to help promote real estate sales.

[0131] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0132] Step 1:

[0133] A user takes a photo of their living room with their smartphone and uploads it via a dedicated application. The input is the captured photo data in JPEG format, and the output is an upload request to the server via the application. When the upload button is pressed within the application, the photo data is sent over the Internet to a specified server URL.

[0134] Step 2:

[0135] The device receives the uploaded photo data and temporarily stores it on the device. The input is the JPEG photo data uploaded by the user, and the output is the operation of temporarily storing this data. The stored photo data is ready to be transferred to the server.

[0136] Step 3:

[0137] The server saves the received photo data to storage. An online storage service (e.g., Amazon S3 bucket) is used for saving. The input is the JPEG photo data sent from the device, and the output is the URL of the photo data saved in storage. This URL is used in subsequent processing.

[0138] Step 4:

[0139] The server uses the OpenCV library to erase unnecessary elements in the image. The input is photo data retrieved from storage, and the output is new image data with unnecessary elements erased. Specifically, OpenCV is used to detect cardboard boxes and old furniture in the image and apply inpainting techniques to erase them.

[0140] Step 5:

[0141] The server inputs the preprocessed image data into an interior design AI model. This AI model is built with TensorFlow and uses pre-trained design patterns. The input is image data with unnecessary elements removed, and the output is multiple interior design options. The model analyzes the image and generates design options based on styles such as "modern," "classic," and "casual."

[0142] Step 6:

[0143] The server stores the generated design options in online storage and provides them to the user's information processing device. The input is multiple design options generated by the interior design AI model, and the output is a list of URLs stored in online storage. This list of URLs is then sent to the user's device.

[0144] Step 7:

[0145] The device receives a list of design option URLs and displays each design option in the user interface. The input is the URL list received from the server, and the output is an image of the design option that the user can view. The user can swipe to switch between multiple design options.

[0146] Step 8:

[0147] The user enters feedback on the displayed design options. For example, specific opinions such as "make the wall color lighter" or "rearrange the furniture" are entered into the text input field. The input is the user's feedback text, and the output is a request to send the feedback information to the server.

[0148] Step 9:

[0149] The server analyzes the received feedback and re-inputs it into the interior design AI model. The input is the feedback data received from the user, and the output is the newly generated design options. For example, in response to the feedback "make the wall color lighter," a color conversion process is performed to generate a new interior design.

[0150] Step 10:

[0151] The server saves the newly generated design options in online storage again and provides them to the user's information processing device. The input is the newly generated design options, and the output is a list of URLs saved in online storage. This list of URLs is then sent to the user's device again.

[0152] Step 11:

[0153] The device again receives new design options and displays them in the user interface. The input is the new URL list received from the server, and the output is an image of the latest design options that the user can view. The user again reviews the designs and continues to provide feedback as needed.

[0154] Step 12:

[0155] When the user is satisfied with the design, they press the confirm button in the app to confirm it as the final design. The input is the design option that the user confirms, and the output is the operation of sending the confirmed design to the server and saving it in the database.

[0156] This series of processes allows the user to interactively experiment with interior designs and find the optimal design.

[0157] (Application example 1)

[0158] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0159] In real estate sales, there are limited means to promote vacant properties, making it difficult for potential buyers to visualize the interior of the room. Furthermore, to realize virtual home staging, a flexible system is required that allows users to easily reflect their own preferences and requests, but conventional technology has difficulty meeting this requirement. Furthermore, the same issue exists in virtual stores, as there are not enough functions to design interiors in virtual spaces and save and display the results.

[0160] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0161] In this invention, the server includes a means for uploading interior photos taken by a user, a means for an AI to automatically delete unnecessary elements from the uploaded photos, a means for the interior design AI to generate multiple design options based on the deleted images, a means for accepting user feedback and modifying the interior design based on the feedback, a means for re-receiving user feedback and regenerating the interior design based on the feedback, and a means for saving and displaying the finalized design in the virtual store. This allows users to design interiors in a virtual space based on photos of their actual rooms and to find their ideal interior by repeatedly modifying and regenerating the design. Furthermore, using the finalized design in the virtual store can promote real estate sales and improve the convenience of interior customization in virtual stores.

[0162] "User" refers to the end user who uses the system to upload interior photos and provide feedback on interior design.

[0163] "Uploading means" refers to the interface and communication means for sending photos taken by the user to the server.

[0164] "Means for AI to automatically remove unnecessary elements" refers to a function that uses AI technology to automatically analyze and remove unnecessary objects and items from uploaded photos.

[0165] "Interior Design AI" refers to an artificial intelligence module that generates multiple design options based on a cleaned-up image.

[0166] "Design options" refers to multiple interior design options generated by the interior design AI.

[0167] "User feedback" refers to the user's evaluation of and requests for modifications to the provided design options.

[0168] "Means to modify interior design" refers to the function of reviewing and regenerating interior design based on user feedback.

[0169] A "virtual store" refers to a virtual shopping or exhibition space where products and interiors can be arranged and customized within a virtual space.

[0170] "Final design" refers to the interior design that is finalized as a result of incorporating user feedback.

[0171] "Means for saving and displaying" refers to the function for saving the finalized design within the system and displaying it within the virtual store.

[0172] This invention relates to a virtual home staging system that uses AI to generate a virtual interior design based on indoor photos taken by the user, and allows the user to interactively modify and adjust the design.

[0173] Overall system overview

[0174] The system primarily consists of a user device, a server, and an interior design AI module. Users take photos of the interior using a device such as a smartphone or virtual glasses and send them to the server via an application. The server analyzes the received photos, automatically removes unnecessary elements, and then generates multiple design options using the interior design AI module. The generated design options are returned to the user's device, and the user provides feedback based on the displayed designs. The server passes this feedback back to the AI, which then generates a new design and provides it to the user. The finalized design is saved and displayed in the virtual store.

[0175] Hardware / Software used

[0176] Hardware: Smartphone, virtual glasses, and server

[0177] Software: Python, TensorFlow, OpenCV, Flask

[0178] Program processing details

[0179] 1. User photo uploads

[0180] Users can take photos of the interior of a room using a smartphone app and upload them to a server via a dedicated application.

[0181] 2. Image preprocessing by the server

[0182] The server stores the received images in storage and uses OpenCV to automatically detect and remove unwanted elements (e.g., cardboard boxes, old furniture, etc.) from the photos, generating a clean base image.

[0183] 3. Interior design generation

[0184] The server passes the preprocessed images to an interior design AI module using TensorFlow, which generates multiple design options based on preset styles such as "modern," "classic," and "casual." The generated design options are stored in temporary storage and sent to the user's device.

[0185] 4. User Feedback

[0186] The user reviews the displayed design options and enters specific feedback into the application, such as "I'd like the wall colors to be brighter" or "I'd like the furniture to be a little more modern." This feedback is then sent to the server.

[0187] 5. Interior design revisions

[0188] The server then uses the user's feedback to generate new design options using the interior design AI module, and this process is repeated until the user is satisfied.

[0189] 6. Finalize the design and display it in the virtual store

[0190] Once the user is satisfied with the design, it is saved and displayed in the virtual store, allowing them to visualize how the interior design will look in a real room in a virtual space.

[0191] Specific examples

[0192] For example, if a user submits a photo of their living room and requests a "modern design," the AI ​​will generate multiple design options incorporating elements such as "light-colored walls," "simple modern furniture," and "spacious layout." A specific example of a prompt would be, "I have submitted a photo of my living room. Please generate a modern interior design that suits this room. I would like the furniture to be simple and the colors to be light." By sending this instruction to the server, a design based on the user's requests will be generated.

[0193] As described above, this system allows users to easily experiment with interior designs to find their ideal design. The final design can also be saved and displayed in the virtual store, contributing to promoting real estate sales and improving the user experience.

[0194] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0195] Step 1:

[0196] User-uploaded photos

[0197] Users take photos of the interior of a room using a smartphone app and upload them to the server through a dedicated application. The input is the photo of the room taken with the user's smartphone, and the output is the photo file sent to the server. Specifically, when the "upload photo" button is pressed on the application, the photo is automatically sent to the server.

[0198] Step 2:

[0199] Image preprocessing by the server

[0200] The server stores the received image file in temporary storage and uses the OpenCV library to automatically detect and remove unwanted elements (e.g., cardboard boxes, old furniture, etc.) in the photo. The input is the image file stored on the server, and the output is a clean image with unwanted elements removed. Specifically, the server converts the image to grayscale, performs binary thresholding to mask and remove unwanted objects.

[0201] Step 3:

[0202] Interior Design Generation

[0203] The server sends the preprocessed image to the interior design AI module, which uses TensorFlow to generate multiple design options, such as "modern," "classic," and "casual." The input is clean image data, and the output is multiple interior design options. Specifically, the server passes the image to the AI ​​module, which generates multiple design options based on preset styles.

[0204] Step 4:

[0205] User Feedback

[0206] The user checks the generated design options on a smartphone app and provides feedback such as "change the wall color to a lighter color" or "add more modern furniture." The input is the user's feedback data, and the output is the feedback information sent to the server. Specifically, when the user presses the "Send Feedback" button on the app, the comment is sent to the server.

[0207] Step 5:

[0208] Interior design modifications

[0209] The server receives feedback from the user, and the interior design AI module regenerates new design options based on that feedback. The input is the user's feedback data and preprocessed images, and the output is a new design option modified based on the feedback. Specifically, the server passes the feedback to the AI ​​and saves the regenerated design option in temporary storage.

[0210] Step 6:

[0211] Finalize the design and display it in the virtual store

[0212] The user finally decides on a design that satisfies them, and that design is saved and displayed in the virtual store. The input is the final design data decided by the user, and the output is the final design saved in the virtual store system. Specifically, when the user presses the "Confirm Design" button on the app, the design is saved in the virtual store and becomes available for the user to view in the virtual space.

[0213] This series of processes allows users to find their ideal interior design through trial and error, and ultimately to check the design in a virtual store.

[0214] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0215] ---

[0216] This invention relates to a virtual home staging system for increasing the desire to purchase vacant properties in real estate sales. This system uses AI to generate virtual interior designs based on interior photos taken by the user, which the user can then interactively modify and adjust. In addition, by combining it with an emotion engine that recognizes the user's emotions, the user's emotions can be reflected in the design generation.

[0217] Overall system overview

[0218] This system consists of a user's device, a server, an interior design AI, and an emotion engine. Users take photos of the interior using a device such as a smartphone or tablet and send them to the server via an application. The server analyzes the received photos, automatically removes unnecessary elements, and then uses the interior design AI to generate multiple design options. The generated design options are returned to the user's device, and the user provides feedback based on the displayed designs. The emotion engine then analyzes the user's emotions and reflects the results in the interior design AI to generate new designs.

[0219] Program processing flow (example)

[0220] User-uploaded photos

[0221] User: Take a photo of the living room with a smartphone, launch the dedicated application and log in.

[0222] Device: Tap the "Upload Photo" button on the application screen and select the photo you have taken.

[0223] On the device: Send the selected photo to the backend server.

[0224] Image preprocessing

[0225] Server: Stores the received photo data in temporary storage.

[0226] Server: Passes the saved image data to the AI ​​image analysis module.

[0227] Server: The image analysis module automatically detects unwanted elements in the image (e.g., cardboard boxes, old furniture) and digitally removes them.

[0228] Generate multiple interior design options

[0229] Server: Obtains clean image data with unnecessary elements removed and passes it to the interior design AI module.

[0230] Server: The interior design AI module generates multiple design options based on preset styles such as "modern," "classic," and "casual."

[0231] Server: Saves each generated design option to temporary storage.

[0232] Providing design options

[0233] Server: Creates a response for sending image data from multiple saved design options to the user's device.

[0234] Terminal: Receives image data of design options sent from the server and displays it to the user.

[0235] User Feedback

[0236] User: Review the design options presented and provide specific feedback if necessary (e.g., "I'd like the walls to be a lighter color" or "I'd like the furniture to be a little more modern").

[0237] Terminal: Makes a request to send the user's input feedback to the server.

[0238] Manipulating the Emotion Engine

[0239] Terminal: When inputting feedback, the user's face is photographed with a camera and the image is sent to the emotion engine in real time.

[0240] Emotion engine: Analyzes the user's facial expressions to detect the user's emotional state (e.g., joy, surprise, dissatisfaction).

[0241] Emotion engine: Sends the detected emotional state and feedback content to the server.

[0242] Interior design modifications

[0243] Server: Receives feedback and emotional state, and the interior design AI module regenerates new design options based on that.

[0244] Server: Saves the newly generated design options in temporary storage and sends them back to the user's device.

[0245] Terminal: Re-display new design options received to the user.

[0246] Final design confirmation

[0247] User: If the new design is satisfactory, the user confirms it as the final design, which is then saved in the system and can be reviewed later.

[0248] This series of processes allows users to customize interior designs that take their emotions into account, enabling virtual home staging that is more suited to the user. This system can increase purchasing motivation and maximize the appeal of the property.

[0249] The processing flow will be explained below.

[0250] ---

[0251] Step 1:

[0252] User: Launch the dedicated application on your smartphone and log in.

[0253] User: Takes a photo of the living room with his smartphone.

[0254] Step 2:

[0255] Device: Tap the "Upload Photo" button on the application screen and select the photo you have taken.

[0256] Terminal: Compress the selected photo data and send it to the server.

[0257] Step 3:

[0258] Server: Stores the received photo data in temporary storage.

[0259] Server: Passes the saved image data to the image analysis module.

[0260] Step 4:

[0261] Server: The image analysis module analyzes the image and automatically detects unwanted elements (e.g., old furniture or cardboard boxes).

[0262] Server: Digitally erases detected unwanted elements to generate clean image data.

[0263] Step 5:

[0264] Server: Passes the clean image data to the interior design AI module and instructs it to generate multiple design options.

[0265] Server: The interior design AI module generates multiple design options based on preset styles such as "modern," "classic," and "casual."

[0266] Step 6:

[0267] Server: Stores the generated design options in temporary storage and creates a response to send to the user's device.

[0268] Terminal: Receives image data of design options sent from the server and displays it to the user.

[0269] Step 7:

[0270] User: Review each design option presented and provide feedback (e.g., "I'd like the walls to be a lighter color" or "I'd like the furniture to be a little more modern").

[0271] Device: When inputting feedback, the user's face is photographed with the smartphone camera and the image is sent to the emotion engine in real time.

[0272] Step 8:

[0273] Emotion engine: Analyzes the user's facial expressions to detect their emotional state (e.g., happiness, surprise, dissatisfaction).

[0274] Emotion engine: Sends the detected emotional state and feedback content to the server.

[0275] Step 9:

[0276] Server: Analyzes the received feedback and emotional state and instructs the interior design AI module to regenerate new design options based on that.

[0277] Server: The interior design AI module generates new design options based on user feedback and sentiment.

[0278] Step 10:

[0279] Server: Stores the newly generated design options in temporary storage and creates a response to send to the user's device.

[0280] Terminal: Re-display new design options received to the user.

[0281] Step 11:

[0282] Users: Review the new design options and provide feedback again if needed.

[0283] User: When a satisfactory design is confirmed, it is confirmed as the final design and saved in the system.

[0284] ---

[0285] This series of processes enables customization of interior designs that take emotion analysis into account, realizing optimal virtual home staging for each user. This system makes it possible to propose designs that reflect the user's emotions, maximizing the appeal of the property.

[0286] Example 2

[0287] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0288] Conventional interior design systems have had the drawback of requiring the user to adjust the design themselves, and of making it difficult to generate a design that reflects the user's emotions. This has resulted in the problem that it takes a lot of time and effort for the user to obtain a design that actually satisfies them. The objective of the present invention is to provide a system that can easily generate an interior design that meets the user's needs by analyzing the user's emotional state and reflecting that in the design.

[0289] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for uploading images of an interior taken by a user, means for automatically erasing unnecessary elements contained in the uploaded images using artificial intelligence, means for generating a plurality of interior design options based on the erased images, means for providing the generated plurality of design options to the user's information processing device, means for receiving user feedback and modifying the interior design based on the feedback, and means for analyzing the user's emotions and reflecting the user's emotional state in the design generation. This makes it possible to generate an interior design that takes the user's emotions into consideration, thereby improving user satisfaction, stimulating purchasing motivation, and maximizing the appeal of the property.

[0290] ---

[0291] That's all.

[0292] ---

[0293] "User" refers to a person who uses this system.

[0294] "Interior images" refer to digital images of interior spaces such as homes and commercial facilities.

[0295] "Uploading" refers to the act of sending data from a user's device to a server.

[0296] "Artificial intelligence" refers to computer programs and systems that automatically perform advanced processes such as data analysis and image processing.

[0297] "Unwanted elements" refer to objects or elements that are present in the image but should be removed when generating the interior design.

[0298] "Interior design options" refers to multiple interior design proposals generated based on a specific style or theme.

[0299] "Information processing device" refers to digital devices used by users, such as computers, smartphones, and tablets.

[0300] "Feedback" refers to user-provided comments and requests regarding the design.

[0301] "Analyzing emotions" refers to the process of determining a user's emotional state from their facial expressions and behavior.

[0302] An "emotional state" refers to the emotion (e.g., joy, surprise, dissatisfaction, etc.) that a user is feeling at a particular point in time.

[0303] ---

[0304] That's all.

[0305] ---

[0306] This invention relates to a virtual home staging system for increasing the desire to purchase vacant properties in real estate sales. This system uses artificial intelligence to generate virtual interior designs based on images of the interior taken by the user, which the user can then interactively modify and adjust. In addition, by combining it with an emotion engine that recognizes the user's emotions, the user's emotions can be reflected in the design generation.

[0307] The overall system consists of a user's device, a server, an interior design AI, and an emotion engine. Users take photos of the interior using a device such as a smartphone or tablet and send them to the server via a dedicated application. The server analyzes the received images, automatically removes unnecessary elements, and then uses the interior design AI to generate multiple design options. These design options are sent back to the user's device, and the user provides feedback based on the displayed designs. The emotion engine then analyzes the user's emotions and reflects the results in the interior design AI to generate new designs.

[0308] The specific hardware used is a smartphone or tablet. These devices communicate with the server through a dedicated application (for example, an app developed with React Native). The server-side software uses Python, OpenCV, Amazon S3, and a TensorFlow-based neural network model. The emotion engine uses Microsoft's Azure Emotion API.

[0309] Furthermore, to allow users to provide more specific feedback, the system can generate interior designs using multiple preset styles (e.g., "Modern," "Classic," and "Casual"), allowing users to easily select and adjust design options according to their preferences.

[0310] As a concrete example, consider a user taking and uploading a photo of their living room. The user launches the application, enters their login information, and logs in to their account. Next, they tap the app's "Upload Photo" button and select a photo they took from their device's gallery. The selected photo is automatically sent to the server, which performs image analysis and removes unnecessary elements such as cardboard boxes and old furniture. The interior design AI then generates multiple design options and sends them back to the user's device. The user reviews these designs and provides specific feedback. Along with the feedback, the emotion engine analyzes the user's facial expressions and incorporates their emotional state into the generated design.

[0311] Here are some example prompts to input to a generative AI model:

[0312] "Users upload photos of their living rooms, and AI eliminates unnecessary elements and generates multiple design options. We've built a system that modifies and optimizes the design based on user feedback and emotional state."

[0313] By inputting this prompt into the generative AI model, you can get specific advice on how to implement the above processing flow in detail and on optimization points.

[0314] ---

[0315] That's all.

[0316] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0317] ---

[0318] Step 1: User takes and uploads a photo

[0319] User: Takes a photo of their living room with their smartphone. The input is a high-quality digital image.

[0320] User: Launches the dedicated application, enters login information, and logs in to the account. Next, taps the "Upload Photo" button, selects a photo from the gallery, and sends it to the server. The output is the image data sent to the server.

[0321] Step 2: Image preprocessing

[0322] Server: Stores the received image data in temporary storage (e.g., cloud storage). The input is the image data sent by the user.

[0323] Server: The stored image data is passed to the image analysis module using Python and the OpenCV library. The output is clean image data with unnecessary elements removed.

[0324] How it works: OpenCV's contour detection function is used to detect cardboard boxes and old furniture in the image, and these unwanted elements are then removed using masking techniques.

[0325] Step 3: Generate interior design options

[0326] Server: Passes the clean image data to the interior design AI module using a TensorFlow-based neural network. The input is the preprocessed clean image data.

[0327] Server: The interior design AI module generates multiple design options (e.g., "modern," "classic," "casual"). The output is multiple interior design images.

[0328] How it works: The neural network model analyzes the input image and generates different interior designs based on each preset style.

[0329] Step 4: Providing design options

[0330] Server: Creates a response to send the generated multiple design options to the user's terminal. The input is the generated design options.

[0331] Terminal: Receives image data of the design options sent from the server and displays it to the user. The output is the design options displayed on the user terminal.

[0332] What it does: Use React Native to build an interface that lets users explore design options.

[0333] Step 5: User Provides Feedback

[0334] User: Review the displayed design options and enter specific changes in the app's feedback form. The input is the user's feedback.

[0335] Terminal: Generates and sends a request to the server to send the user's input feedback. The output is the feedback information.

[0336] Step 6: Manipulating the Emotion Engine

[0337] Terminal: When inputting feedback, the user's facial expression is captured by a camera and the video is sent to the emotion engine in real time. The input is video data of the user's facial expression.

[0338] Emotion Engine: Analyzes the user's facial expressions to detect the user's emotional state. The output is the user's emotional state data.

[0339] Emotion engine: The detected emotional state and feedback content are sent to the server. The output is integrated data of the emotional state and feedback.

[0340] Specific operation: Uses Microsoft's Azure Emotion API to analyze the user's emotional state from their facial expressions in real time.

[0341] Step 7: Interior design revisions

[0342] Server: The interior design AI module regenerates new design options based on the received feedback and emotional state. The input is the feedback and emotional state data.

[0343] Server: The regenerated design options are stored in temporary storage and sent to the user's device again. The output is the modified design options.

[0344] What it does: The neural network model recreates the design, taking into account feedback and emotional data.

[0345] Step 8: Finalize the design

[0346] User: If the new design is satisfactory, confirm it as the final design. Input is the selection information for the final design.

[0347] Server: Saves the final design to the system so that it can be reviewed later. The output is the saved final design.

[0348] ---

[0349] That's all.

[0350] (Application example 2)

[0351] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0352] Conventional interior design systems have difficulty reflecting the individual emotions and reactions of users, making it impossible to provide optimal designs that meet the user's needs. Furthermore, it is difficult to perfectly reproduce the interior experience in the real world, making it impossible to sufficiently stimulate the user's purchasing motivation. There is a need to solve these problems and provide optimal interior proposals by allowing users to experience designs in real time in a virtual reality environment while taking into account their emotions.

[0353] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0354] In this invention, the server includes means for uploading interior photos taken by a user, means for an AI to automatically delete unnecessary elements included in the uploaded photos, means for the interior design AI to generate multiple design options based on the deleted images, means for providing the generated multiple design options to the user's device, means for receiving user feedback and modifying the interior design based on the feedback, means for analyzing the user's emotional state and modifying the interior design based on the analysis results, and means for displaying the generated interior design options in a virtual reality environment. This makes it possible to provide interior designs that reflect the user's emotions, and further to stimulate greater purchasing motivation through a real-time experience in a virtual reality environment.

[0355] "User's device" refers to a device used by a user to input and display information, such as a smartphone, head-mounted display (HMD), or tablet.

[0356] "AI automatically removes unnecessary elements" is a process that uses artificial intelligence technology to automatically detect and remove unnecessary elements such as cardboard boxes and old furniture from indoor photos.

[0357] "Interior Design AI" is an artificial intelligence algorithm for generating interior designs and layouts, and has the ability to generate optimal design options taking into account user feedback and emotional state.

[0358] "Design options" are multiple interior design options generated by the interior design AI, including different styles and layouts.

[0359] "User feedback" refers to specific opinions and requests that users input regarding interior design options, such as "make the wall colors brighter" or "use more modern furniture."

[0360] "User emotional state" refers to the emotions the user feels when reviewing the design, such as joy, surprise, or dissatisfaction, which are analyzed in real time using artificial intelligence technology.

[0361] A "virtual reality environment" is a three-dimensional virtual space that users can virtually experience using a head-mounted display (HMD), allowing them to check and adjust interior design options in real time.

[0362] "Display in real time" means that the generated interior design options are displayed without delay so that the user can check them immediately.

[0363] The present invention relates to a virtual home staging system for increasing the willingness to purchase vacant properties in real estate sales. Specific embodiments of the system are described below.

[0364] Hardware and software used

[0365] Hardware:

[0366] Smartphones (latest iPhones and Android devices)

[0367] Head-mounted displays (HMDs) (e.g., Oculus Quest 2)

[0368] Camera (built-in smartphone or HMD camera)

[0369] software:

[0370] User device application (compatible with iOS / Android)

[0371] Emotion recognition libraries (e.g., Emotion AI SDK)

[0372] Interior design AI module (e.g., OpenAI GPT-4)

[0373] Backend servers (e.g., AWS, Google Cloud)

[0374] Data processing and calculation

[0375] 1. Image capture and transmission:

[0376] The user takes a photo of the room using a smartphone or the camera built into the HMD, and the image data is sent to a back-end server.

[0377] 2. Image analysis and removal of unwanted elements:

[0378] The server receives the image data and uses an AI image analysis module to automatically detect and remove unwanted elements (such as cardboard boxes, old furniture, etc.), resulting in a clean image.

[0379] 3. Interior design generation:

[0380] The server passes the clean image data to an interior design AI module, which generates multiple design options (e.g., "modern," "classic," "casual," etc.).

[0381] 4. User feedback and sentiment analysis:

[0382] The user reviews the generated design options via a smartphone or HMD and enters their feedback. When entering feedback, the user's face is photographed with a camera and the video data is sent to an emotion recognition library, which analyzes the user's emotional state (happiness, surprise, dissatisfaction, etc.).

[0383] 5. Regenerate the design and display it in a virtual reality environment:

[0384] The server receives the feedback and emotional state, and the interior design AI module regenerates new design options, which are then sent back to the user's device and displayed in real time in the virtual reality environment.

[0385] Examples and prompts

[0386] For example, if a user provides feedback such as "I would like the walls to be a lighter color," and the camera captures the user's face and determines their emotional state as "happy," the interior design AI module might receive the following prompt:

[0387] Example prompt sentence:

[0388] Suggest a modern interior design that best suits the user's joyful moments, taking into account feedback that the wall colors should be lighter.

[0389] This allows optimal designs that reflect the user's emotions to be generated in real time and displayed in a virtual reality environment, increasing the user's desire to purchase.

[0390] This series of processes allows us to provide interior designs that take into account the user's needs and emotions, and by experiencing them realistically through a virtual environment, we can further increase their desire to purchase.

[0391] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0392] Step 1:

[0393] A user takes a photo of the room using a smartphone or a head-mounted display (HMD) and uploads the photo to the server via a dedicated application. In this step, the input is the photo of the room taken by the user, and the output is that the photo is sent to the server.

[0394] Step 2:

[0395] The server receives the uploaded photo and saves the image data in temporary storage. The saved image data is then passed to the AI ​​image analysis module. In this step, the input is the received photo data and the output is the saved image data.

[0396] Step 3:

[0397] The server uses an image analysis module to automatically detect unwanted elements in the image (e.g., cardboard boxes or old furniture) and digitally remove them. The input for this step is the stored image data, and the output is clean image data with unwanted elements removed.

[0398] Step 4:

[0399] The cleaned image data is passed to an interior design AI module to generate multiple interior design options. For example, designs based on preset styles such as "modern," "classic," and "casual" are generated. The input in this step is the cleaned image data, and the output is multiple design options.

[0400] Step 5:

[0401] The server sends the generated design options to the user's terminal. The user's terminal receives the design options and displays them to the user. In this step, the input is the generated design options and the output is the displayed design options.

[0402] Step 6:

[0403] The user reviews the displayed design options and enters their feedback, which can include specific suggestions such as "I would like the wall color to be lighter." The input in this step is the user's feedback, and the output is the content of the feedback.

[0404] Step 7:

[0405] When the user inputs feedback, the camera on the smartphone or HMD captures the user's face, and the video data is sent to the emotion recognition library in real time. The input of this step is the user's facial video data, and the output is the analyzed emotional state.

[0406] Step 8:

[0407] The emotion recognition library detects the user's emotional state (e.g., joy, surprise, dissatisfaction, etc.) and sends the data to the server. The input in this step is facial video data, and the output is emotional state data.

[0408] Step 9:

[0409] The server receives feedback and emotional state data, and the interior design AI module regenerates new design options based on that. For example, if the emotional state is "joy" and feedback is "make the wall color brighter," a prompt sentence is generated to suggest the optimal design. The input of this step is the feedback and emotional state data, and the output is a new design option.

[0410] Step 10:

[0411] The new design options are resent to the user's device and displayed in the virtual reality environment in real time. The user then checks the new design in the VR space. The input of this step is the regenerated design options, and the output is the design options displayed in the VR space.

[0412] This series of processes provides optimal interior designs in real time based on the user's emotions and feedback.

[0413] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0414] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0415] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0416] [Second embodiment]

[0417] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0418] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0419] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0420] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0421] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0422] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0423] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0424] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0425] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0426] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0427] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0428] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0429] ---

[0430] This invention relates to a virtual home staging system for increasing the desire to purchase vacant properties in real estate sales. This system uses AI to generate virtual interior designs based on photos of the interior taken by the user, and the user can interactively modify and adjust them.

[0431] Overall system overview

[0432] This system mainly consists of a user's device, a server, and an AI module. Users take photos of the interior of a room using a device such as a smartphone or tablet and send them to the server via an application. The server analyzes the received photos, automatically removes unnecessary elements, and then generates multiple design options using the interior design AI module. The generated design options are returned to the user's device, and the user provides feedback based on the displayed designs. The server passes this feedback back to the AI, which then generates a new design and provides it to the user.

[0433] Program processing flow (example)

[0434] 1. User photo uploads

[0435] User: Takes a photo of the living room with a smartphone and uploads it to a dedicated application.

[0436] Device: Sends uploaded photos to the server.

[0437] 2. Image Preprocessing

[0438] Server: Saves the received images in temporary storage.

[0439] Server: Uses AI to automatically detect and remove unwanted elements (e.g., cardboard boxes, old furniture, etc.) from images.

[0440] 3. Generate multiple interior design options

[0441] Server: Passes the preprocessed images to the interior design AI module.

[0442] Server: The module generates multiple design options based on preset styles (e.g., modern, classic, casual).

[0443] Server: Stores the generated design options in temporary storage.

[0444] 4. Providing design options

[0445] Server: Sends the generated design options to the user's device.

[0446] Terminal: Displays the received design options to the user.

[0447] 5. User Feedback

[0448] User: Review the displayed design and enter feedback into the application, such as "I'd like the wall colors to be brighter" or "I'd like the furniture to be a little more modern."

[0449] Terminal: Sends user feedback to the server.

[0450] 6. Interior design revisions

[0451] Server: Receives feedback and uses it to generate new design options for the interior design AI module.

[0452] Server: Saves the modified design options in temporary storage and sends them back to the user's device.

[0453] Terminal: Redisplays new design options to the user.

[0454] 7. Finalize the design

[0455] User: If the new design is satisfactory, the user confirms it as the final design, which is then saved in the system and can be reviewed later.

[0456] This series of processes allows users to easily and interactively experiment with interior designs to find the optimal design. This system is expected to promote real estate sales.

[0457] The processing flow will be explained below.

[0458] ---

[0459] Step 1:

[0460] User: Take a photo of the living room with a smartphone, launch the dedicated application and log in.

[0461] Device: Tap the "Upload Photo" button on the application screen and select the photo you have taken.

[0462] On the device: Send the selected photo to the backend server.

[0463] Step 2:

[0464] Server: Stores the received photo data in temporary storage.

[0465] Server: Passes the saved image data to the AI ​​image analysis module.

[0466] Server: The image analysis module automatically detects unwanted elements in the image (e.g., cardboard boxes, old furniture) and digitally removes them.

[0467] Step 3:

[0468] Server: Obtains clean image data with unnecessary elements removed and passes it to the interior design AI module.

[0469] Server: The interior design AI module generates multiple design options based on preset styles such as "modern," "classic," and "casual."

[0470] Server: Saves each generated design option to temporary storage.

[0471] Step 4:

[0472] Server: Creates a response for sending image data from the saved multiple design options to the user's terminal.

[0473] Terminal: Receives image data of design options sent from the server and displays it to the user.

[0474] Step 5:

[0475] User: Review the design options presented and provide specific feedback if necessary (e.g., "I'd like the walls to be a lighter color" or "I'd like the furniture to be a little more modern").

[0476] Terminal: Makes a request to send the user's input feedback to the server.

[0477] Step 6:

[0478] Server: Analyzes the received feedback and passes it to the interior design AI module.

[0479] Server: The interior design AI module regenerates new design options based on feedback.

[0480] Server: Saves the newly generated design options to temporary storage.

[0481] Step 7:

[0482] Server: Creates a response to send the new design options to the user's device.

[0483] Terminal: Re-display new design options received to the user.

[0484] User: Checks the new design and, if satisfied, provides input to confirm the final design.

[0485] Terminal: Sends final design confirmation information to the server.

[0486] ---

[0487] These are the specific processing steps of the program for the entire system, which allow users to interactively customize the interior design and realize a virtual home staging that reflects their ideals.

[0488] Example 1

[0489] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0490] In real estate sales, there is a demand for methods to visually enhance the appeal of vacant properties. Installing realistic interior designs is costly and time-consuming, and potential buyers have difficulty forming a concrete image of the property. Conventional methods make it difficult for users to select the design they want through trial and error, which discourages them from purchasing.

[0491] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0492] In this invention, the server includes means for uploading interior photos taken by a user, means for automatically erasing unnecessary elements included in the uploaded photos by an artificial intelligence, means for the interior design artificial intelligence to generate multiple design options based on the erased images, means for providing the generated multiple design options to the user's information processing device, means for receiving user feedback and modifying the interior design based on the feedback, means for regenerating new design options based on the user's feedback, means for saving the generated design options in online storage, and means for providing the regenerated design options again to the user's information processing device, thereby enabling the user to interactively experiment with interior designs and find the optimal design.

[0493] "User" refers to a person who uses this system to take photos of an interior and receive interior design suggestions.

[0494] "Server" refers to a computing device that receives users' photos and uses artificial intelligence to process the images and generate designs.

[0495] "Information processing device" refers to an electronic device (e.g., smartphone, tablet, PC) that a user uses to take photos and receive design suggestions from a server.

[0496] "Unwanted elements" refer to objects or backgrounds (e.g., cardboard boxes, old furniture) that are included in the interior photos taken by the user but should be removed when proposing interior designs.

[0497] "Artificial intelligence" refers to technology that automates image processing and design generation using large amounts of data and machine learning algorithms.

[0498] "Interior design artificial intelligence" refers to a specific artificial intelligence model used to automatically generate interior design options from a user's photos.

[0499] "Design options" refers to multiple different interior design proposals generated by the interior design AI.

[0500] "Online storage" refers to a data storage service that allows you to store data via the Internet and access it as needed.

[0501] "Feedback" refers to specific opinions and change requests that users give regarding design options.

[0502] This invention is a virtual home staging system for visually enhancing the appeal of vacant properties in real estate sales. This system uses artificial intelligence (AI) to generate virtual interior designs based on interior photos taken by the user, and allows the user to interactively modify and adjust them.

[0503] Overall system overview

[0504] This system primarily consists of the user's information processing device (smartphone, tablet, PC, etc.), a server, and an interior design AI module. The user uses the information processing device to take photos of the interior and sends them to the server via a dedicated application. The server analyzes the received photos, automatically removes unnecessary elements, and then uses the interior design AI module to generate multiple design options. The generated design options are provided to the user's information processing device, and the user provides feedback based on the displayed designs. The server passes this feedback back to the AI ​​module, which then generates a new design and provides it to the user.

[0505] Hardware and software used

[0506] Hardware:

[0507] Information processing devices (smartphones, tablets, PCs, etc.)

[0508] Server (for data processing and storage)

[0509] software:

[0510] Dedicated application (installed on the user's information processing device)

[0511] Server-side software (online storage such as Amazon S3, Python scripts, OpenCV, TensorFlow-based interior design AI module)

[0512] Specific examples of processing

[0513] Example 1: User uploads a photo

[0514] A user takes a photo of their living room with their smartphone and uploads it to the server using a dedicated application. The user presses the upload button in the application to start sending the photo. The server temporarily stores the received photo in an Amazon S3 bucket.

[0515] Example 2: Removing unnecessary elements

[0516] The server uses the OpenCV library to analyze the stored photos, automatically detecting unwanted elements in the photo, such as cardboard boxes or old furniture, and identifying them through masking. It then applies an inpainting technique to remove the unwanted elements from the image.

[0517] Example 3: Generating interior designs

[0518] The preprocessed images are input into a TensorFlow-based interior design AI module, which generates multiple design options based on pre-trained design patterns (e.g., modern, classic, casual).

[0519] Example prompt sentence:

[0520] For example, if a user "wants a modern style living room," they would enter the following as the prompt:

[0521] For example: "Please add a modern style interior design to the living room in the photo. I would like light wall colors and simple furniture."

[0522] This system allows users to interactively experiment with interior designs to find the best fit, and is expected to help promote real estate sales.

[0523] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0524] Step 1:

[0525] A user takes a photo of their living room with their smartphone and uploads it via a dedicated application. The input is the captured photo data in JPEG format, and the output is an upload request to the server via the application. When the upload button is pressed within the application, the photo data is sent over the Internet to a specified server URL.

[0526] Step 2:

[0527] The device receives the uploaded photo data and temporarily stores it on the device. The input is the JPEG photo data uploaded by the user, and the output is the operation of temporarily storing this data. The stored photo data is ready to be transferred to the server.

[0528] Step 3:

[0529] The server saves the received photo data to storage. An online storage service (e.g., Amazon S3 bucket) is used for saving. The input is the JPEG photo data sent from the device, and the output is the URL of the photo data saved in storage. This URL is used in subsequent processing.

[0530] Step 4:

[0531] The server uses the OpenCV library to erase unnecessary elements in the image. The input is photo data retrieved from storage, and the output is new image data with unnecessary elements erased. Specifically, OpenCV is used to detect cardboard boxes and old furniture in the image and apply inpainting techniques to erase them.

[0532] Step 5:

[0533] The server inputs the preprocessed image data into an interior design AI model. This AI model is built with TensorFlow and uses pre-trained design patterns. The input is image data with unnecessary elements removed, and the output is multiple interior design options. The model analyzes the image and generates design options based on styles such as "modern," "classic," and "casual."

[0534] Step 6:

[0535] The server stores the generated design options in online storage and provides them to the user's information processing device. The input is multiple design options generated by the interior design AI model, and the output is a list of URLs stored in online storage. This list of URLs is then sent to the user's device.

[0536] Step 7:

[0537] The device receives a list of design option URLs and displays each design option in the user interface. The input is the URL list received from the server, and the output is an image of the design option that the user can view. The user can swipe to switch between multiple design options.

[0538] Step 8:

[0539] The user enters feedback on the displayed design options. For example, specific opinions such as "make the wall color lighter" or "rearrange the furniture" are entered into the text input field. The input is the user's feedback text, and the output is a request to send the feedback information to the server.

[0540] Step 9:

[0541] The server analyzes the received feedback and re-inputs it into the interior design AI model. The input is the feedback data received from the user, and the output is the newly generated design options. For example, in response to the feedback "make the wall color lighter," a color conversion process is performed to generate a new interior design.

[0542] Step 10:

[0543] The server saves the newly generated design options in online storage again and provides them to the user's information processing device. The input is the newly generated design options, and the output is a list of URLs saved in online storage. This list of URLs is then sent to the user's device again.

[0544] Step 11:

[0545] The device again receives new design options and displays them in the user interface. The input is the new URL list received from the server, and the output is an image of the latest design options that the user can view. The user again reviews the designs and continues to provide feedback as needed.

[0546] Step 12:

[0547] When the user is satisfied with the design, they press the confirm button in the app to confirm it as the final design. The input is the design option that the user confirms, and the output is the operation of sending the confirmed design to the server and saving it in the database.

[0548] This series of processes allows the user to interactively experiment with interior designs and find the optimal design.

[0549] (Application example 1)

[0550] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0551] In real estate sales, there are limited means to promote vacant properties, making it difficult for potential buyers to visualize the interior of the room. Furthermore, to realize virtual home staging, a flexible system is required that allows users to easily reflect their own preferences and requests, but conventional technology has difficulty meeting this requirement. Furthermore, the same issue exists in virtual stores, as there are not enough functions to design interiors in virtual spaces and save and display the results.

[0552] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0553] In this invention, the server includes a means for uploading interior photos taken by a user, a means for an AI to automatically delete unnecessary elements from the uploaded photos, a means for the interior design AI to generate multiple design options based on the deleted images, a means for accepting user feedback and modifying the interior design based on the feedback, a means for re-receiving user feedback and regenerating the interior design based on the feedback, and a means for saving and displaying the finalized design in the virtual store. This allows users to design interiors in a virtual space based on photos of their actual rooms and to find their ideal interior by repeatedly modifying and regenerating the design. Furthermore, using the finalized design in the virtual store can promote real estate sales and improve the convenience of interior customization in virtual stores.

[0554] "User" refers to the end user who uses the system to upload interior photos and provide feedback on interior design.

[0555] "Uploading means" refers to the interface and communication means for sending photos taken by the user to the server.

[0556] "Means for AI to automatically remove unnecessary elements" refers to a function that uses AI technology to automatically analyze and remove unnecessary objects and items from uploaded photos.

[0557] "Interior Design AI" refers to an artificial intelligence module that generates multiple design options based on a cleaned-up image.

[0558] "Design options" refers to multiple interior design options generated by the interior design AI.

[0559] "User feedback" refers to the user's evaluation of and requests for modifications to the provided design options.

[0560] "Means to modify interior design" refers to the function of reviewing and regenerating interior design based on user feedback.

[0561] A "virtual store" refers to a virtual shopping or exhibition space where products and interiors can be arranged and customized within a virtual space.

[0562] "Final design" refers to the interior design that is finalized as a result of incorporating user feedback.

[0563] "Means for saving and displaying" refers to the function for saving the finalized design within the system and displaying it within the virtual store.

[0564] This invention relates to a virtual home staging system that uses AI to generate a virtual interior design based on indoor photos taken by the user, and allows the user to interactively modify and adjust the design.

[0565] Overall system overview

[0566] The system primarily consists of a user device, a server, and an interior design AI module. Users take photos of the interior using a device such as a smartphone or virtual glasses and send them to the server via an application. The server analyzes the received photos, automatically removes unnecessary elements, and then generates multiple design options using the interior design AI module. The generated design options are returned to the user's device, and the user provides feedback based on the displayed designs. The server passes this feedback back to the AI, which then generates a new design and provides it to the user. The finalized design is saved and displayed in the virtual store.

[0567] Hardware / Software used

[0568] Hardware: Smartphone, virtual glasses, and server

[0569] Software: Python, TensorFlow, OpenCV, Flask

[0570] Program processing details

[0571] 1. User photo uploads

[0572] Users can take photos of the interior of a room using a smartphone app and upload them to a server via a dedicated application.

[0573] 2. Image preprocessing by the server

[0574] The server stores the received images in storage and uses OpenCV to automatically detect and remove unwanted elements (e.g., cardboard boxes, old furniture, etc.) from the photos, generating a clean base image.

[0575] 3. Interior design generation

[0576] The server passes the preprocessed images to an interior design AI module using TensorFlow, which generates multiple design options based on preset styles such as "modern," "classic," and "casual." The generated design options are stored in temporary storage and sent to the user's device.

[0577] 4. User Feedback

[0578] The user reviews the displayed design options and enters specific feedback into the application, such as "I'd like the wall colors to be brighter" or "I'd like the furniture to be a little more modern." This feedback is then sent to the server.

[0579] 5. Interior design revisions

[0580] The server then uses the user's feedback to generate new design options using the interior design AI module, and this process is repeated until the user is satisfied.

[0581] 6. Finalize the design and display it in the virtual store

[0582] Once the user is satisfied with the design, it is saved and displayed in the virtual store, allowing them to visualize how the interior design will look in a real room in a virtual space.

[0583] Specific examples

[0584] For example, if a user submits a photo of their living room and requests a "modern design," the AI ​​will generate multiple design options incorporating elements such as "light-colored walls," "simple modern furniture," and "spacious layout." A specific example of a prompt would be, "I have submitted a photo of my living room. Please generate a modern interior design that suits this room. I would like the furniture to be simple and the colors to be light." By sending this instruction to the server, a design based on the user's requests will be generated.

[0585] As described above, this system allows users to easily experiment with interior designs to find their ideal design. The final design can also be saved and displayed in the virtual store, contributing to promoting real estate sales and improving the user experience.

[0586] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0587] Step 1:

[0588] User-uploaded photos

[0589] Users take photos of the interior of a room using a smartphone app and upload them to the server through a dedicated application. The input is the photo of the room taken with the user's smartphone, and the output is the photo file sent to the server. Specifically, when the "upload photo" button is pressed on the application, the photo is automatically sent to the server.

[0590] Step 2:

[0591] Image preprocessing by the server

[0592] The server stores the received image file in temporary storage and uses the OpenCV library to automatically detect and remove unwanted elements (e.g., cardboard boxes, old furniture, etc.) in the photo. The input is the image file stored on the server, and the output is a clean image with unwanted elements removed. Specifically, the server converts the image to grayscale, performs binary thresholding to mask and remove unwanted objects.

[0593] Step 3:

[0594] Interior Design Generation

[0595] The server sends the preprocessed image to the interior design AI module, which uses TensorFlow to generate multiple design options, such as "modern," "classic," and "casual." The input is clean image data, and the output is multiple interior design options. Specifically, the server passes the image to the AI ​​module, which generates multiple design options based on preset styles.

[0596] Step 4:

[0597] User Feedback

[0598] The user checks the generated design options on a smartphone app and provides feedback such as "change the wall color to a lighter color" or "add more modern furniture." The input is the user's feedback data, and the output is the feedback information sent to the server. Specifically, when the user presses the "Send Feedback" button on the app, the comment is sent to the server.

[0599] Step 5:

[0600] Interior design modifications

[0601] The server receives feedback from the user, and the interior design AI module regenerates new design options based on that feedback. The input is the user's feedback data and preprocessed images, and the output is a new design option modified based on the feedback. Specifically, the server passes the feedback to the AI ​​and saves the regenerated design option in temporary storage.

[0602] Step 6:

[0603] Finalize the design and display it in the virtual store

[0604] The user finally decides on a design that satisfies them, and that design is saved and displayed in the virtual store. The input is the final design data decided by the user, and the output is the final design saved in the virtual store system. Specifically, when the user presses the "Confirm Design" button on the app, the design is saved in the virtual store and becomes available for the user to view in the virtual space.

[0605] This series of processes allows users to find their ideal interior design through trial and error, and ultimately to check the design in a virtual store.

[0606] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0607] ---

[0608] This invention relates to a virtual home staging system for increasing the desire to purchase vacant properties in real estate sales. This system uses AI to generate virtual interior designs based on interior photos taken by the user, which the user can then interactively modify and adjust. In addition, by combining it with an emotion engine that recognizes the user's emotions, the user's emotions can be reflected in the design generation.

[0609] Overall system overview

[0610] This system consists of a user's device, a server, an interior design AI, and an emotion engine. Users take photos of the interior using a device such as a smartphone or tablet and send them to the server via an application. The server analyzes the received photos, automatically removes unnecessary elements, and then uses the interior design AI to generate multiple design options. The generated design options are returned to the user's device, and the user provides feedback based on the displayed designs. The emotion engine then analyzes the user's emotions and reflects the results in the interior design AI to generate new designs.

[0611] Program processing flow (example)

[0612] User-uploaded photos

[0613] User: Take a photo of the living room with a smartphone, launch the dedicated application and log in.

[0614] Device: Tap the "Upload Photo" button on the application screen and select the photo you have taken.

[0615] On the device: Send the selected photo to the backend server.

[0616] Image preprocessing

[0617] Server: Stores the received photo data in temporary storage.

[0618] Server: Passes the saved image data to the AI ​​image analysis module.

[0619] Server: The image analysis module automatically detects unwanted elements in the image (e.g., cardboard boxes, old furniture) and digitally removes them.

[0620] Generate multiple interior design options

[0621] Server: Obtains clean image data with unnecessary elements removed and passes it to the interior design AI module.

[0622] Server: The interior design AI module generates multiple design options based on preset styles such as "modern," "classic," and "casual."

[0623] Server: Saves each generated design option to temporary storage.

[0624] Providing design options

[0625] Server: Creates a response for sending image data from multiple saved design options to the user's device.

[0626] Terminal: Receives image data of design options sent from the server and displays it to the user.

[0627] User Feedback

[0628] User: Review the design options presented and provide specific feedback if necessary (e.g., "I'd like the walls to be a lighter color" or "I'd like the furniture to be a little more modern").

[0629] Terminal: Makes a request to send the user's input feedback to the server.

[0630] Manipulating the Emotion Engine

[0631] Terminal: When inputting feedback, the user's face is photographed with a camera and the image is sent to the emotion engine in real time.

[0632] Emotion engine: Analyzes the user's facial expressions to detect the user's emotional state (e.g., joy, surprise, dissatisfaction).

[0633] Emotion engine: Sends the detected emotional state and feedback content to the server.

[0634] Interior design modifications

[0635] Server: Receives feedback and emotional state, and the interior design AI module regenerates new design options based on that.

[0636] Server: Saves the newly generated design options in temporary storage and sends them back to the user's device.

[0637] Terminal: Re-display new design options received to the user.

[0638] Final design confirmation

[0639] User: If the new design is satisfactory, the user confirms it as the final design, which is then saved in the system and can be reviewed later.

[0640] This series of processes allows users to customize interior designs that take their emotions into account, enabling virtual home staging that is more suited to the user. This system can increase purchasing motivation and maximize the appeal of the property.

[0641] The processing flow will be explained below.

[0642] ---

[0643] Step 1:

[0644] User: Launch the dedicated application on your smartphone and log in.

[0645] User: Takes a photo of the living room with his smartphone.

[0646] Step 2:

[0647] Device: Tap the "Upload Photo" button on the application screen and select the photo you have taken.

[0648] Terminal: Compress the selected photo data and send it to the server.

[0649] Step 3:

[0650] Server: Stores the received photo data in temporary storage.

[0651] Server: Passes the saved image data to the image analysis module.

[0652] Step 4:

[0653] Server: The image analysis module analyzes the image and automatically detects unwanted elements (e.g., old furniture or cardboard boxes).

[0654] Server: Digitally erases detected unwanted elements to generate clean image data.

[0655] Step 5:

[0656] Server: Passes the clean image data to the interior design AI module and instructs it to generate multiple design options.

[0657] Server: The interior design AI module generates multiple design options based on preset styles such as "modern," "classic," and "casual."

[0658] Step 6:

[0659] Server: Stores the generated design options in temporary storage and creates a response to send to the user's device.

[0660] Terminal: Receives image data of design options sent from the server and displays it to the user.

[0661] Step 7:

[0662] User: Review each design option presented and provide feedback (e.g., "I'd like the walls to be a lighter color" or "I'd like the furniture to be a little more modern").

[0663] Device: When inputting feedback, the user's face is photographed with the smartphone camera and the image is sent to the emotion engine in real time.

[0664] Step 8:

[0665] Emotion engine: Analyzes the user's facial expressions to detect their emotional state (e.g., happiness, surprise, dissatisfaction).

[0666] Emotion engine: Sends the detected emotional state and feedback content to the server.

[0667] Step 9:

[0668] Server: Analyzes the received feedback and emotional state and instructs the interior design AI module to regenerate new design options based on that.

[0669] Server: The interior design AI module generates new design options based on user feedback and sentiment.

[0670] Step 10:

[0671] Server: Stores the newly generated design options in temporary storage and creates a response to send to the user's device.

[0672] Terminal: Re-display new design options received to the user.

[0673] Step 11:

[0674] Users: Review the new design options and provide feedback again if needed.

[0675] User: When a satisfactory design is confirmed, it is confirmed as the final design and saved in the system.

[0676] ---

[0677] This series of processes enables customization of interior designs that take emotion analysis into account, realizing optimal virtual home staging for each user. This system makes it possible to propose designs that reflect the user's emotions, maximizing the appeal of the property.

[0678] Example 2

[0679] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0680] Conventional interior design systems have had the drawback of requiring the user to adjust the design themselves, and of making it difficult to generate a design that reflects the user's emotions. This has resulted in the problem that it takes a lot of time and effort for the user to obtain a design that actually satisfies them. The objective of the present invention is to provide a system that can easily generate an interior design that meets the user's needs by analyzing the user's emotional state and reflecting that in the design.

[0681] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for uploading images of an interior taken by a user, means for automatically erasing unnecessary elements contained in the uploaded images using artificial intelligence, means for generating a plurality of interior design options based on the erased images, means for providing the generated plurality of design options to the user's information processing device, means for receiving user feedback and modifying the interior design based on the feedback, and means for analyzing the user's emotions and reflecting the user's emotional state in the design generation. This makes it possible to generate an interior design that takes the user's emotions into consideration, thereby improving user satisfaction, stimulating purchasing motivation, and maximizing the appeal of the property.

[0682] ---

[0683] That's all.

[0684] ---

[0685] "User" refers to a person who uses this system.

[0686] "Interior images" refer to digital images of interior spaces such as homes and commercial facilities.

[0687] "Uploading" refers to the act of sending data from a user's device to a server.

[0688] "Artificial intelligence" refers to computer programs and systems that automatically perform advanced processes such as data analysis and image processing.

[0689] "Unwanted elements" refer to objects or elements that are present in the image but should be removed when generating the interior design.

[0690] "Interior design options" refers to multiple interior design proposals generated based on a specific style or theme.

[0691] "Information processing device" refers to digital devices used by users, such as computers, smartphones, and tablets.

[0692] "Feedback" refers to user-provided comments and requests regarding the design.

[0693] "Analyzing emotions" refers to the process of determining a user's emotional state from their facial expressions and behavior.

[0694] An "emotional state" refers to the emotion (e.g., joy, surprise, dissatisfaction, etc.) that a user is feeling at a particular point in time.

[0695] ---

[0696] That's all.

[0697] ---

[0698] This invention relates to a virtual home staging system for increasing the desire to purchase vacant properties in real estate sales. This system uses artificial intelligence to generate virtual interior designs based on images of the interior taken by the user, which the user can then interactively modify and adjust. In addition, by combining it with an emotion engine that recognizes the user's emotions, the user's emotions can be reflected in the design generation.

[0699] The overall system consists of a user's device, a server, an interior design AI, and an emotion engine. Users take photos of the interior using a device such as a smartphone or tablet and send them to the server via a dedicated application. The server analyzes the received images, automatically removes unnecessary elements, and then uses the interior design AI to generate multiple design options. These design options are sent back to the user's device, and the user provides feedback based on the displayed designs. The emotion engine then analyzes the user's emotions and reflects the results in the interior design AI to generate new designs.

[0700] The specific hardware used is a smartphone or tablet. These devices communicate with the server through a dedicated application (for example, an app developed with React Native). The server-side software uses Python, OpenCV, Amazon S3, and a TensorFlow-based neural network model. The emotion engine uses Microsoft's Azure Emotion API.

[0701] Furthermore, to allow users to provide more specific feedback, the system can generate interior designs using multiple preset styles (e.g., "Modern," "Classic," and "Casual"), allowing users to easily select and adjust design options according to their preferences.

[0702] As a concrete example, consider a user taking and uploading a photo of their living room. The user launches the application, enters their login information, and logs in to their account. Next, they tap the app's "Upload Photo" button and select a photo they took from their device's gallery. The selected photo is automatically sent to the server, which performs image analysis and removes unnecessary elements such as cardboard boxes and old furniture. The interior design AI then generates multiple design options and sends them back to the user's device. The user reviews these designs and provides specific feedback. Along with the feedback, the emotion engine analyzes the user's facial expressions and incorporates their emotional state into the generated design.

[0703] Here are some example prompts to input to a generative AI model:

[0704] "Users upload photos of their living rooms, and AI eliminates unnecessary elements and generates multiple design options. We've built a system that modifies and optimizes the design based on user feedback and emotional state."

[0705] By inputting this prompt into the generative AI model, you can get specific advice on how to implement the above processing flow in detail and on optimization points.

[0706] ---

[0707] That's all.

[0708] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0709] ---

[0710] Step 1: User takes and uploads a photo

[0711] User: Takes a photo of their living room with their smartphone. The input is a high-quality digital image.

[0712] User: Launches the dedicated application, enters login information, and logs in to the account. Next, taps the "Upload Photo" button, selects a photo from the gallery, and sends it to the server. The output is the image data sent to the server.

[0713] Step 2: Image preprocessing

[0714] Server: Stores the received image data in temporary storage (e.g., cloud storage). The input is the image data sent by the user.

[0715] Server: The stored image data is passed to the image analysis module using Python and the OpenCV library. The output is clean image data with unnecessary elements removed.

[0716] How it works: OpenCV's contour detection function is used to detect cardboard boxes and old furniture in the image, and these unwanted elements are then removed using masking techniques.

[0717] Step 3: Generate interior design options

[0718] Server: Passes the clean image data to the interior design AI module using a TensorFlow-based neural network. The input is the preprocessed clean image data.

[0719] Server: The interior design AI module generates multiple design options (e.g., "modern," "classic," "casual"). The output is multiple interior design images.

[0720] How it works: The neural network model analyzes the input image and generates different interior designs based on each preset style.

[0721] Step 4: Providing design options

[0722] Server: Creates a response to send the generated multiple design options to the user's terminal. The input is the generated design options.

[0723] Terminal: Receives image data of the design options sent from the server and displays it to the user. The output is the design options displayed on the user terminal.

[0724] What it does: Use React Native to build an interface that lets users explore design options.

[0725] Step 5: User Provides Feedback

[0726] User: Review the displayed design options and enter specific changes in the app's feedback form. The input is the user's feedback.

[0727] Terminal: Generates and sends a request to the server to send the user's input feedback. The output is the feedback information.

[0728] Step 6: Manipulating the Emotion Engine

[0729] Terminal: When inputting feedback, the user's facial expression is captured by a camera and the video is sent to the emotion engine in real time. The input is video data of the user's facial expression.

[0730] Emotion Engine: Analyzes the user's facial expressions to detect the user's emotional state. The output is the user's emotional state data.

[0731] Emotion engine: The detected emotional state and feedback content are sent to the server. The output is integrated data of the emotional state and feedback.

[0732] Specific operation: Uses Microsoft's Azure Emotion API to analyze the user's emotional state from their facial expressions in real time.

[0733] Step 7: Interior design revisions

[0734] Server: The interior design AI module regenerates new design options based on the received feedback and emotional state. The input is the feedback and emotional state data.

[0735] Server: The regenerated design options are stored in temporary storage and sent to the user's device again. The output is the modified design options.

[0736] What it does: The neural network model recreates the design, taking into account feedback and emotional data.

[0737] Step 8: Finalize the design

[0738] User: If the new design is satisfactory, confirm it as the final design. Input is the selection information for the final design.

[0739] Server: Saves the final design to the system so that it can be reviewed later. The output is the saved final design.

[0740] ---

[0741] That's all.

[0742] (Application example 2)

[0743] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0744] Conventional interior design systems have difficulty reflecting the individual emotions and reactions of users, making it impossible to provide optimal designs that meet the user's needs. Furthermore, it is difficult to perfectly reproduce the interior experience in the real world, making it impossible to sufficiently stimulate the user's purchasing motivation. There is a need to solve these problems and provide optimal interior proposals by allowing users to experience designs in real time in a virtual reality environment while taking into account their emotions.

[0745] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0746] In this invention, the server includes means for uploading interior photos taken by a user, means for an AI to automatically delete unnecessary elements included in the uploaded photos, means for the interior design AI to generate multiple design options based on the deleted images, means for providing the generated multiple design options to the user's device, means for receiving user feedback and modifying the interior design based on the feedback, means for analyzing the user's emotional state and modifying the interior design based on the analysis results, and means for displaying the generated interior design options in a virtual reality environment. This makes it possible to provide interior designs that reflect the user's emotions, and further to stimulate greater purchasing motivation through a real-time experience in a virtual reality environment.

[0747] "User's device" refers to a device used by a user to input and display information, such as a smartphone, head-mounted display (HMD), or tablet.

[0748] "AI automatically removes unnecessary elements" is a process that uses artificial intelligence technology to automatically detect and remove unnecessary elements such as cardboard boxes and old furniture from indoor photos.

[0749] "Interior Design AI" is an artificial intelligence algorithm for generating interior designs and layouts, and has the ability to generate optimal design options taking into account user feedback and emotional state.

[0750] "Design options" are multiple interior design options generated by the interior design AI, including different styles and layouts.

[0751] "User feedback" refers to specific opinions and requests that users input regarding interior design options, such as "make the wall colors brighter" or "use more modern furniture."

[0752] "User emotional state" refers to the emotions the user feels when reviewing the design, such as joy, surprise, or dissatisfaction, which are analyzed in real time using artificial intelligence technology.

[0753] A "virtual reality environment" is a three-dimensional virtual space that users can virtually experience using a head-mounted display (HMD), allowing them to check and adjust interior design options in real time.

[0754] "Display in real time" means that the generated interior design options are displayed without delay so that the user can check them immediately.

[0755] The present invention relates to a virtual home staging system for increasing the willingness to purchase vacant properties in real estate sales. Specific embodiments of the system are described below.

[0756] Hardware and software used

[0757] Hardware:

[0758] Smartphones (latest iPhones and Android devices)

[0759] Head-mounted displays (HMDs) (e.g., Oculus Quest 2)

[0760] Camera (built-in smartphone or HMD camera)

[0761] software:

[0762] User device application (compatible with iOS / Android)

[0763] Emotion recognition libraries (e.g., Emotion AI SDK)

[0764] Interior design AI module (e.g., OpenAI GPT-4)

[0765] Backend servers (e.g., AWS, Google Cloud)

[0766] Data processing and calculation

[0767] 1. Image capture and transmission:

[0768] The user takes a photo of the room using a smartphone or the camera built into the HMD, and the image data is sent to a back-end server.

[0769] 2. Image analysis and removal of unwanted elements:

[0770] The server receives the image data and uses an AI image analysis module to automatically detect and remove unwanted elements (such as cardboard boxes, old furniture, etc.), resulting in a clean image.

[0771] 3. Interior design generation:

[0772] The server passes the clean image data to an interior design AI module, which generates multiple design options (e.g., "modern," "classic," "casual," etc.).

[0773] 4. User feedback and sentiment analysis:

[0774] The user reviews the generated design options via a smartphone or HMD and enters their feedback. When entering feedback, the user's face is photographed with a camera and the video data is sent to an emotion recognition library, which analyzes the user's emotional state (happiness, surprise, dissatisfaction, etc.).

[0775] 5. Regenerate the design and display it in a virtual reality environment:

[0776] The server receives the feedback and emotional state, and the interior design AI module regenerates new design options, which are then sent back to the user's device and displayed in real time in the virtual reality environment.

[0777] Examples and prompts

[0778] For example, if a user provides feedback such as "I would like the walls to be a lighter color," and the camera captures the user's face and determines their emotional state as "happy," the interior design AI module might receive the following prompt:

[0779] Example prompt sentence:

[0780] Suggest a modern interior design that best suits the user's joyful moments, taking into account feedback that the wall colors should be lighter.

[0781] This allows optimal designs that reflect the user's emotions to be generated in real time and displayed in a virtual reality environment, increasing the user's desire to purchase.

[0782] This series of processes allows us to provide interior designs that take into account the user's needs and emotions, and by experiencing them realistically through a virtual environment, we can further increase their desire to purchase.

[0783] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0784] Step 1:

[0785] A user takes a photo of the room using a smartphone or a head-mounted display (HMD) and uploads the photo to the server via a dedicated application. In this step, the input is the photo of the room taken by the user, and the output is that the photo is sent to the server.

[0786] Step 2:

[0787] The server receives the uploaded photo and saves the image data in temporary storage. The saved image data is then passed to the AI ​​image analysis module. In this step, the input is the received photo data and the output is the saved image data.

[0788] Step 3:

[0789] The server uses an image analysis module to automatically detect unwanted elements in the image (e.g., cardboard boxes or old furniture) and digitally remove them. The input for this step is the stored image data, and the output is clean image data with unwanted elements removed.

[0790] Step 4:

[0791] The cleaned image data is passed to an interior design AI module to generate multiple interior design options. For example, designs based on preset styles such as "modern," "classic," and "casual" are generated. The input in this step is the cleaned image data, and the output is multiple design options.

[0792] Step 5:

[0793] The server sends the generated design options to the user's terminal. The user's terminal receives the design options and displays them to the user. In this step, the input is the generated design options and the output is the displayed design options.

[0794] Step 6:

[0795] The user reviews the displayed design options and enters their feedback, which can include specific suggestions such as "I would like the wall color to be lighter." The input in this step is the user's feedback, and the output is the content of the feedback.

[0796] Step 7:

[0797] When the user inputs feedback, the camera on the smartphone or HMD captures the user's face, and the video data is sent to the emotion recognition library in real time. The input of this step is the user's facial video data, and the output is the analyzed emotional state.

[0798] Step 8:

[0799] The emotion recognition library detects the user's emotional state (e.g., joy, surprise, dissatisfaction, etc.) and sends the data to the server. The input in this step is facial video data, and the output is emotional state data.

[0800] Step 9:

[0801] The server receives feedback and emotional state data, and the interior design AI module regenerates new design options based on that. For example, if the emotional state is "joy" and feedback is "make the wall color brighter," a prompt sentence is generated to suggest the optimal design. The input of this step is the feedback and emotional state data, and the output is a new design option.

[0802] Step 10:

[0803] The new design options are resent to the user's device and displayed in the virtual reality environment in real time. The user then checks the new design in the VR space. The input of this step is the regenerated design options, and the output is the design options displayed in the VR space.

[0804] This series of processes provides optimal interior designs in real time based on the user's emotions and feedback.

[0805] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0806] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0807] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0808] [Third embodiment]

[0809] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0810] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0811] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0812] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0813] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0814] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0815] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0816] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0817] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0818] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0819] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0820] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0821] ---

[0822] This invention relates to a virtual home staging system for increasing the desire to purchase vacant properties in real estate sales. This system uses AI to generate virtual interior designs based on photos of the interior taken by the user, and the user can interactively modify and adjust them.

[0823] Overall system overview

[0824] This system mainly consists of a user's device, a server, and an AI module. Users take photos of the interior of a room using a device such as a smartphone or tablet and send them to the server via an application. The server analyzes the received photos, automatically removes unnecessary elements, and then generates multiple design options using the interior design AI module. The generated design options are returned to the user's device, and the user provides feedback based on the displayed designs. The server passes this feedback back to the AI, which then generates a new design and provides it to the user.

[0825] Program processing flow (example)

[0826] 1. User photo uploads

[0827] User: Takes a photo of the living room with a smartphone and uploads it to a dedicated application.

[0828] Device: Sends uploaded photos to the server.

[0829] 2. Image Preprocessing

[0830] Server: Saves the received images in temporary storage.

[0831] Server: Uses AI to automatically detect and remove unwanted elements (e.g., cardboard boxes, old furniture, etc.) from images.

[0832] 3. Generate multiple interior design options

[0833] Server: Passes the preprocessed images to the interior design AI module.

[0834] Server: The module generates multiple design options based on preset styles (e.g., modern, classic, casual).

[0835] Server: Stores the generated design options in temporary storage.

[0836] 4. Providing design options

[0837] Server: Sends the generated design options to the user's device.

[0838] Terminal: Displays the received design options to the user.

[0839] 5. User Feedback

[0840] User: Review the displayed design and enter feedback into the application, such as "I'd like the wall colors to be brighter" or "I'd like the furniture to be a little more modern."

[0841] Terminal: Sends user feedback to the server.

[0842] 6. Interior design revisions

[0843] Server: Receives feedback and uses it to generate new design options for the interior design AI module.

[0844] Server: Saves the modified design options in temporary storage and sends them back to the user's device.

[0845] Terminal: Redisplays new design options to the user.

[0846] 7. Finalize the design

[0847] User: If the new design is satisfactory, the user confirms it as the final design, which is then saved in the system and can be reviewed later.

[0848] This series of processes allows users to easily and interactively experiment with interior designs to find the optimal design. This system is expected to promote real estate sales.

[0849] The processing flow will be explained below.

[0850] ---

[0851] Step 1:

[0852] User: Take a photo of the living room with a smartphone, launch the dedicated application and log in.

[0853] Device: Tap the "Upload Photo" button on the application screen and select the photo you have taken.

[0854] On the device: Send the selected photo to the backend server.

[0855] Step 2:

[0856] Server: Stores the received photo data in temporary storage.

[0857] Server: Passes the saved image data to the AI ​​image analysis module.

[0858] Server: The image analysis module automatically detects unwanted elements in the image (e.g., cardboard boxes, old furniture) and digitally removes them.

[0859] Step 3:

[0860] Server: Obtains clean image data with unnecessary elements removed and passes it to the interior design AI module.

[0861] Server: The interior design AI module generates multiple design options based on preset styles such as "modern," "classic," and "casual."

[0862] Server: Saves each generated design option to temporary storage.

[0863] Step 4:

[0864] Server: Creates a response for sending image data from the saved multiple design options to the user's terminal.

[0865] Terminal: Receives image data of design options sent from the server and displays it to the user.

[0866] Step 5:

[0867] User: Review the design options presented and provide specific feedback if necessary (e.g., "I'd like the walls to be a lighter color" or "I'd like the furniture to be a little more modern").

[0868] Terminal: Makes a request to send the user's input feedback to the server.

[0869] Step 6:

[0870] Server: Analyzes the received feedback and passes it to the interior design AI module.

[0871] Server: The interior design AI module regenerates new design options based on feedback.

[0872] Server: Saves the newly generated design options to temporary storage.

[0873] Step 7:

[0874] Server: Creates a response to send the new design options to the user's device.

[0875] Terminal: Re-display new design options received to the user.

[0876] User: Checks the new design and, if satisfied, provides input to confirm the final design.

[0877] Terminal: Sends final design confirmation information to the server.

[0878] ---

[0879] These are the specific processing steps of the program for the entire system, which allow users to interactively customize the interior design and realize a virtual home staging that reflects their ideals.

[0880] Example 1

[0881] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0882] In real estate sales, there is a demand for methods to visually enhance the appeal of vacant properties. Installing realistic interior designs is costly and time-consuming, and potential buyers have difficulty forming a concrete image of the property. Conventional methods make it difficult for users to select the design they want through trial and error, which discourages them from purchasing.

[0883] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0884] In this invention, the server includes means for uploading interior photos taken by a user, means for automatically erasing unnecessary elements included in the uploaded photos by an artificial intelligence, means for the interior design artificial intelligence to generate multiple design options based on the erased images, means for providing the generated multiple design options to the user's information processing device, means for receiving user feedback and modifying the interior design based on the feedback, means for regenerating new design options based on the user's feedback, means for saving the generated design options in online storage, and means for providing the regenerated design options again to the user's information processing device, thereby enabling the user to interactively experiment with interior designs and find the optimal design.

[0885] "User" refers to a person who uses this system to take photos of an interior and receive interior design suggestions.

[0886] "Server" refers to a computing device that receives users' photos and uses artificial intelligence to process the images and generate designs.

[0887] "Information processing device" refers to an electronic device (e.g., smartphone, tablet, PC) that a user uses to take photos and receive design suggestions from a server.

[0888] "Unwanted elements" refer to objects or backgrounds (e.g., cardboard boxes, old furniture) that are included in the interior photos taken by the user but should be removed when proposing interior designs.

[0889] "Artificial intelligence" refers to technology that automates image processing and design generation using large amounts of data and machine learning algorithms.

[0890] "Interior design artificial intelligence" refers to a specific artificial intelligence model used to automatically generate interior design options from a user's photos.

[0891] "Design options" refers to multiple different interior design proposals generated by the interior design AI.

[0892] "Online storage" refers to a data storage service that allows you to store data via the Internet and access it as needed.

[0893] "Feedback" refers to specific opinions and change requests that users give regarding design options.

[0894] This invention is a virtual home staging system for visually enhancing the appeal of vacant properties in real estate sales. This system uses artificial intelligence (AI) to generate virtual interior designs based on interior photos taken by the user, and allows the user to interactively modify and adjust them.

[0895] Overall system overview

[0896] This system primarily consists of the user's information processing device (smartphone, tablet, PC, etc.), a server, and an interior design AI module. The user uses the information processing device to take photos of the interior and sends them to the server via a dedicated application. The server analyzes the received photos, automatically removes unnecessary elements, and then uses the interior design AI module to generate multiple design options. The generated design options are provided to the user's information processing device, and the user provides feedback based on the displayed designs. The server passes this feedback back to the AI ​​module, which then generates a new design and provides it to the user.

[0897] Hardware and software used

[0898] Hardware:

[0899] Information processing devices (smartphones, tablets, PCs, etc.)

[0900] Server (for data processing and storage)

[0901] software:

[0902] Dedicated application (installed on the user's information processing device)

[0903] Server-side software (online storage such as Amazon S3, Python scripts, OpenCV, TensorFlow-based interior design AI module)

[0904] Specific examples of processing

[0905] Example 1: User uploads a photo

[0906] A user takes a photo of their living room with their smartphone and uploads it to the server using a dedicated application. The user presses the upload button in the application to start sending the photo. The server temporarily stores the received photo in an Amazon S3 bucket.

[0907] Example 2: Removing unnecessary elements

[0908] The server uses the OpenCV library to analyze the stored photos, automatically detecting unwanted elements in the photo, such as cardboard boxes or old furniture, and identifying them through masking. It then applies an inpainting technique to remove the unwanted elements from the image.

[0909] Example 3: Generating interior designs

[0910] The preprocessed images are input into a TensorFlow-based interior design AI module, which generates multiple design options based on pre-trained design patterns (e.g., modern, classic, casual).

[0911] Example prompt sentence:

[0912] For example, if a user "wants a modern style living room," they would enter the following as the prompt:

[0913] For example: "Please add a modern style interior design to the living room in the photo. I would like light wall colors and simple furniture."

[0914] This system allows users to interactively experiment with interior designs to find the best fit, and is expected to help promote real estate sales.

[0915] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0916] Step 1:

[0917] A user takes a photo of their living room with their smartphone and uploads it via a dedicated application. The input is the captured photo data in JPEG format, and the output is an upload request to the server via the application. When the upload button is pressed within the application, the photo data is sent over the Internet to a specified server URL.

[0918] Step 2:

[0919] The device receives the uploaded photo data and temporarily stores it on the device. The input is the JPEG photo data uploaded by the user, and the output is the operation of temporarily storing this data. The stored photo data is ready to be transferred to the server.

[0920] Step 3:

[0921] The server saves the received photo data to storage. An online storage service (e.g., Amazon S3 bucket) is used for saving. The input is the JPEG photo data sent from the device, and the output is the URL of the photo data saved in storage. This URL is used in subsequent processing.

[0922] Step 4:

[0923] The server uses the OpenCV library to erase unnecessary elements in the image. The input is photo data retrieved from storage, and the output is new image data with unnecessary elements erased. Specifically, OpenCV is used to detect cardboard boxes and old furniture in the image and apply inpainting techniques to erase them.

[0924] Step 5:

[0925] The server inputs the preprocessed image data into an interior design AI model. This AI model is built with TensorFlow and uses pre-trained design patterns. The input is image data with unnecessary elements removed, and the output is multiple interior design options. The model analyzes the image and generates design options based on styles such as "modern," "classic," and "casual."

[0926] Step 6:

[0927] The server stores the generated design options in online storage and provides them to the user's information processing device. The input is multiple design options generated by the interior design AI model, and the output is a list of URLs stored in online storage. This list of URLs is then sent to the user's device.

[0928] Step 7:

[0929] The device receives a list of design option URLs and displays each design option in the user interface. The input is the URL list received from the server, and the output is an image of the design option that the user can view. The user can swipe to switch between multiple design options.

[0930] Step 8:

[0931] The user enters feedback on the displayed design options. For example, specific opinions such as "make the wall color lighter" or "rearrange the furniture" are entered into the text input field. The input is the user's feedback text, and the output is a request to send the feedback information to the server.

[0932] Step 9:

[0933] The server analyzes the received feedback and re-inputs it into the interior design AI model. The input is the feedback data received from the user, and the output is the newly generated design options. For example, in response to the feedback "make the wall color lighter," a color conversion process is performed to generate a new interior design.

[0934] Step 10:

[0935] The server saves the newly generated design options in online storage again and provides them to the user's information processing device. The input is the newly generated design options, and the output is a list of URLs saved in online storage. This list of URLs is then sent to the user's device again.

[0936] Step 11:

[0937] The device again receives new design options and displays them in the user interface. The input is the new URL list received from the server, and the output is an image of the latest design options that the user can view. The user again reviews the designs and continues to provide feedback as needed.

[0938] Step 12:

[0939] When the user is satisfied with the design, they press the confirm button in the app to confirm it as the final design. The input is the design option that the user confirms, and the output is the operation of sending the confirmed design to the server and saving it in the database.

[0940] This series of processes allows the user to interactively experiment with interior designs and find the optimal design.

[0941] (Application example 1)

[0942] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0943] In real estate sales, there are limited means to promote vacant properties, making it difficult for potential buyers to visualize the interior of the room. Furthermore, to realize virtual home staging, a flexible system is required that allows users to easily reflect their own preferences and requests, but conventional technology has difficulty meeting this requirement. Furthermore, the same issue exists in virtual stores, as there are not enough functions to design interiors in virtual spaces and save and display the results.

[0944] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0945] In this invention, the server includes a means for uploading interior photos taken by a user, a means for an AI to automatically delete unnecessary elements from the uploaded photos, a means for the interior design AI to generate multiple design options based on the deleted images, a means for accepting user feedback and modifying the interior design based on the feedback, a means for re-receiving user feedback and regenerating the interior design based on the feedback, and a means for saving and displaying the finalized design in the virtual store. This allows users to design interiors in a virtual space based on photos of their actual rooms and to find their ideal interior by repeatedly modifying and regenerating the design. Furthermore, using the finalized design in the virtual store can promote real estate sales and improve the convenience of interior customization in virtual stores.

[0946] "User" refers to the end user who uses the system to upload interior photos and provide feedback on interior design.

[0947] "Uploading means" refers to the interface and communication means for sending photos taken by the user to the server.

[0948] "Means for AI to automatically remove unnecessary elements" refers to a function that uses AI technology to automatically analyze and remove unnecessary objects and items from uploaded photos.

[0949] "Interior Design AI" refers to an artificial intelligence module that generates multiple design options based on a cleaned-up image.

[0950] "Design options" refers to multiple interior design options generated by the interior design AI.

[0951] "User feedback" refers to the user's evaluation of and requests for modifications to the provided design options.

[0952] "Means to modify interior design" refers to the function of reviewing and regenerating interior design based on user feedback.

[0953] A "virtual store" refers to a virtual shopping or exhibition space where products and interiors can be arranged and customized within a virtual space.

[0954] "Final design" refers to the interior design that is finalized as a result of incorporating user feedback.

[0955] "Means for saving and displaying" refers to the function for saving the finalized design within the system and displaying it within the virtual store.

[0956] This invention relates to a virtual home staging system that uses AI to generate a virtual interior design based on indoor photos taken by the user, and allows the user to interactively modify and adjust the design.

[0957] Overall system overview

[0958] The system primarily consists of a user device, a server, and an interior design AI module. Users take photos of the interior using a device such as a smartphone or virtual glasses and send them to the server via an application. The server analyzes the received photos, automatically removes unnecessary elements, and then generates multiple design options using the interior design AI module. The generated design options are returned to the user's device, and the user provides feedback based on the displayed designs. The server passes this feedback back to the AI, which then generates a new design and provides it to the user. The finalized design is saved and displayed in the virtual store.

[0959] Hardware / Software used

[0960] Hardware: Smartphone, virtual glasses, and server

[0961] Software: Python, TensorFlow, OpenCV, Flask

[0962] Program processing details

[0963] 1. User photo uploads

[0964] Users can take photos of the interior of a room using a smartphone app and upload them to a server via a dedicated application.

[0965] 2. Image preprocessing by the server

[0966] The server stores the received images in storage and uses OpenCV to automatically detect and remove unwanted elements (e.g., cardboard boxes, old furniture, etc.) from the photos, generating a clean base image.

[0967] 3. Interior design generation

[0968] The server passes the preprocessed images to an interior design AI module using TensorFlow, which generates multiple design options based on preset styles such as "modern," "classic," and "casual." The generated design options are stored in temporary storage and sent to the user's device.

[0969] 4. User Feedback

[0970] The user reviews the displayed design options and enters specific feedback into the application, such as "I'd like the wall colors to be brighter" or "I'd like the furniture to be a little more modern." This feedback is then sent to the server.

[0971] 5. Interior design revisions

[0972] The server then uses the user's feedback to generate new design options using the interior design AI module, and this process is repeated until the user is satisfied.

[0973] 6. Finalize the design and display it in the virtual store

[0974] Once the user is satisfied with the design, it is saved and displayed in the virtual store, allowing them to visualize how the interior design will look in a real room in a virtual space.

[0975] Specific examples

[0976] For example, if a user submits a photo of their living room and requests a "modern design," the AI ​​will generate multiple design options incorporating elements such as "light-colored walls," "simple modern furniture," and "spacious layout." A specific example of a prompt would be, "I have submitted a photo of my living room. Please generate a modern interior design that suits this room. I would like the furniture to be simple and the colors to be light." By sending this instruction to the server, a design based on the user's requests will be generated.

[0977] As described above, this system allows users to easily experiment with interior designs to find their ideal design. The final design can also be saved and displayed in the virtual store, contributing to promoting real estate sales and improving the user experience.

[0978] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0979] Step 1:

[0980] User-uploaded photos

[0981] Users take photos of the interior of a room using a smartphone app and upload them to the server through a dedicated application. The input is the photo of the room taken with the user's smartphone, and the output is the photo file sent to the server. Specifically, when the "upload photo" button is pressed on the application, the photo is automatically sent to the server.

[0982] Step 2:

[0983] Image preprocessing by the server

[0984] The server stores the received image file in temporary storage and uses the OpenCV library to automatically detect and remove unwanted elements (e.g., cardboard boxes, old furniture, etc.) in the photo. The input is the image file stored on the server, and the output is a clean image with unwanted elements removed. Specifically, the server converts the image to grayscale, performs binary thresholding to mask and remove unwanted objects.

[0985] Step 3:

[0986] Interior Design Generation

[0987] The server sends the preprocessed image to the interior design AI module, which uses TensorFlow to generate multiple design options, such as "modern," "classic," and "casual." The input is clean image data, and the output is multiple interior design options. Specifically, the server passes the image to the AI ​​module, which generates multiple design options based on preset styles.

[0988] Step 4:

[0989] User Feedback

[0990] The user checks the generated design options on a smartphone app and provides feedback such as "change the wall color to a lighter color" or "add more modern furniture." The input is the user's feedback data, and the output is the feedback information sent to the server. Specifically, when the user presses the "Send Feedback" button on the app, the comment is sent to the server.

[0991] Step 5:

[0992] Interior design modifications

[0993] The server receives feedback from the user, and the interior design AI module regenerates new design options based on that feedback. The input is the user's feedback data and preprocessed images, and the output is a new design option modified based on the feedback. Specifically, the server passes the feedback to the AI ​​and saves the regenerated design option in temporary storage.

[0994] Step 6:

[0995] Finalize the design and display it in the virtual store

[0996] The user finally decides on a design that satisfies them, and that design is saved and displayed in the virtual store. The input is the final design data decided by the user, and the output is the final design saved in the virtual store system. Specifically, when the user presses the "Confirm Design" button on the app, the design is saved in the virtual store and becomes available for the user to view in the virtual space.

[0997] This series of processes allows users to find their ideal interior design through trial and error, and ultimately to check the design in a virtual store.

[0998] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0999] ---

[1000] This invention relates to a virtual home staging system for increasing the desire to purchase vacant properties in real estate sales. This system uses AI to generate virtual interior designs based on interior photos taken by the user, which the user can then interactively modify and adjust. In addition, by combining it with an emotion engine that recognizes the user's emotions, the user's emotions can be reflected in the design generation.

[1001] Overall system overview

[1002] This system consists of a user's device, a server, an interior design AI, and an emotion engine. Users take photos of the interior using a device such as a smartphone or tablet and send them to the server via an application. The server analyzes the received photos, automatically removes unnecessary elements, and then uses the interior design AI to generate multiple design options. The generated design options are returned to the user's device, and the user provides feedback based on the displayed designs. The emotion engine then analyzes the user's emotions and reflects the results in the interior design AI to generate new designs.

[1003] Program processing flow (example)

[1004] User-uploaded photos

[1005] User: Take a photo of the living room with a smartphone, launch the dedicated application and log in.

[1006] Device: Tap the "Upload Photo" button on the application screen and select the photo you have taken.

[1007] On the device: Send the selected photo to the backend server.

[1008] Image preprocessing

[1009] Server: Stores the received photo data in temporary storage.

[1010] Server: Passes the saved image data to the AI ​​image analysis module.

[1011] Server: The image analysis module automatically detects unwanted elements in the image (e.g., cardboard boxes, old furniture) and digitally removes them.

[1012] Generate multiple interior design options

[1013] Server: Obtains clean image data with unnecessary elements removed and passes it to the interior design AI module.

[1014] Server: The interior design AI module generates multiple design options based on preset styles such as "modern," "classic," and "casual."

[1015] Server: Saves each generated design option to temporary storage.

[1016] Providing design options

[1017] Server: Creates a response for sending image data from the saved multiple design options to the user's terminal.

[1018] Terminal: Receives image data of design options sent from the server and displays it to the user.

[1019] User Feedback

[1020] User: Review the design options presented and provide specific feedback if necessary (e.g., "I'd like the walls to be a lighter color" or "I'd like the furniture to be a little more modern").

[1021] Terminal: Makes a request to send the user's input feedback to the server.

[1022] Manipulating the Emotion Engine

[1023] Terminal: When inputting feedback, the user's face is photographed with a camera and the image is sent to the emotion engine in real time.

[1024] Emotion engine: Analyzes the user's facial expressions to detect the user's emotional state (e.g., joy, surprise, dissatisfaction).

[1025] Emotion engine: Sends the detected emotional state and feedback content to the server.

[1026] Interior design modifications

[1027] Server: Receives feedback and emotional state, and the interior design AI module regenerates new design options based on that.

[1028] Server: Saves the newly generated design options in temporary storage and sends them back to the user's device.

[1029] Terminal: Re-display new design options received to the user.

[1030] Final design confirmation

[1031] User: If the new design is satisfactory, the user confirms it as the final design, which is then saved in the system and can be reviewed later.

[1032] This series of processes allows users to customize interior designs that take their emotions into account, enabling virtual home staging that is more suited to the user. This system can increase purchasing motivation and maximize the appeal of the property.

[1033] The processing flow will be explained below.

[1034] ---

[1035] Step 1:

[1036] User: Launch the dedicated application on your smartphone and log in.

[1037] User: Takes a photo of the living room with his smartphone.

[1038] Step 2:

[1039] Device: Tap the "Upload Photo" button on the application screen and select the photo you have taken.

[1040] Terminal: Compress the selected photo data and send it to the server.

[1041] Step 3:

[1042] Server: Stores the received photo data in temporary storage.

[1043] Server: Passes the saved image data to the image analysis module.

[1044] Step 4:

[1045] Server: The image analysis module analyzes the image and automatically detects unwanted elements (e.g., old furniture or cardboard boxes).

[1046] Server: Digitally erases detected unwanted elements to generate clean image data.

[1047] Step 5:

[1048] Server: Passes the clean image data to the interior design AI module and instructs it to generate multiple design options.

[1049] Server: The interior design AI module generates multiple design options based on preset styles such as "modern," "classic," and "casual."

[1050] Step 6:

[1051] Server: Stores the generated design options in temporary storage and creates a response to send to the user's device.

[1052] Terminal: Receives image data of design options sent from the server and displays it to the user.

[1053] Step 7:

[1054] User: Review each design option presented and provide feedback (e.g., "I'd like the walls to be a lighter color" or "I'd like the furniture to be a little more modern").

[1055] Device: When inputting feedback, the user's face is photographed with the smartphone camera and the image is sent to the emotion engine in real time.

[1056] Step 8:

[1057] Emotion engine: Analyzes the user's facial expressions to detect their emotional state (e.g., happiness, surprise, dissatisfaction).

[1058] Emotion engine: Sends the detected emotional state and feedback content to the server.

[1059] Step 9:

[1060] Server: Analyzes the received feedback and emotional state and instructs the interior design AI module to regenerate new design options based on that.

[1061] Server: The interior design AI module generates new design options based on user feedback and sentiment.

[1062] Step 10:

[1063] Server: Stores the newly generated design options in temporary storage and creates a response to send to the user's device.

[1064] Terminal: Re-display new design options received to the user.

[1065] Step 11:

[1066] Users: Review the new design options and provide feedback again if needed.

[1067] User: When a satisfactory design is confirmed, it is confirmed as the final design and saved in the system.

[1068] ---

[1069] This series of processes enables customization of interior designs that take emotion analysis into account, realizing optimal virtual home staging for each user. This system makes it possible to propose designs that reflect the user's emotions, maximizing the appeal of the property.

[1070] Example 2

[1071] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1072] Conventional interior design systems have had the drawback of requiring the user to adjust the design themselves, and of making it difficult to generate a design that reflects the user's emotions. This has resulted in the problem that it takes a lot of time and effort for the user to obtain a design that actually satisfies them. The objective of the present invention is to provide a system that can easily generate an interior design that meets the user's needs by analyzing the user's emotional state and reflecting that in the design.

[1073] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for uploading images of an interior taken by a user, means for automatically erasing unnecessary elements contained in the uploaded images using artificial intelligence, means for generating a plurality of interior design options based on the erased images, means for providing the generated plurality of design options to the user's information processing device, means for receiving user feedback and modifying the interior design based on the feedback, and means for analyzing the user's emotions and reflecting the user's emotional state in the design generation. This makes it possible to generate an interior design that takes the user's emotions into consideration, thereby improving user satisfaction, stimulating purchasing motivation, and maximizing the appeal of the property.

[1074] ---

[1075] That's all.

[1076] ---

[1077] "User" refers to a person who uses this system.

[1078] "Interior images" refer to digital images of interior spaces such as homes and commercial facilities.

[1079] "Uploading" refers to the act of sending data from a user's device to a server.

[1080] "Artificial intelligence" refers to computer programs and systems that automatically perform advanced processes such as data analysis and image processing.

[1081] "Unwanted elements" refer to objects or elements that are present in the image but should be removed when generating the interior design.

[1082] "Interior design options" refers to multiple interior design proposals generated based on a specific style or theme.

[1083] "Information processing device" refers to digital devices used by users, such as computers, smartphones, and tablets.

[1084] "Feedback" refers to user-provided comments and requests regarding the design.

[1085] "Analyzing emotions" refers to the process of determining a user's emotional state from their facial expressions and behavior.

[1086] An "emotional state" refers to the emotion (e.g., joy, surprise, dissatisfaction, etc.) that a user is feeling at a particular point in time.

[1087] ---

[1088] That's all.

[1089] ---

[1090] This invention relates to a virtual home staging system for increasing the desire to purchase vacant properties in real estate sales. This system uses artificial intelligence to generate virtual interior designs based on images of the interior taken by the user, which the user can then interactively modify and adjust. In addition, by combining it with an emotion engine that recognizes the user's emotions, the user's emotions can be reflected in the design generation.

[1091] The overall system consists of a user's device, a server, an interior design AI, and an emotion engine. Users take photos of the interior using a device such as a smartphone or tablet and send them to the server via a dedicated application. The server analyzes the received images, automatically removes unnecessary elements, and then uses the interior design AI to generate multiple design options. These design options are sent back to the user's device, and the user provides feedback based on the displayed designs. The emotion engine then analyzes the user's emotions and reflects the results in the interior design AI to generate new designs.

[1092] The specific hardware used is a smartphone or tablet. These devices communicate with the server through a dedicated application (for example, an app developed with React Native). The server-side software uses Python, OpenCV, Amazon S3, and a TensorFlow-based neural network model. The emotion engine uses Microsoft's Azure Emotion API.

[1093] Furthermore, to allow users to provide more specific feedback, the system can generate interior designs using multiple preset styles (e.g., "Modern," "Classic," and "Casual"), allowing users to easily select and adjust design options according to their preferences.

[1094] As a concrete example, consider a user taking and uploading a photo of their living room. The user launches the application, enters their login information, and logs in to their account. Next, they tap the app's "Upload Photo" button and select a photo they took from their device's gallery. The selected photo is automatically sent to the server, which performs image analysis and removes unnecessary elements such as cardboard boxes and old furniture. The interior design AI then generates multiple design options and sends them back to the user's device. The user reviews these designs and provides specific feedback. Along with the feedback, the emotion engine analyzes the user's facial expressions and incorporates their emotional state into the generated design.

[1095] Here are some example prompts to input to a generative AI model:

[1096] "Users upload photos of their living rooms, and AI eliminates unnecessary elements and generates multiple design options. We've built a system that modifies and optimizes the design based on user feedback and emotional state."

[1097] By inputting this prompt into the generative AI model, you can get specific advice on how to implement the above processing flow in detail and on optimization points.

[1098] ---

[1099] That's all.

[1100] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1101] ---

[1102] Step 1: User takes and uploads a photo

[1103] User: Takes a photo of their living room with their smartphone. The input is a high-quality digital image.

[1104] User: Launches the dedicated application, enters login information, and logs in to the account. Next, taps the "Upload Photo" button, selects a photo from the gallery, and sends it to the server. The output is the image data sent to the server.

[1105] Step 2: Image preprocessing

[1106] Server: Stores the received image data in temporary storage (e.g., cloud storage). The input is the image data sent by the user.

[1107] Server: The stored image data is passed to the image analysis module using Python and the OpenCV library. The output is clean image data with unnecessary elements removed.

[1108] How it works: OpenCV's contour detection function is used to detect cardboard boxes and old furniture in the image, and these unwanted elements are then removed using masking techniques.

[1109] Step 3: Generate interior design options

[1110] Server: Passes the clean image data to the interior design AI module using a TensorFlow-based neural network. The input is the preprocessed clean image data.

[1111] Server: The interior design AI module generates multiple design options (e.g., "modern," "classic," "casual"). The output is multiple interior design images.

[1112] How it works: The neural network model analyzes the input image and generates different interior designs based on each preset style.

[1113] Step 4: Providing design options

[1114] Server: Creates a response to send the generated multiple design options to the user's terminal. The input is the generated design options.

[1115] Terminal: Receives image data of the design options sent from the server and displays it to the user. The output is the design options displayed on the user terminal.

[1116] What it does: Use React Native to build an interface that lets users explore design options.

[1117] Step 5: User Provides Feedback

[1118] User: Review the displayed design options and enter specific changes in the app's feedback form. The input is the user's feedback.

[1119] Terminal: Generates and sends a request to the server to send the user's input feedback. The output is the feedback information.

[1120] Step 6: Manipulating the Emotion Engine

[1121] Terminal: When inputting feedback, the user's facial expression is captured by a camera and the video is sent to the emotion engine in real time. The input is video data of the user's facial expression.

[1122] Emotion Engine: Analyzes the user's facial expressions to detect the user's emotional state. The output is the user's emotional state data.

[1123] Emotion engine: The detected emotional state and feedback content are sent to the server. The output is integrated data of the emotional state and feedback.

[1124] Specific operation: Uses Microsoft's Azure Emotion API to analyze the user's emotional state from their facial expressions in real time.

[1125] Step 7: Interior design revisions

[1126] Server: The interior design AI module regenerates new design options based on the received feedback and emotional state. The input is the feedback and emotional state data.

[1127] Server: The regenerated design options are stored in temporary storage and sent to the user's device again. The output is the modified design options.

[1128] What it does: The neural network model recreates the design, taking into account feedback and emotional data.

[1129] Step 8: Finalize the design

[1130] User: If the new design is satisfactory, confirm it as the final design. Input is the selection information for the final design.

[1131] Server: Saves the final design to the system so that it can be reviewed later. The output is the saved final design.

[1132] ---

[1133] That's all.

[1134] (Application example 2)

[1135] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1136] Conventional interior design systems have difficulty reflecting the individual emotions and reactions of users, making it impossible to provide optimal designs that meet the user's needs. Furthermore, it is difficult to perfectly reproduce the interior experience in the real world, making it impossible to sufficiently stimulate the user's purchasing motivation. There is a need to solve these problems and provide optimal interior proposals by allowing users to experience designs in real time in a virtual reality environment while taking into account their emotions.

[1137] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1138] In this invention, the server includes means for uploading interior photos taken by a user, means for an AI to automatically delete unnecessary elements included in the uploaded photos, means for the interior design AI to generate multiple design options based on the deleted images, means for providing the generated multiple design options to the user's device, means for receiving user feedback and modifying the interior design based on the feedback, means for analyzing the user's emotional state and modifying the interior design based on the analysis results, and means for displaying the generated interior design options in a virtual reality environment. This makes it possible to provide interior designs that reflect the user's emotions, and further to stimulate greater purchasing motivation through a real-time experience in a virtual reality environment.

[1139] "User's device" refers to a device used by a user to input and display information, such as a smartphone, head-mounted display (HMD), or tablet.

[1140] "AI automatically removes unnecessary elements" is a process that uses artificial intelligence technology to automatically detect and remove unnecessary elements such as cardboard boxes and old furniture from indoor photos.

[1141] "Interior Design AI" is an artificial intelligence algorithm for generating interior designs and layouts, and has the ability to generate optimal design options taking into account user feedback and emotional state.

[1142] "Design options" are multiple interior design options generated by the interior design AI, including different styles and layouts.

[1143] "User feedback" refers to specific opinions and requests that users input regarding interior design options, such as "make the wall colors brighter" or "use more modern furniture."

[1144] "User emotional state" refers to the emotions the user feels when reviewing the design, such as joy, surprise, or dissatisfaction, which are analyzed in real time using artificial intelligence technology.

[1145] A "virtual reality environment" is a three-dimensional virtual space that users can virtually experience using a head-mounted display (HMD), allowing them to check and adjust interior design options in real time.

[1146] "Display in real time" means that the generated interior design options are displayed without delay so that the user can check them immediately.

[1147] The present invention relates to a virtual home staging system for increasing the willingness to purchase vacant properties in real estate sales. Specific embodiments of the system are described below.

[1148] Hardware and software used

[1149] Hardware:

[1150] Smartphones (latest iPhones and Android devices)

[1151] Head-mounted displays (HMDs) (e.g., Oculus Quest 2)

[1152] Camera (built-in smartphone or HMD camera)

[1153] software:

[1154] User device application (compatible with iOS / Android)

[1155] Emotion recognition libraries (e.g., Emotion AI SDK)

[1156] Interior design AI module (e.g., OpenAI GPT-4)

[1157] Backend servers (e.g., AWS, Google Cloud)

[1158] Data processing and calculation

[1159] 1. Image capture and transmission:

[1160] The user takes a photo of the room using a smartphone or the camera built into the HMD, and the image data is sent to a back-end server.

[1161] 2. Image analysis and removal of unwanted elements:

[1162] The server receives the image data and uses an AI image analysis module to automatically detect and remove unwanted elements (such as cardboard boxes, old furniture, etc.), resulting in a clean image.

[1163] 3. Interior design generation:

[1164] The server passes the clean image data to an interior design AI module, which generates multiple design options (e.g., "modern," "classic," "casual," etc.).

[1165] 4. User feedback and sentiment analysis:

[1166] The user reviews the generated design options via a smartphone or HMD and enters their feedback. When entering feedback, the user's face is photographed with a camera and the video data is sent to an emotion recognition library, which analyzes the user's emotional state (happiness, surprise, dissatisfaction, etc.).

[1167] 5. Regenerate the design and display it in a virtual reality environment:

[1168] The server receives the feedback and emotional state, and the interior design AI module regenerates new design options, which are then sent back to the user's device and displayed in real time in the virtual reality environment.

[1169] Examples and prompts

[1170] For example, if a user provides feedback such as "I would like the walls to be a lighter color," and the camera captures the user's face and determines their emotional state as "happy," the interior design AI module might receive the following prompt:

[1171] Example prompt sentence:

[1172] Suggest a modern interior design that best suits the user's joyful moments, taking into account feedback that the wall colors should be lighter.

[1173] This allows optimal designs that reflect the user's emotions to be generated in real time and displayed in a virtual reality environment, increasing the user's desire to purchase.

[1174] This series of processes allows us to provide interior designs that take into account the user's needs and emotions, and by experiencing them realistically through a virtual environment, we can further increase their desire to purchase.

[1175] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1176] Step 1:

[1177] A user takes a photo of the room using a smartphone or a head-mounted display (HMD) and uploads the photo to the server via a dedicated application. In this step, the input is the photo of the room taken by the user, and the output is that the photo is sent to the server.

[1178] Step 2:

[1179] The server receives the uploaded photo and saves the image data in temporary storage. The saved image data is then passed to the AI ​​image analysis module. In this step, the input is the received photo data and the output is the saved image data.

[1180] Step 3:

[1181] The server uses an image analysis module to automatically detect unwanted elements in the image (e.g., cardboard boxes or old furniture) and digitally remove them. The input for this step is the stored image data, and the output is clean image data with unwanted elements removed.

[1182] Step 4:

[1183] The cleaned image data is passed to an interior design AI module to generate multiple interior design options. For example, designs based on preset styles such as "modern," "classic," and "casual" are generated. The input in this step is the cleaned image data, and the output is multiple design options.

[1184] Step 5:

[1185] The server sends the generated design options to the user's terminal. The user's terminal receives the design options and displays them to the user. In this step, the input is the generated design options and the output is the displayed design options.

[1186] Step 6:

[1187] The user reviews the displayed design options and enters their feedback, which can include specific suggestions such as "I would like the wall color to be lighter." The input in this step is the user's feedback, and the output is the content of the feedback.

[1188] Step 7:

[1189] When the user inputs feedback, the camera on the smartphone or HMD captures the user's face, and the video data is sent to the emotion recognition library in real time. The input of this step is the user's facial video data, and the output is the analyzed emotional state.

[1190] Step 8:

[1191] The emotion recognition library detects the user's emotional state (e.g., joy, surprise, dissatisfaction, etc.) and sends the data to the server. The input in this step is facial video data, and the output is emotional state data.

[1192] Step 9:

[1193] The server receives feedback and emotional state data, and the interior design AI module regenerates new design options based on that. For example, if the emotional state is "joy" and feedback is "make the wall color brighter," a prompt sentence is generated to suggest the optimal design. The input of this step is the feedback and emotional state data, and the output is a new design option.

[1194] Step 10:

[1195] The new design options are resent to the user's device and displayed in the virtual reality environment in real time. The user then checks the new design in the VR space. The input of this step is the regenerated design options, and the output is the design options displayed in the VR space.

[1196] This series of processes provides optimal interior designs in real time based on the user's emotions and feedback.

[1197] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1198] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1199] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1200] [Fourth embodiment]

[1201] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1202] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1203] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1204] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1205] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1206] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1207] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1208] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1209] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1210] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1211] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1212] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1213] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1214] ---

[1215] This invention relates to a virtual home staging system for increasing the desire to purchase vacant properties in real estate sales. This system uses AI to generate virtual interior designs based on photos of the interior taken by the user, and the user can interactively modify and adjust them.

[1216] Overall system overview

[1217] This system mainly consists of a user's device, a server, and an AI module. Users take photos of the interior of a room using a device such as a smartphone or tablet and send them to the server via an application. The server analyzes the received photos, automatically removes unnecessary elements, and then generates multiple design options using the interior design AI module. The generated design options are returned to the user's device, and the user provides feedback based on the displayed designs. The server passes this feedback back to the AI, which then generates a new design and provides it to the user.

[1218] Program processing flow (example)

[1219] 1. User photo uploads

[1220] User: Takes a photo of the living room with a smartphone and uploads it to a dedicated application.

[1221] Device: Sends uploaded photos to the server.

[1222] 2. Image Preprocessing

[1223] Server: Saves the received images in temporary storage.

[1224] Server: Uses AI to automatically detect and remove unwanted elements (e.g., cardboard boxes, old furniture, etc.) from images.

[1225] 3. Generate multiple interior design options

[1226] Server: Passes the preprocessed images to the interior design AI module.

[1227] Server: The module generates multiple design options based on preset styles (e.g., modern, classic, casual).

[1228] Server: Stores the generated design options in temporary storage.

[1229] 4. Providing design options

[1230] Server: Sends the generated design options to the user's device.

[1231] Terminal: Displays the received design options to the user.

[1232] 5. User Feedback

[1233] User: Review the displayed design and enter feedback into the application, such as "I'd like the wall colors to be brighter" or "I'd like the furniture to be a little more modern."

[1234] Terminal: Sends user feedback to the server.

[1235] 6. Interior design revisions

[1236] Server: Receives feedback and uses it to generate new design options for the interior design AI module.

[1237] Server: Saves the modified design options in temporary storage and sends them back to the user's device.

[1238] Terminal: Redisplays new design options to the user.

[1239] 7. Finalize the design

[1240] User: If the new design is satisfactory, the user confirms it as the final design, which is then saved in the system and can be reviewed later.

[1241] This series of processes allows users to easily and interactively experiment with interior designs to find the optimal design. This system is expected to promote real estate sales.

[1242] The processing flow will be explained below.

[1243] ---

[1244] Step 1:

[1245] User: Take a photo of the living room with a smartphone, launch the dedicated application and log in.

[1246] Device: Tap the "Upload Photo" button on the application screen and select the photo you have taken.

[1247] On the device: Send the selected photo to the backend server.

[1248] Step 2:

[1249] Server: Stores the received photo data in temporary storage.

[1250] Server: Passes the saved image data to the AI ​​image analysis module.

[1251] Server: The image analysis module automatically detects unwanted elements in the image (e.g., cardboard boxes, old furniture) and digitally removes them.

[1252] Step 3:

[1253] Server: Obtains clean image data with unnecessary elements removed and passes it to the interior design AI module.

[1254] Server: The interior design AI module generates multiple design options based on preset styles such as "modern," "classic," and "casual."

[1255] Server: Saves each generated design option to temporary storage.

[1256] Step 4:

[1257] Server: Creates a response for sending image data from the saved multiple design options to the user's terminal.

[1258] Terminal: Receives image data of design options sent from the server and displays it to the user.

[1259] Step 5:

[1260] User: Review the design options presented and provide specific feedback if necessary (e.g., "I'd like the walls to be a lighter color" or "I'd like the furniture to be a little more modern").

[1261] Terminal: Makes a request to send the user's input feedback to the server.

[1262] Step 6:

[1263] Server: Analyzes the received feedback and passes it to the interior design AI module.

[1264] Server: The interior design AI module regenerates new design options based on feedback.

[1265] Server: Saves the newly generated design options to temporary storage.

[1266] Step 7:

[1267] Server: Creates a response to send the new design options to the user's device.

[1268] Terminal: Re-display new design options received to the user.

[1269] User: Checks the new design and, if satisfied, provides input to confirm the final design.

[1270] Terminal: Sends final design confirmation information to the server.

[1271] ---

[1272] These are the specific processing steps of the program for the entire system, which allow users to interactively customize the interior design and realize a virtual home staging that reflects their ideals.

[1273] Example 1

[1274] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1275] In real estate sales, there is a demand for methods to visually enhance the appeal of vacant properties. Installing realistic interior designs is costly and time-consuming, and potential buyers have difficulty forming a concrete image of the property. Conventional methods make it difficult for users to select the design they want through trial and error, which discourages them from purchasing.

[1276] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1277] In this invention, the server includes means for uploading interior photos taken by a user, means for automatically erasing unnecessary elements included in the uploaded photos by an artificial intelligence, means for the interior design artificial intelligence to generate multiple design options based on the erased images, means for providing the generated multiple design options to the user's information processing device, means for receiving user feedback and modifying the interior design based on the feedback, means for regenerating new design options based on the user's feedback, means for saving the generated design options in online storage, and means for providing the regenerated design options again to the user's information processing device, thereby enabling the user to interactively experiment with interior designs and find the optimal design.

[1278] "User" refers to a person who uses this system to take photos of an interior and receive interior design suggestions.

[1279] "Server" refers to a computing device that receives users' photos and uses artificial intelligence to process the images and generate designs.

[1280] "Information processing device" refers to an electronic device (e.g., smartphone, tablet, PC) that a user uses to take photos and receive design suggestions from a server.

[1281] "Unwanted elements" refer to objects or backgrounds (e.g., cardboard boxes, old furniture) that are included in the interior photos taken by the user but should be removed when proposing interior designs.

[1282] "Artificial intelligence" refers to technology that automates image processing and design generation using large amounts of data and machine learning algorithms.

[1283] "Interior design artificial intelligence" refers to a specific artificial intelligence model used to automatically generate interior design options from a user's photos.

[1284] "Design options" refers to multiple different interior design proposals generated by the interior design AI.

[1285] "Online storage" refers to a data storage service that allows you to store data via the Internet and access it as needed.

[1286] "Feedback" refers to specific opinions and change requests that users give regarding design options.

[1287] This invention is a virtual home staging system for visually enhancing the appeal of vacant properties in real estate sales. This system uses artificial intelligence (AI) to generate virtual interior designs based on interior photos taken by the user, and allows the user to interactively modify and adjust them.

[1288] Overall system overview

[1289] This system primarily consists of the user's information processing device (smartphone, tablet, PC, etc.), a server, and an interior design AI module. The user uses the information processing device to take photos of the interior and sends them to the server via a dedicated application. The server analyzes the received photos, automatically removes unnecessary elements, and then uses the interior design AI module to generate multiple design options. The generated design options are provided to the user's information processing device, and the user provides feedback based on the displayed designs. The server passes this feedback back to the AI ​​module, which then generates a new design and provides it to the user.

[1290] Hardware and software used

[1291] Hardware:

[1292] Information processing devices (smartphones, tablets, PCs, etc.)

[1293] Server (for data processing and storage)

[1294] software:

[1295] Dedicated application (installed on the user's information processing device)

[1296] Server-side software (online storage such as Amazon S3, Python scripts, OpenCV, TensorFlow-based interior design AI module)

[1297] Specific examples of processing

[1298] Example 1: User uploads a photo

[1299] A user takes a photo of their living room with their smartphone and uploads it to the server using a dedicated application. The user presses the upload button in the application to start sending the photo. The server temporarily stores the received photo in an Amazon S3 bucket.

[1300] Example 2: Removing unnecessary elements

[1301] The server uses the OpenCV library to analyze the stored photos, automatically detecting unwanted elements in the photo, such as cardboard boxes or old furniture, and identifying them through masking. It then applies an inpainting technique to remove the unwanted elements from the image.

[1302] Example 3: Generating interior designs

[1303] The preprocessed images are input into a TensorFlow-based interior design AI module, which generates multiple design options based on pre-trained design patterns (e.g., modern, classic, casual).

[1304] Example prompt sentence:

[1305] For example, if a user "wants a modern style living room," they would enter the following as the prompt:

[1306] For example: "Please add a modern style interior design to the living room in the photo. I would like light wall colors and simple furniture."

[1307] This system allows users to interactively experiment with interior designs to find the best fit, and is expected to help promote real estate sales.

[1308] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1309] Step 1:

[1310] A user takes a photo of their living room with their smartphone and uploads it via a dedicated application. The input is the captured photo data in JPEG format, and the output is an upload request to the server via the application. When the upload button is pressed within the application, the photo data is sent over the Internet to a specified server URL.

[1311] Step 2:

[1312] The device receives the uploaded photo data and temporarily stores it on the device. The input is the JPEG photo data uploaded by the user, and the output is the operation of temporarily storing this data. The stored photo data is ready to be transferred to the server.

[1313] Step 3:

[1314] The server saves the received photo data to storage. An online storage service (e.g., Amazon S3 bucket) is used for saving. The input is the JPEG photo data sent from the device, and the output is the URL of the photo data saved in storage. This URL is used in subsequent processing.

[1315] Step 4:

[1316] The server uses the OpenCV library to erase unnecessary elements in the image. The input is photo data retrieved from storage, and the output is new image data with unnecessary elements erased. Specifically, OpenCV is used to detect cardboard boxes and old furniture in the image and apply inpainting techniques to erase them.

[1317] Step 5:

[1318] The server inputs the preprocessed image data into an interior design AI model. This AI model is built with TensorFlow and uses pre-trained design patterns. The input is image data with unnecessary elements removed, and the output is multiple interior design options. The model analyzes the image and generates design options based on styles such as "modern," "classic," and "casual."

[1319] Step 6:

[1320] The server stores the generated design options in online storage and provides them to the user's information processing device. The input is multiple design options generated by the interior design AI model, and the output is a list of URLs stored in online storage. This list of URLs is then sent to the user's device.

[1321] Step 7:

[1322] The device receives a list of design option URLs and displays each design option in the user interface. The input is the URL list received from the server, and the output is an image of the design option that the user can view. The user can swipe to switch between multiple design options.

[1323] Step 8:

[1324] The user enters feedback on the displayed design options. For example, specific opinions such as "make the wall color lighter" or "rearrange the furniture" are entered into the text input field. The input is the user's feedback text, and the output is a request to send the feedback information to the server.

[1325] Step 9:

[1326] The server analyzes the received feedback and re-inputs it into the interior design AI model. The input is the feedback data received from the user, and the output is the newly generated design options. For example, in response to the feedback "make the wall color lighter," a color conversion process is performed to generate a new interior design.

[1327] Step 10:

[1328] The server saves the newly generated design options in online storage again and provides them to the user's information processing device. The input is the newly generated design options, and the output is a list of URLs saved in online storage. This list of URLs is then sent to the user's device again.

[1329] Step 11:

[1330] The device again receives new design options and displays them in the user interface. The input is the new URL list received from the server, and the output is an image of the latest design options that the user can view. The user again reviews the designs and continues to provide feedback as needed.

[1331] Step 12:

[1332] When the user is satisfied with the design, they press the confirm button in the app to confirm it as the final design. The input is the design option that the user confirms, and the output is the operation of sending the confirmed design to the server and saving it in the database.

[1333] This series of processes allows the user to interactively experiment with interior designs and find the optimal design.

[1334] (Application example 1)

[1335] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1336] In real estate sales, there are limited means to promote vacant properties, making it difficult for potential buyers to visualize the interior of the room. Furthermore, to realize virtual home staging, a flexible system is required that allows users to easily reflect their own preferences and requests, but conventional technology has difficulty meeting this requirement. Furthermore, the same issue exists in virtual stores, as there are not enough functions to design interiors in virtual spaces and save and display the results.

[1337] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1338] In this invention, the server includes a means for uploading interior photos taken by a user, a means for an AI to automatically delete unnecessary elements from the uploaded photos, a means for the interior design AI to generate multiple design options based on the deleted images, a means for accepting user feedback and modifying the interior design based on the feedback, a means for re-receiving user feedback and regenerating the interior design based on the feedback, and a means for saving and displaying the finalized design in the virtual store. This allows users to design interiors in a virtual space based on photos of their actual rooms and to find their ideal interior by repeatedly modifying and regenerating the design. Furthermore, using the finalized design in the virtual store can promote real estate sales and improve the convenience of interior customization in virtual stores.

[1339] "User" refers to the end user who uses the system to upload interior photos and provide feedback on interior design.

[1340] "Uploading means" refers to the interface and communication means for sending photos taken by the user to the server.

[1341] "Means for AI to automatically remove unnecessary elements" refers to a function that uses AI technology to automatically analyze and remove unnecessary objects and items from uploaded photos.

[1342] "Interior Design AI" refers to an artificial intelligence module that generates multiple design options based on a cleaned-up image.

[1343] "Design options" refers to multiple interior design options generated by the interior design AI.

[1344] "User feedback" refers to the user's evaluation of and requests for modifications to the provided design options.

[1345] "Means to modify interior design" refers to the function of reviewing and regenerating interior design based on user feedback.

[1346] A "virtual store" refers to a virtual shopping or exhibition space where products and interiors can be arranged and customized within a virtual space.

[1347] "Final design" refers to the interior design that is finalized as a result of incorporating user feedback.

[1348] "Means for saving and displaying" refers to the function for saving the finalized design within the system and displaying it within the virtual store.

[1349] This invention relates to a virtual home staging system that uses AI to generate a virtual interior design based on indoor photos taken by the user, and allows the user to interactively modify and adjust the design.

[1350] Overall system overview

[1351] The system primarily consists of a user device, a server, and an interior design AI module. Users take photos of the interior using a device such as a smartphone or virtual glasses and send them to the server via an application. The server analyzes the received photos, automatically removes unnecessary elements, and then generates multiple design options using the interior design AI module. The generated design options are returned to the user's device, and the user provides feedback based on the displayed designs. The server passes this feedback back to the AI, which then generates a new design and provides it to the user. The finalized design is saved and displayed in the virtual store.

[1352] Hardware / Software used

[1353] Hardware: Smartphone, virtual glasses, and server

[1354] Software: Python, TensorFlow, OpenCV, Flask

[1355] Program processing details

[1356] 1. User photo uploads

[1357] Users can take photos of the interior of a room using a smartphone app and upload them to a server via a dedicated application.

[1358] 2. Image preprocessing by the server

[1359] The server stores the received images in storage and uses OpenCV to automatically detect and remove unwanted elements (e.g., cardboard boxes, old furniture, etc.) from the photos, generating a clean base image.

[1360] 3. Interior design generation

[1361] The server passes the preprocessed images to an interior design AI module using TensorFlow, which generates multiple design options based on preset styles such as "modern," "classic," and "casual." The generated design options are stored in temporary storage and sent to the user's device.

[1362] 4. User Feedback

[1363] The user reviews the displayed design options and enters specific feedback into the application, such as "I'd like the wall colors to be brighter" or "I'd like the furniture to be a little more modern." This feedback is then sent to the server.

[1364] 5. Interior design revisions

[1365] The server then uses the user's feedback to generate new design options using the interior design AI module, and this process is repeated until the user is satisfied.

[1366] 6. Finalize the design and display it in the virtual store

[1367] Once the user is satisfied with the design, it is saved and displayed in the virtual store, allowing them to visualize how the interior design will look in a real room in a virtual space.

[1368] Specific examples

[1369] For example, if a user submits a photo of their living room and requests a "modern design," the AI ​​will generate multiple design options incorporating elements such as "light-colored walls," "simple modern furniture," and "spacious layout." A specific example of a prompt would be, "I have submitted a photo of my living room. Please generate a modern interior design that suits this room. I would like the furniture to be simple and the colors to be light." By sending this instruction to the server, a design based on the user's requests will be generated.

[1370] As described above, this system allows users to easily experiment with interior designs to find their ideal design. The final design can also be saved and displayed in the virtual store, contributing to promoting real estate sales and improving the user experience.

[1371] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1372] Step 1:

[1373] User-uploaded photos

[1374] Users take photos of the interior of a room using a smartphone app and upload them to the server through a dedicated application. The input is the photo of the room taken with the user's smartphone, and the output is the photo file sent to the server. Specifically, when the "upload photo" button is pressed on the application, the photo is automatically sent to the server.

[1375] Step 2:

[1376] Image preprocessing by the server

[1377] The server stores the received image file in temporary storage and uses the OpenCV library to automatically detect and remove unwanted elements (e.g., cardboard boxes, old furniture, etc.) in the photo. The input is the image file stored on the server, and the output is a clean image with unwanted elements removed. Specifically, the server converts the image to grayscale, performs binary thresholding to mask and remove unwanted objects.

[1378] Step 3:

[1379] Interior Design Generation

[1380] The server sends the preprocessed image to the interior design AI module, which uses TensorFlow to generate multiple design options, such as "modern," "classic," and "casual." The input is clean image data, and the output is multiple interior design options. Specifically, the server passes the image to the AI ​​module, which generates multiple design options based on preset styles.

[1381] Step 4:

[1382] User Feedback

[1383] The user checks the generated design options on a smartphone app and provides feedback such as "change the wall color to a lighter color" or "add more modern furniture." The input is the user's feedback data, and the output is the feedback information sent to the server. Specifically, when the user presses the "Send Feedback" button on the app, the comment is sent to the server.

[1384] Step 5:

[1385] Interior design modifications

[1386] The server receives feedback from the user, and the interior design AI module regenerates new design options based on that feedback. The input is the user's feedback data and preprocessed images, and the output is a new design option modified based on the feedback. Specifically, the server passes the feedback to the AI ​​and saves the regenerated design option in temporary storage.

[1387] Step 6:

[1388] Finalize the design and display it in the virtual store

[1389] The user finally decides on a design that satisfies them, and that design is saved and displayed in the virtual store. The input is the final design data decided by the user, and the output is the final design saved in the virtual store system. Specifically, when the user presses the "Confirm Design" button on the app, the design is saved in the virtual store and becomes available for the user to view in the virtual space.

[1390] This series of processes allows users to find their ideal interior design through trial and error, and ultimately to check the design in a virtual store.

[1391] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1392] ---

[1393] This invention relates to a virtual home staging system for increasing the desire to purchase vacant properties in real estate sales. This system uses AI to generate virtual interior designs based on interior photos taken by the user, which the user can then interactively modify and adjust. In addition, by combining it with an emotion engine that recognizes the user's emotions, the user's emotions can be reflected in the design generation.

[1394] Overall system overview

[1395] This system consists of a user's device, a server, an interior design AI, and an emotion engine. Users take photos of the interior using a device such as a smartphone or tablet and send them to the server via an application. The server analyzes the received photos, automatically removes unnecessary elements, and then uses the interior design AI to generate multiple design options. The generated design options are returned to the user's device, and the user provides feedback based on the displayed designs. The emotion engine then analyzes the user's emotions and reflects the results in the interior design AI to generate new designs.

[1396] Program processing flow (example)

[1397] User-uploaded photos

[1398] User: Take a photo of the living room with a smartphone, launch the dedicated application and log in.

[1399] Device: Tap the "Upload Photo" button on the application screen and select the photo you have taken.

[1400] On the device: Send the selected photo to the backend server.

[1401] Image preprocessing

[1402] Server: Stores the received photo data in temporary storage.

[1403] Server: Passes the saved image data to the AI ​​image analysis module.

[1404] Server: The image analysis module automatically detects unwanted elements in the image (e.g., cardboard boxes, old furniture) and digitally removes them.

[1405] Generate multiple interior design options

[1406] Server: Obtains clean image data with unnecessary elements removed and passes it to the interior design AI module.

[1407] Server: The interior design AI module generates multiple design options based on preset styles such as "modern," "classic," and "casual."

[1408] Server: Saves each generated design option to temporary storage.

[1409] Providing design options

[1410] Server: Creates a response for sending image data from the saved multiple design options to the user's terminal.

[1411] Terminal: Receives image data of design options sent from the server and displays it to the user.

[1412] User Feedback

[1413] User: Review the design options presented and provide specific feedback if necessary (e.g., "I'd like the walls to be a lighter color" or "I'd like the furniture to be a little more modern").

[1414] Terminal: Makes a request to send the user's input feedback to the server.

[1415] Manipulating the Emotion Engine

[1416] Terminal: When inputting feedback, the user's face is photographed with a camera and the image is sent to the emotion engine in real time.

[1417] Emotion engine: Analyzes the user's facial expressions to detect the user's emotional state (e.g., joy, surprise, dissatisfaction).

[1418] Emotion engine: Sends the detected emotional state and feedback content to the server.

[1419] Interior design modifications

[1420] Server: Receives feedback and emotional state, and the interior design AI module regenerates new design options based on that.

[1421] Server: Saves the newly generated design options in temporary storage and sends them back to the user's device.

[1422] Terminal: Re-display new design options received to the user.

[1423] Final design confirmation

[1424] User: If the new design is satisfactory, the user confirms it as the final design, which is then saved in the system and can be reviewed later.

[1425] This series of processes allows users to customize interior designs that take their emotions into account, enabling virtual home staging that is more suited to the user. This system can increase purchasing motivation and maximize the appeal of the property.

[1426] The processing flow will be explained below.

[1427] ---

[1428] Step 1:

[1429] User: Launch the dedicated application on your smartphone and log in.

[1430] User: Takes a photo of the living room with his smartphone.

[1431] Step 2:

[1432] Device: Tap the "Upload Photo" button on the application screen and select the photo you have taken.

[1433] Terminal: Compress the selected photo data and send it to the server.

[1434] Step 3:

[1435] Server: Stores the received photo data in temporary storage.

[1436] Server: Passes the saved image data to the image analysis module.

[1437] Step 4:

[1438] Server: The image analysis module analyzes the image and automatically detects unwanted elements (e.g., old furniture or cardboard boxes).

[1439] Server: Digitally erases detected unwanted elements to generate clean image data.

[1440] Step 5:

[1441] Server: Passes the clean image data to the interior design AI module and instructs it to generate multiple design options.

[1442] Server: The interior design AI module generates multiple design options based on preset styles such as "modern," "classic," and "casual."

[1443] Step 6:

[1444] Server: Stores the generated design options in temporary storage and creates a response to send to the user's device.

[1445] Terminal: Receives image data of design options sent from the server and displays it to the user.

[1446] Step 7:

[1447] User: Review each design option presented and provide feedback (e.g., "I'd like the walls to be a lighter color" or "I'd like the furniture to be a little more modern").

[1448] Device: When inputting feedback, the user's face is photographed with the smartphone camera and the image is sent to the emotion engine in real time.

[1449] Step 8:

[1450] Emotion engine: Analyzes the user's facial expressions to detect their emotional state (e.g., happiness, surprise, dissatisfaction).

[1451] Emotion engine: Sends the detected emotional state and feedback content to the server.

[1452] Step 9:

[1453] Server: Analyzes the received feedback and emotional state and instructs the interior design AI module to regenerate new design options based on that.

[1454] Server: The interior design AI module generates new design options based on user feedback and sentiment.

[1455] Step 10:

[1456] Server: Stores the newly generated design options in temporary storage and creates a response to send to the user's device.

[1457] Terminal: Re-display new design options received to the user.

[1458] Step 11:

[1459] Users: Review the new design options and provide feedback again if needed.

[1460] User: When a satisfactory design is confirmed, it is confirmed as the final design and saved in the system.

[1461] ---

[1462] This series of processes enables customization of interior designs that take emotion analysis into account, realizing optimal virtual home staging for each user. This system makes it possible to propose designs that reflect the user's emotions, maximizing the appeal of the property.

[1463] Example 2

[1464] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1465] Conventional interior design systems have had the drawback of requiring the user to adjust the design themselves, and of making it difficult to generate a design that reflects the user's emotions. This has resulted in the problem that it takes a lot of time and effort for the user to obtain a design that actually satisfies them. The objective of the present invention is to provide a system that can easily generate an interior design that meets the user's needs by analyzing the user's emotional state and reflecting that in the design.

[1466] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for uploading images of an interior taken by a user, means for automatically erasing unnecessary elements contained in the uploaded images using artificial intelligence, means for generating a plurality of interior design options based on the erased images, means for providing the generated plurality of design options to the user's information processing device, means for receiving user feedback and modifying the interior design based on the feedback, and means for analyzing the user's emotions and reflecting the user's emotional state in the design generation. This makes it possible to generate an interior design that takes the user's emotions into consideration, thereby improving user satisfaction, stimulating purchasing motivation, and maximizing the appeal of the property.

[1467] ---

[1468] That's all.

[1469] ---

[1470] "User" refers to a person who uses this system.

[1471] "Interior images" refer to digital images of interior spaces such as homes and commercial facilities.

[1472] "Uploading" refers to the act of sending data from a user's device to a server.

[1473] "Artificial intelligence" refers to computer programs and systems that automatically perform advanced processes such as data analysis and image processing.

[1474] "Unwanted elements" refer to objects or elements that are present in the image but should be removed when generating the interior design.

[1475] "Interior design options" refers to multiple interior design proposals generated based on a specific style or theme.

[1476] "Information processing device" refers to digital devices used by users, such as computers, smartphones, and tablets.

[1477] "Feedback" refers to user-provided comments and requests regarding the design.

[1478] "Analyzing emotions" refers to the process of determining a user's emotional state from their facial expressions and behavior.

[1479] An "emotional state" refers to the emotion (e.g., joy, surprise, dissatisfaction, etc.) that a user is feeling at a particular point in time.

[1480] ---

[1481] That's all.

[1482] ---

[1483] This invention relates to a virtual home staging system for increasing the desire to purchase vacant properties in real estate sales. This system uses artificial intelligence to generate virtual interior designs based on images of the interior taken by the user, which the user can then interactively modify and adjust. In addition, by combining it with an emotion engine that recognizes the user's emotions, the user's emotions can be reflected in the design generation.

[1484] The overall system consists of a user's device, a server, an interior design AI, and an emotion engine. Users take photos of the interior using a device such as a smartphone or tablet and send them to the server via a dedicated application. The server analyzes the received images, automatically removes unnecessary elements, and then uses the interior design AI to generate multiple design options. These design options are sent back to the user's device, and the user provides feedback based on the displayed designs. The emotion engine then analyzes the user's emotions and reflects the results in the interior design AI to generate new designs.

[1485] The specific hardware used is a smartphone or tablet. These devices communicate with the server through a dedicated application (for example, an app developed with React Native). The server-side software uses Python, OpenCV, Amazon S3, and a TensorFlow-based neural network model. The emotion engine uses Microsoft's Azure Emotion API.

[1486] Furthermore, to allow users to provide more specific feedback, the system can generate interior designs using multiple preset styles (e.g., "Modern," "Classic," and "Casual"), allowing users to easily select and adjust design options according to their preferences.

[1487] As a concrete example, consider a user taking and uploading a photo of their living room. The user launches the application, enters their login information, and logs in to their account. Next, they tap the app's "Upload Photo" button and select a photo they took from their device's gallery. The selected photo is automatically sent to the server, which performs image analysis and removes unnecessary elements such as cardboard boxes and old furniture. The interior design AI then generates multiple design options and sends them back to the user's device. The user reviews these designs and provides specific feedback. Along with the feedback, the emotion engine analyzes the user's facial expressions and incorporates their emotional state into the generated design.

[1488] Here are some example prompts to input to a generative AI model:

[1489] "Users upload photos of their living rooms, and AI eliminates unnecessary elements and generates multiple design options. We've built a system that modifies and optimizes the design based on user feedback and emotional state."

[1490] By inputting this prompt into the generative AI model, you can get specific advice on how to implement the above processing flow in detail and on optimization points.

[1491] ---

[1492] That's all.

[1493] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1494] ---

[1495] Step 1: User takes and uploads a photo

[1496] User: Takes a photo of their living room with their smartphone. The input is a high-quality digital image.

[1497] User: Launches the dedicated application, enters login information, and logs in to the account. Next, taps the "Upload Photo" button, selects a photo from the gallery, and sends it to the server. The output is the image data sent to the server.

[1498] Step 2: Image preprocessing

[1499] Server: Stores the received image data in temporary storage (e.g., cloud storage). The input is the image data sent by the user.

[1500] Server: The stored image data is passed to the image analysis module using Python and the OpenCV library. The output is clean image data with unnecessary elements removed.

[1501] How it works: OpenCV's contour detection function is used to detect cardboard boxes and old furniture in the image, and these unwanted elements are then removed using masking techniques.

[1502] Step 3: Generate interior design options

[1503] Server: Passes the clean image data to the interior design AI module using a TensorFlow-based neural network. The input is the preprocessed clean image data.

[1504] Server: The interior design AI module generates multiple design options (e.g., "modern," "classic," "casual"). The output is multiple interior design images.

[1505] How it works: The neural network model analyzes the input image and generates different interior designs based on each preset style.

[1506] Step 4: Providing design options

[1507] Server: Creates a response to send the generated multiple design options to the user's terminal. The input is the generated design options.

[1508] Terminal: Receives image data of the design options sent from the server and displays it to the user. The output is the design options displayed on the user terminal.

[1509] What it does: Use React Native to build an interface that lets users explore design options.

[1510] Step 5: User Provides Feedback

[1511] User: Review the displayed design options and enter specific changes in the app's feedback form. The input is the user's feedback.

[1512] Terminal: Generates and sends a request to the server to send the user's input feedback. The output is the feedback information.

[1513] Step 6: Manipulating the Emotion Engine

[1514] Terminal: When inputting feedback, the user's facial expression is captured by a camera and the video is sent to the emotion engine in real time. The input is video data of the user's facial expression.

[1515] Emotion Engine: Analyzes the user's facial expressions to detect the user's emotional state. The output is the user's emotional state data.

[1516] Emotion engine: The detected emotional state and feedback content are sent to the server. The output is integrated data of the emotional state and feedback.

[1517] Specific operation: Uses Microsoft's Azure Emotion API to analyze the user's emotional state from their facial expressions in real time.

[1518] Step 7: Interior design revisions

[1519] Server: The interior design AI module regenerates new design options based on the received feedback and emotional state. The input is the feedback and emotional state data.

[1520] Server: The regenerated design options are stored in temporary storage and sent to the user's device again. The output is the modified design options.

[1521] What it does: The neural network model recreates the design, taking into account feedback and emotional data.

[1522] Step 8: Finalize the design

[1523] User: If the new design is satisfactory, confirm it as the final design. Input is the selection information for the final design.

[1524] Server: Saves the final design to the system so that it can be reviewed later. The output is the saved final design.

[1525] ---

[1526] That's all.

[1527] (Application example 2)

[1528] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1529] Conventional interior design systems have difficulty reflecting the individual emotions and reactions of users, making it impossible to provide optimal designs that meet the user's needs. Furthermore, it is difficult to perfectly reproduce the interior experience in the real world, making it impossible to sufficiently stimulate the user's purchasing motivation. There is a need to solve these problems and provide optimal interior proposals by allowing users to experience designs in real time in a virtual reality environment while taking into account their emotions.

[1530] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1531] In this invention, the server includes means for uploading interior photos taken by a user, means for an AI to automatically delete unnecessary elements included in the uploaded photos, means for the interior design AI to generate multiple design options based on the deleted images, means for providing the generated multiple design options to the user's device, means for receiving user feedback and modifying the interior design based on the feedback, means for analyzing the user's emotional state and modifying the interior design based on the analysis results, and means for displaying the generated interior design options in a virtual reality environment. This makes it possible to provide interior designs that reflect the user's emotions, and further to stimulate greater purchasing motivation through a real-time experience in a virtual reality environment.

[1532] "User's device" refers to a device used by a user to input and display information, such as a smartphone, head-mounted display (HMD), or tablet.

[1533] "AI automatically removes unnecessary elements" is a process that uses artificial intelligence technology to automatically detect and remove unnecessary elements such as cardboard boxes and old furniture from indoor photos.

[1534] "Interior Design AI" is an artificial intelligence algorithm for generating interior designs and layouts, and has the ability to generate optimal design options taking into account user feedback and emotional state.

[1535] "Design options" are multiple interior design options generated by the interior design AI, including different styles and layouts.

[1536] "User feedback" refers to specific opinions and requests that users input regarding interior design options, such as "make the wall colors brighter" or "use more modern furniture."

[1537] "User emotional state" refers to the emotions the user feels when reviewing the design, such as joy, surprise, or dissatisfaction, which are analyzed in real time using artificial intelligence technology.

[1538] A "virtual reality environment" is a three-dimensional virtual space that users can virtually experience using a head-mounted display (HMD), allowing them to check and adjust interior design options in real time.

[1539] "Display in real time" means that the generated interior design options are displayed without delay so that the user can check them immediately.

[1540] The present invention relates to a virtual home staging system for increasing the willingness to purchase vacant properties in real estate sales. Specific embodiments of the system are described below.

[1541] Hardware and software used

[1542] Hardware:

[1543] Smartphones (latest iPhones and Android devices)

[1544] Head-mounted displays (HMDs) (e.g., Oculus Quest 2)

[1545] Camera (built-in smartphone or HMD camera)

[1546] software:

[1547] User device application (compatible with iOS / Android)

[1548] Emotion recognition libraries (e.g., Emotion AI SDK)

[1549] Interior design AI module (e.g., OpenAI GPT-4)

[1550] Backend servers (e.g., AWS, Google Cloud)

[1551] Data processing and calculation

[1552] 1. Image capture and transmission:

[1553] The user takes a photo of the room using a smartphone or the camera built into the HMD, and the image data is sent to a back-end server.

[1554] 2. Image analysis and removal of unwanted elements:

[1555] The server receives the image data and uses an AI image analysis module to automatically detect and remove unwanted elements (such as cardboard boxes, old furniture, etc.), resulting in a clean image.

[1556] 3. Interior design generation:

[1557] The server passes the clean image data to an interior design AI module, which generates multiple design options (e.g., "modern," "classic," "casual," etc.).

[1558] 4. User feedback and sentiment analysis:

[1559] The user reviews the generated design options via a smartphone or HMD and enters their feedback. When entering feedback, the user's face is photographed with a camera and the video data is sent to an emotion recognition library, which analyzes the user's emotional state (happiness, surprise, dissatisfaction, etc.).

[1560] 5. Regenerate the design and display it in a virtual reality environment:

[1561] The server receives the feedback and emotional state, and the interior design AI module regenerates new design options, which are then sent back to the user's device and displayed in real time in the virtual reality environment.

[1562] Examples and prompts

[1563] For example, if a user provides feedback such as "I would like the walls to be a lighter color," and the camera captures the user's face and determines their emotional state as "happy," the interior design AI module might receive the following prompt:

[1564] Example prompt sentence:

[1565] Suggest a modern interior design that best suits the user's joyful moments, taking into account feedback that the wall colors should be lighter.

[1566] This allows optimal designs that reflect the user's emotions to be generated in real time and displayed in a virtual reality environment, increasing the user's desire to purchase.

[1567] This series of processes allows us to provide interior designs that take into account the user's needs and emotions, and by experiencing them realistically through a virtual environment, we can further increase their desire to purchase.

[1568] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1569] Step 1:

[1570] A user takes a photo of the room using a smartphone or a head-mounted display (HMD) and uploads the photo to the server via a dedicated application. In this step, the input is the photo of the room taken by the user, and the output is that the photo is sent to the server.

[1571] Step 2:

[1572] The server receives the uploaded photo and saves the image data in temporary storage. The saved image data is then passed to the AI ​​image analysis module. In this step, the input is the received photo data and the output is the saved image data.

[1573] Step 3:

[1574] The server uses an image analysis module to automatically detect unwanted elements in the image (e.g., cardboard boxes or old furniture) and digitally remove them. The input for this step is the stored image data, and the output is clean image data with unwanted elements removed.

[1575] Step 4:

[1576] The cleaned image data is passed to an interior design AI module to generate multiple interior design options. For example, designs based on preset styles such as "modern," "classic," and "casual" are generated. The input in this step is the cleaned image data, and the output is multiple design options.

[1577] Step 5:

[1578] The server sends the generated design options to the user's terminal. The user's terminal receives the design options and displays them to the user. In this step, the input is the generated design options and the output is the displayed design options.

[1579] Step 6:

[1580] The user reviews the displayed design options and enters their feedback, which can include specific suggestions such as "I would like the wall color to be lighter." The input in this step is the user's feedback, and the output is the content of the feedback.

[1581] Step 7:

[1582] When the user inputs feedback, the camera on the smartphone or HMD captures the user's face, and the video data is sent to the emotion recognition library in real time. The input of this step is the user's facial video data, and the output is the analyzed emotional state.

[1583] Step 8:

[1584] The emotion recognition library detects the user's emotional state (e.g., joy, surprise, dissatisfaction, etc.) and sends the data to the server. The input in this step is facial video data, and the output is emotional state data.

[1585] Step 9:

[1586] The server receives feedback and emotional state data, and the interior design AI module regenerates new design options based on that. For example, if the emotional state is "joy" and feedback is "make the wall color brighter," a prompt sentence is generated to suggest the optimal design. The input of this step is the feedback and emotional state data, and the output is a new design option.

[1587] Step 10:

[1588] The new design options are resent to the user's device and displayed in the virtual reality environment in real time. The user then checks the new design in the VR space. The input of this step is the regenerated design options, and the output is the design options displayed in the VR space.

[1589] This series of processes provides optimal interior designs in real time based on the user's emotions and feedback.

[1590] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1591] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1592] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1593] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1594] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1595] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1596] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1597] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1598] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1599] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1600] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1601] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1602] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1603] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1604] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1605] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1606] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1607] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1608] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1609] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1610] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1611] The following is further disclosed regarding the above embodiment.

[1612] (Claim 1)

[1613] a means for uploading photos of the interior taken by the user;

[1614] A method for AI to automatically remove unnecessary elements from uploaded photos,

[1615] A means for interior design AI to generate multiple design options based on the deleted images;

[1616] means for providing the generated plurality of design options to a user's terminal;

[1617] a means for accepting user feedback and modifying the interior design based on the feedback;

[1618] A system including:

[1619] (Claim 2)

[1620] 10. The system of claim 1, further comprising means for the interior design AI to regenerate new design options based on user feedback.

[1621] (Claim 3)

[1622] The system of claim 1, further comprising means for generating an interior design based on preset styles such as "modern," "classic," and "casual" for the preprocessed image.

[1623] "Example 1"

[1624] (Claim 1)

[1625] a means for uploading photos of the interior taken by the user;

[1626] A method for automatically erasing unnecessary elements from uploaded photos using artificial intelligence,

[1627] A means for an interior design AI to generate multiple design options based on the deleted images;

[1628] means for providing the generated plurality of design options to a user's information processing device;

[1629] a means for accepting user feedback and modifying the interior design based on the feedback;

[1630] a means for regenerating new design options based on user feedback;

[1631] A means for saving the generated design options to online storage;

[1632] means for providing the regenerated design options to the user's information processing device again;

[1633] A system including:

[1634] (Claim 2)

[1635] The system of claim 1, further comprising means for generating an interior design based on preset styles such as "modern," "classic," and "casual" for the preprocessed image.

[1636] (Claim 3)

[1637] 10. The system of claim 1, wherein the artificial intelligence model includes means for analyzing user feedback and dynamically adjusting design elements based on change requests.

[1638] "Application Example 1"

[1639] (Claim 1)

[1640] a means for uploading photos of the interior taken by the user;

[1641] A method for AI to automatically remove unnecessary elements from uploaded photos,

[1642] A means for interior design AI to generate multiple design options based on the deleted images;

[1643] means for providing the generated plurality of design options to a user's terminal;

[1644] a means for accepting user feedback and modifying the interior design based on the feedback;

[1645] means for re-receiving user feedback and re-generating the interior design based on the feedback;

[1646] A means to save and display the finalized design in the virtual store,

[1647] A system including:

[1648] (Claim 2)

[1649] The system of claim 1 further comprising means for the interior design AI to regenerate new design options based on user feedback for use within the virtual store.

[1650] (Claim 3)

[1651] The system of claim 1 further includes a means for generating an interior design based on preset styles such as "modern," "classic," and "casual" for the preprocessed image, and customizing the interior design within the virtual store.

[1652] "Example 2: Combining Emotion Engines"

[1653] ---

[1654] (Claim 1)

[1655] A means for uploading images of the room taken by a user;

[1656] A method for artificial intelligence to automatically remove unnecessary elements from uploaded images,

[1657] a means for generating a plurality of interior design options based on the erased image;

[1658] means for providing the generated plurality of design options to a user's information processing device;

[1659] a means for accepting user feedback and modifying the interior design based on the feedback;

[1660] A means for analyzing a user's emotions and reflecting the emotional state in the design generation;

[1661] A system including:

[1662] (Claim 2)

[1663] 10. The system of claim 1, further comprising means for regenerating new design options based on user feedback and emotional state.

[1664] (Claim 3)

[1665] 10. The system of claim 1, further comprising means for generating an interior design based on a plurality of preset styles for the preprocessed image.

[1666] ---

[1667] That's all.

[1668] "Application example 2 when combining emotion engines"

[1669] (Claim 1)

[1670] a means for uploading photos of the interior taken by the user;

[1671] A method for AI to automatically remove unnecessary elements from uploaded photos,

[1672] A means for interior design AI to generate multiple design options based on the deleted images;

[1673] means for providing the generated plurality of design options to a user's terminal;

[1674] a means for accepting user feedback and modifying the interior design based on the feedback;

[1675] means for analyzing the emotional state of a user and modifying the interior design based on the analysis results;

[1676] means for displaying the generated interior design options in a virtual reality environment; and

[1677] A system including:

[1678] (Claim 2)

[1679] 10. The system of claim 1, further comprising means for the interior design AI to regenerate new design options based on user feedback.

[1680] (Claim 3)

[1681] The system of claim 1, further comprising means for generating an interior design based on preset styles such as "modern," "classic," and "casual" for the preprocessed image. [Explanation of symbols]

[1682] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. a means for uploading photos of the interior taken by the user; A method for AI to automatically remove unnecessary elements from uploaded photos, A means for interior design AI to generate multiple design options based on the deleted images; means for providing the generated plurality of design options to a user's terminal; a means for accepting user feedback and modifying the interior design based on the feedback; A system including:

2. 10. The system of claim 1, further comprising means for the interior design AI to regenerate new design options based on user feedback.

3. The system of claim 1, further comprising means for generating an interior design based on preset styles such as "modern," "classic," and "casual" for the preprocessed image.

Images