System

Abstract

A system is provided.SOLUTION: A system comprising: means for inputting a current image of a room captured by a user; means for inputting a desired rearrangement condition of the user; means for generating an image of the room after rearrangement using an artificial intelligence model based on the input image and desired rearrangement condition; means for displaying the generated image on a user terminal; means for adding link information of a product related to furniture or home appliances in the displayed image; and means for accessing an online sales page of the product when the user selects the link information in the image.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Traditionally, redecorating a room has been difficult for users because it is difficult to visualize the desired look. Selecting the right furniture and appliances requires a lot of time and effort, and if they are manually arranged, the results are likely to not meet the user's expectations. Furthermore, the process of selecting and purchasing furniture and appliances on online shops is disconnected, resulting in a poor user experience. [Means for solving the problem]

[0005] This invention provides a system that includes a means for inputting an image of the current state of a room taken by a user and a means for inputting the user's desired redecorating conditions. It also provides a means for generating an image of the redecorated room using an artificial intelligence model based on this input information. The generated image is displayed on the user's terminal, and link information for products related to the furniture or home appliances in the displayed image is added. The system also includes a means for the user to access the online sales page for the relevant product by selecting the link information in the image. The system also includes a means for optimizing the furniture or home appliances in the generated image to match the user's preferences and a means for displaying price information, allowing the user to easily create their ideal room and providing a comfortable redecorating experience.

[0006] The term "user" refers to an entity that attempts to rearrange a room using the system of the present invention.

[0007] "Input means" refers to an interface that allows a user to provide the system with the current state of their room and their desired redecorating conditions.

[0008] "Image" refers to data taken by a user that visually shows the current state of a room.

[0009] "Desired redecorating conditions" refer to the user's desired items such as the ideal room design and style, budget, and favorite colors.

[0010] "Artificial intelligence model" refers to a program that uses machine learning or deep learning technology to generate an image of a redecorated room based on input data.

[0011] The "means for generating an image" refers to a process for analyzing input data and creating a visual image of the remodeled room as a result.

[0012] "Display means" refers to an interface that outputs the generated redecorating image to the user's terminal so that the image can be visually confirmed.

[0013] "Link information" refers to access pointing information to the web page of an online shop related to the furniture or home appliance in the displayed image.

[0014] "Online sales page" refers to a store page on the Internet that users access to purchase furniture or home appliances.

[0015] "Optimization" refers to the process of adjusting the characteristics of the proposed furniture and appliances to fit the style and budget of the selected room based on the user's desired conditions.

[0016] "Means for displaying price information" refers to a function that presents the user with the selling price information of a product when the user accesses a link for furniture or home appliances. [Brief explanation of the drawings]

[0017] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0018] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0019] First, the terms used in the following description will be explained.

[0020] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0021] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0022] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0023] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0024] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0025] [First embodiment]

[0026] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0027] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0028] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0029] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0030] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0031] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0032] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0033] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0034] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0035] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0036] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0037] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0038] This invention is a system that makes it easy to rearrange a room. Users take photos of the current state of the room and input their desired conditions, and AI generates an image of the room after rearranging, allowing them to purchase furniture and home appliances in that image online. The program of the system of the present invention and its specific processing content are described below.

[0039] System program and specific processing

[0040] Step 1: Upload a photo of the current room

[0041] The user launches the app and takes a photo of the current state of the room.

[0042] The device converts the captured photo into JPEG or PNG format and sends this data to the server.

[0043] Step 2: Enter your desired criteria

[0044] Users enter their room style and desired conditions (e.g., Nordic style, budget within 100,000 yen, blue as the main color) in a form within the app.

[0045] The terminal encodes the entered desired conditions into JSON format or similar and sends them to the server.

[0046] Step 3: AI-generated redecorating image

[0047] Based on the received photos of the room and the desired conditions, the server uses image analysis technology and artificial intelligence models (such as deep learning models) to generate an image of the room after redecorating.

[0048] The server selects furniture and home appliances from a database that meet the user's desired conditions and incorporates them into the generated image.

[0049] Step 4: Displaying the image

[0050] The server sends the generated image photograph to the terminal.

[0051] The device displays the received image to the user, allowing the user to visually check the atmosphere of the room after the makeover.

[0052] Step 5: Link furniture and appliances

[0053] The server adds link information to online shops related to each piece of furniture or home appliance in the image photo.

[0054] The terminal displays the image photograph with the link information added again to the user, allowing the user to tap it.

[0055] Step 6: Buy furniture and appliances

[0056] Users tap on the furniture or home appliance they like and access the online shop page.

[0057] The terminal opens the page of the relevant online shop, and the user can complete the purchase procedure.

[0058] Specific examples

[0059] Here are some examples:

[0060] 1. Step 1: Upload a photo of your current room

[0061] The user opens the app, takes a photo of their living room, and presses the "upload" button within the app.

[0062] The device sends the captured photo (example.jpg) to the server.

[0063] 2. Step 2: Enter your desired conditions

[0064] Users enter their desired conditions, such as "Nordic style," "budget under 100,000 yen," and "blue as the main color," into a form within the app.

[0065] The device sends the desired conditions to the server in JSON format ({"style": "Nordic style", "budget": 100000, "color": "blue"}).

[0066] 3. Step 3: AI-generated redecorating image

[0067] The server analyzes the received photos and desired conditions, and uses an AI model to generate an image of the redecorated home.

[0068] The server selects images from a database that include Scandinavian-style furniture and blue accents and incorporates them into the photo.

[0069] 4. Step 4: Displaying the image

[0070] The server sends the generated image photo (new_living_room.jpg) to the terminal.

[0071] The terminal displays this image photo to the user, allowing the user to check the atmosphere of the living room after the makeover.

[0072] 5. Step 5: Linking furniture and appliances

[0073] The server adds link information related to the furniture and home appliances in the image.

[0074] The device redisplays the linked image photo to the user.

[0075] 6. Step 6: Buy furniture and appliances

[0076] The user taps on the blue sofa to access the online shop page.

[0077] The device will open the relevant shop page (e.g. https: / / example.com / blue-sofa) and the user can proceed with the purchase.

[0078] This invention allows users to rearrange their rooms reliably and easily, and helps them create more comfortable and ideal rooms.

[0079] The processing flow will be explained below.

[0080] Step 1:

[0081] The user launches the app and takes a photo of the current state of the room.

[0082] The device converts the captured photo into JPEG or PNG format and sends this data to the server. Specifically, the device stores the photo file in temporary memory and uploads it to the server using an HTTP request.

[0083] Step 2:

[0084] Users enter their desired redecorating requirements in a form within the app, such as "Style: Scandinavian," "Budget: Under 100,000 yen," and "Favorite color: Blue."

[0085] The device encodes the entered desired conditions into a format such as JSON and sends it to the server. In this case, when the user finishes entering the conditions and presses the "Send" button, the device sends the formatted data to the server as an HTTP request.

[0086] Step 3:

[0087] Based on the received photos of the room and the desired conditions, the server uses image analysis technology to analyze the current room layout. Specifically, it uses an object detection algorithm to identify the location and type of major furniture in the room.

[0088] The server then uses an artificial intelligence model (e.g., a deep learning model) to generate images of the redecorated room that best fit the desired criteria, suggesting furniture and decorative items with a Scandinavian design, within a specified budget, and a blue theme, for example.

[0089] Step 4:

[0090] The server sends the generated redecorating image to the device as an image file in JPEG or PNG format. During this process, the generated image is stored in memory and then sent to the device as an HTTP response.

[0091] The device displays the received image to the user, who can then visually confirm the atmosphere of the room after the makeover.

[0092] Step 5:

[0093] The server adds link information associated with each piece of furniture or appliance in the displayed image, which is the URL of the online shop that corresponds to the selected piece of furniture or appliance, and maps it to a specific area of ​​the image.

[0094] The device then displays the image with the link information attached to it again to the user. When the user taps on a specific piece of furniture or home appliance, the link information corresponding to that item is activated.

[0095] Step 6:

[0096] Users can tap on the furniture or appliance they like in the image to access the corresponding online shop page. For example, tapping on a blue sofa will open the sofa's purchase page.

[0097] Based on the link tapped, the device will launch a web browser and display the corresponding online shop page, where the user can proceed directly with the purchase.

[0098] The above is a specific processing flow in the system of the present invention.

[0099] Example 1

[0100] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0101] The traditional redecorating process is extremely complicated, requiring users to spend time and effort selecting and arranging furniture and appliances. It's also difficult for users to visualize the final image, often resulting in an unsatisfactory redecorating. Furthermore, purchasing furniture and appliances requires a separate process, making it inconvenient. To solve these issues, a system is needed that allows users to easily visualize the redecorating image and immediately purchase the necessary furniture and appliances.

[0102] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0103] In this invention, the server includes means for inputting an image of the current state of the room taken by the user, means for inputting the user's desired redecorating conditions, means for converting the input image into JPEG or PNG format and sending it to the server, means for encoding the desired conditions into JSON format and sending it to the server, means for generating an image of the redecorated room using an artificial intelligence model based on the input image and the desired conditions, means for selecting furniture and home appliances from a database based on the specified conditions and incorporating them into the generated image, means for sending the generated image to the user terminal and displaying it, means for adding link information for products related to the furniture or home appliances in the displayed image, and means for the user to access the online sales page for the corresponding product by selecting the link information in the image. This allows the user to easily visually check the redecorating image and immediately purchase the necessary furniture and home appliances.

[0104] "User" refers to a person who uses the system to rearrange a room.

[0105] "Server" refers to a central processing unit that processes data sent by the user and generates an image of the room after the makeover.

[0106] "Terminal" refers to an electronic device such as a computer or smartphone that a user uses to access the system.

[0107] "Means for converting images into JPEG or PNG format" refers to the process by which a user can convert a photo of a room into a standard image format.

[0108] "Means of encoding the desired conditions into JSON format" refers to the process of converting the desired conditions for the redecoration entered by the user into a JSON format that is easy for a computer to understand.

[0109] "Artificial intelligence model" refers to an algorithm or computational model that uses deep learning or machine learning to analyze data and generate new information.

[0110] "Image analysis" refers to the techniques and processes used to analyze input images and understand their contents.

[0111] "Furniture and home appliances" refers to items that a user should install or select when redecorating a room.

[0112] "Link information" refers to URLs to online shops and information pages for products related to the furniture and home appliances in the image.

[0113] "Online sales page" refers to a web page where users can purchase furniture or home appliances of their choice.

[0114] "Generated image" refers to a visual image of the room after redecorating, generated by an artificial intelligence model based on the user's desired conditions and a photo of the room's current state.

[0115] "Database" refers to an information storage device that the system uses to manage and store information about furniture and home appliances, as well as user preferences.

[0116] "Means for displaying" refers to the function by which a server or terminal visually presents generated images and information to a user.

[0117] This invention is a system that makes it easy to rearrange a room. A user takes a photo of the current state of the room and inputs desired conditions. A generative AI model generates an image of the room after rearranging, and furniture and home appliances displayed in the image can be purchased online. The program of the system of the present invention and its specific processing content are described below.

[0118] System configuration and program processing

[0119] Upload a photo of the current state of the room

[0120] The user launches the app and takes a photo of the current state of the room. The user then presses the "Upload" button to send the photo to the server. The device converts the photo into JPEG or PNG format and uploads it to the server via the Internet.

[0121] Enter your desired conditions

[0122] The user enters their room style and desired conditions in the form within the app. The user presses the "Submit" button to send the desired conditions to the server. The device encodes the entered desired conditions into JSON format and sends it to the server.

[0123] AI-based redecorating image generation

[0124] The server uses image analysis technology and a generative AI model to generate an image of the redecorated room based on the received photos of the room and the desired conditions. The server first analyzes the received photos using image analysis software (e.g., OpenCV) to recognize the room's layout and existing furniture. Next, based on the desired conditions, a generative AI model (e.g., Stable Diffusion) is used to generate a new Nordic-style layout. Furniture and decorations that match the conditions are selected from a database (e.g., PostgresSQL) for the generated image, and the final image is created.

[0125] Displaying image photos

[0126] The server sends the generated image to the device and displays it to the user. The generated image is transferred to the device using the Internet Protocol (HTTP), and the device displays the received image in an image viewer within the app to provide visual feedback to the user.

[0127] Linking furniture and appliances

[0128] The server adds link information for online shops related to each piece of furniture or home appliance in the image photo, and the device displays the image photo with the added link information again to the user. The server retrieves the URL of the online shop for the relevant furniture or home appliance from the database and embeds it in the image. The image with the added link information is sent again to the device, which displays it to the user.

[0129] Purchasing furniture and appliances

[0130] Users can tap on the furniture or home appliance they like to access the online shop page. When users tap on the furniture icon in the displayed image, the device will open the URL of the corresponding online shop in a browser or in-app browser, where they can proceed with the purchase.

[0131] Hardware and software used

[0132] The system uses the following hardware and software:

[0133] Hardware: User devices (smartphones, tablets, personal computers), servers

[0134] Software: Generative AI models (e.g., Stable Diffusion), image analysis software (e.g., OpenCV), databases (e.g., PostgresSQL), network protocols (e.g., HTTP)

[0135] Specific operation example:

[0136] Below are some specific examples and examples of prompts for the generative AI model.

[0137] Specific examples

[0138] 1. Upload a photo of the current state of the room

[0139] A user opens the app, takes a photo of their living room, and presses the "upload" button.

[0140] The device converts the captured photo (example.jpg) into JPEG format and sends it to the server.

[0141] 2. Enter your desired conditions

[0142] Users enter desired conditions such as "Nordic style," "budget under 100,000 yen," and "blue as the main color" into the form within the app and press the "Submit" button.

[0143] The device sends the desired conditions to the server in JSON format ({"style": "Nordic style", "budget": 100000, "color": "blue"}).

[0144] 3. AI-based redecorating image generation

[0145] The server analyzes the received photos and desired conditions using image analysis software, and generates an image of the redecorated space using a generative AI model. The server then selects furniture from its database, including Scandinavian-style furniture and blue accents, and incorporates them into the photos.

[0146] Prompt Sentence Examples

[0147] Below is an example of a prompt sentence to input to the generative AI model.

[0148] markdown

[0149] I'm thinking of creating a system to make it easier to redecorate a room. I'd like you to generate an image of a living room that contains the following criteria:

[0150] Current room photo: example.jpg

[0151] Desired style: Scandinavian style

[0152] Budget: Under 100,000 yen

[0153] Color: Blue

[0154] This image should include appropriate furniture and appliances, with links to purchase each item online.

[0155] This invention allows users to easily rearrange their rooms and helps them create their ideal rooms.

[0156] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0157] Step 1:

[0158] Upload a photo of the current state of the room

[0159] The user launches the app, takes a photo of the current state of the room, presses the "Upload" button, and saves the photo (e.g., example.jpg) on ​​the device.

[0160] Input: Room photo

[0161] How it works: The device converts the captured photo into JPEG or PNG format and uploads it to a server over the Internet. The device sends this image data using an HTTP request.

[0162] Output: Image file saved on the server

[0163] Step 2:

[0164] Enter your desired conditions

[0165] Users input their room style and desired conditions into a form within the app (e.g., "Scandinavian style," "budget within 100,000 yen," "blue as the main color"), and then press the "Submit" button to save the entered desired conditions on their device.

[0166] Input: Desired conditions (e.g., {"style": "Nordic style", "budget": 100000, "color": "blue"})

[0167] Specific operation: The terminal encodes the input information into JSON format and sends it to the server using an HTTP request.

[0168] Output: JSON data of desired conditions saved on the server

[0169] Step 3:

[0170] AI-based redecorating image generation

[0171] Based on the received photos of the room and the desired conditions, the server analyzes the photos using image analysis technology (e.g., OpenCV) and generates an image of the room after redecorating using a generative AI model (e.g., Stable Diffusion).

[0172] Input: JSON data of room photos and desired conditions

[0173] How it works: The server uses image analysis software to recognize the room's layout and existing furniture. It then uses an AI model to generate a new Nordic-inspired layout based on the desired settings. It then selects furniture and appliances from a database that match the settings and creates the final image.

[0174] Output: Image of the room after redecorating (e.g., new_living_room.jpg)

[0175] Step 4:

[0176] Displaying image photos

[0177] The server sends the generated image photo (e.g., new_living_room.jpg) to the terminal and displays it to the user.

[0178] Input: Image of the room after redecorating

[0179] Specific operation: The server sends image data to the device using the HTTP protocol. The device displays the received image in the image viewer within the app, allowing the user to visually confirm it.

[0180] Output: An image that can be viewed by the user

[0181] Step 5:

[0182] Linking furniture and appliances

[0183] The server adds link information for online shops related to each piece of furniture or home appliance in the image photo and sends it to the terminal.

[0184] Input: Image of the remodeled room, link to online shop

[0185] Specific operation: The server retrieves the URL of the online shop for the relevant furniture or home appliance from the database and embeds it in the image. The image with the link information added is then sent back to the terminal, which displays it to the user.

[0186] Output: Image with link information added

[0187] Step 6:

[0188] Purchasing furniture and appliances

[0189] Users tap on the furniture or home appliance they like and access the online shop page.

[0190] Input: Image photo with link information added

[0191] Specific operation: When the user taps on a furniture icon in the displayed image, the device will open the URL of the corresponding online store in the browser or in-app browser, where the user can proceed with the purchase.

[0192] Output: Display of online shop purchase page

[0193] (Application example 1)

[0194] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0195] Conventional redecorating support systems have had problems such as differences between the image the user actually sees and the resulting redecorating result, and the time and effort required to properly find the furniture and home appliances desired. Another issue is the lack of effective advertising methods to encourage the purchase of furniture and home appliances to be used in redecorating.

[0196] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0197] In this invention, the server includes means for inputting an image of the current state of the room photographed by the user, means for inputting the user's desired redecorating conditions, means for generating an image of the redecorated room using an artificial intelligence model based on the input image and the desired conditions, means for displaying the generated image on the user terminal, means for adding link information for products related to furniture or home appliances in the displayed image, means for the user to access an online sales page for the product by selecting the link information in the image, and means for displaying advertisements related to the furniture or home appliances in the generated image. This allows the user to easily purchase the furniture or home appliance they want while checking a more realistic image of the redecorated state, and further increases their desire to purchase through effective advertisements.

[0198] "Input means" refers to a method or interface for providing the user with the captured image and desired conditions for the system.

[0199] An "artificial intelligence model" is an algorithm that analyzes data, makes predictions, and generates an image of the room after it has been redecorated.

[0200] "Display means" refers to the method or technique for visually presenting the generated image on the user's terminal.

[0201] The "means for adding link information" is a technology that adds links to online sales pages related to the furniture and home appliances in the generated image to the image.

[0202] The "means for accessing" refers to a method or system in which a user can select link information in an image to move to a web page that offers the corresponding product.

[0203] The "means for displaying advertisements" refers to a method or technology for associating advertisements for furniture or home appliances within the generated image and presenting them to the user.

[0204] A system for realizing this invention includes a means for inputting an image of the current state of a room taken by a user and transmitting the user's desired redecorating conditions to a server. The server uses an artificial intelligence model to generate an image of the room after redecorating based on the received image and desired conditions, and transmits the generated image to the user's terminal.

[0205] The user terminal has a means for displaying the transmitted image and adding link information for products related to the furniture or home appliances in the displayed image. When the user selects the link information in the image, the user accesses the online sales page for the corresponding product. In addition, the server has a means for displaying advertisements for furniture or home appliances in the generated image, so the user can see advertisements for products used in redecorating.

[0206] Hardware and software used

[0207] Hardware: Smartphone used by the user

[0208] Software: Python, PIL (Python Imaging Library), requests library, artificial intelligence model (deep learning model)

[0209] Data processing and calculation

[0210] The server uses an AI model to analyze the image data of the current state of the room received from the user and generates an image of the redecorated room based on the desired conditions entered. This image incorporates furniture and home appliances that meet the desired conditions. The generated image is sent to the user's device, and link and advertising information is added. The user's device displays the generated image, along with the link and advertising information, allowing the user to access the product page selected.

[0211] Specific examples

[0212] As a concrete example, the process will be shown below when a user inputs conditions such as "Scandinavian style," "budget within 100,000 yen," and "based on blue." The user takes a photo and uploads it to the server, then inputs the set conditions. The server uses an AI model to generate an image of the redecorated home, selects furniture and appliances that take into account the interior style, budget, and color conditions, and sends the image to the user's device. An advertising link is added to this image, allowing the user to view it and purchase the product.

[0213] Example prompt sentence:

[0214] "Generate an image of how to redecorate the room in this photo in a Nordic style, with a budget of 100,000 yen or less and a blue color scheme."

[0215] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0216] Step 1:

[0217] The user takes a photo of the current state of the room and uploads the image to the device via the app. The device converts the image into JPEG or PNG format and sends this data to the server. The input is the photo of the current state of the room, and the output is the image data sent to the server.

[0218] Step 2:

[0219] The user enters desired redecorating conditions. The user enters conditions such as style, budget, and color (e.g., "Scandinavian style," "under 100,000 yen," and "blue") into a form within the app. The device encodes the entered desired conditions into JSON format and sends it to the server. The input is the desired conditions, and the output is the JSON-formatted data sent to the server.

[0220] Step 3:

[0221] The server uses an artificial intelligence model (deep learning model) to generate an image of the redecorated room based on the received images and desired conditions. Using image analysis technology and an AI model, the input image data is analyzed, and furniture and home appliances that meet the user's desired conditions are selected from a database to generate the image. The input is image data and desired condition data in JSON format, and the output is the generated image data.

[0222] Step 4:

[0223] The server sends the generated image photograph to the terminal. The terminal displays the received image photograph to the user, allowing the user to visually confirm the atmosphere of the room after the makeover. The input is the image data sent from the server, and the output is the image photograph displayed on the user's terminal.

[0224] Step 5:

[0225] The server adds link information for products related to the furniture and home appliances in the generated image. It associates an online shop link with each piece of furniture or home appliance in the image and sends it back to the terminal. The terminal then displays the image photo with the added link information to the user again, allowing the user to tap it. The input is the generated image data, and the output is the image photo with the added link information.

[0226] Step 6:

[0227] The user taps on the furniture or home appliance they like to access the online shop page. The device opens the corresponding online shop page, allowing the user to complete the purchase procedure. The input is the user's tap operation, and the output is the online shop's purchase page.

[0228] Step 7:

[0229] The server also displays advertisements for furniture and home appliances within the generated image. By retrieving advertising information from a database and incorporating it into the generated image, users can see advertisements for the products used in the makeover. The input is the generated image data and advertising information, and the output is an image with the advertisement displayed.

[0230] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0231] This invention is a system that optimally proposes room redecorating while taking into account the user's emotions. When the user takes a photo of the current state of the room and inputs their desired conditions, AI and an emotion engine work together to generate an image of the room after redecorating, and then enables online purchasing of furniture and home appliances based on that image. The program of the system of the present invention and its specific processing content are described below.

[0232] System program and specific processing

[0233] Step 1: Upload a photo of the current room

[0234] The user launches the app and takes a photo of the current state of the room.

[0235] The device converts the captured photo into JPEG or PNG format and sends this data to the server. Specifically, the device stores the photo file in temporary memory and uploads it to the server using an HTTP request.

[0236] Step 2: Enter your desired criteria

[0237] Users enter their desired redecorating requirements in a form within the app, such as "Style: Scandinavian," "Budget: Under 100,000 yen," and "Favorite color: Blue."

[0238] The device encodes the entered desired conditions into a format such as JSON and sends it to the server. In this case, when the user finishes entering the conditions and presses the "Send" button, the device sends the formatted data to the server as an HTTP request.

[0239] Step 3: Redecorating image generation using AI and emotion engine

[0240] Based on the received photos of the room and the desired conditions, the server uses image analysis technology to analyze the current room layout. Specifically, it uses an object detection algorithm to identify the location and type of major furniture in the room.

[0241] The server then uses an artificial intelligence model to generate images of the remodeled room that best fit the desired criteria, such as furniture and decorative items that match a Scandinavian design, a specified budget, or a blue theme. Additionally, an emotional engine analyzes the user's past responses and current emotional state and takes them into account when generating the optimal image.

[0242] Step 4: Displaying images and emotional feedback

[0243] The server sends the generated image to the device as a JPEG or PNG image file. During this process, the generated image is stored in memory and sent to the device in an HTTP response.

[0244] The device displays the received image to the user and uses an emotion engine to collect the user's emotional feedback in real time, and based on this feedback, regenerates the image if necessary.

[0245] Step 5: Link furniture and appliances

[0246] The server adds link information associated with each piece of furniture or appliance in the displayed image, which is the URL of the online shop that corresponds to the selected piece of furniture or appliance, and maps it to a specific area of ​​the image.

[0247] The device then displays the image with the link information attached to it again to the user. When the user taps on a specific piece of furniture or home appliance, the link information corresponding to that item is activated.

[0248] Step 6: Buy furniture and appliances

[0249] Users can tap on the furniture or appliance they like in the image to access the corresponding online shop page. For example, tapping on a blue sofa will open the sofa's purchase page.

[0250] Based on the link tapped, the device will launch a web browser and display the corresponding online shop page, where the user can proceed directly with the purchase.

[0251] Specific examples

[0252] Here are some examples:

[0253] 1. Step 1: Upload a photo of your current room

[0254] The user opens the app, takes a photo of their living room, and presses the "upload" button within the app.

[0255] The device sends the captured photo (example.jpg) to the server.

[0256] 2. Step 2: Enter your desired conditions

[0257] Users enter their desired conditions, such as "Nordic style," "budget under 100,000 yen," and "blue as the main color," into a form within the app.

[0258] The device sends the desired conditions to the server in JSON format ({"style": "Nordic style", "budget": 100000, "color": "blue"}).

[0259] 3. Step 3: Redecorating image generation using AI and emotion engine

[0260] The server analyzes the received photos and desired conditions, and generates an image of the redecoration using an AI model and emotion engine. It combines past user reaction data with current facial expression analysis to make the ideal proposal.

[0261] 4. Step 4: Displaying images and emotional feedback

[0262] The server sends the generated image photo (new_living_room.jpg) to the terminal.

[0263] The device displays this image to the user, while the emotion engine simultaneously analyzes the user's real-time reactions.

[0264] 5. Step 5: Linking furniture and appliances

[0265] The server adds link information about the furniture and home appliances in the image.

[0266] The device will display an image photo with a link to the user, allowing them to tap the link.

[0267] 6. Step 6: Buy furniture and appliances

[0268] Users tap to select furniture or home appliances and access the online shop page.

[0269] The terminal opens the linked web page, and the user completes the purchase procedure.

[0270] This system allows users to receive optimal room redecorating suggestions that reflect their own emotions, resulting in a more satisfying shopping experience.

[0271] The processing flow will be explained below.

[0272] Step 1:

[0273] The user launches the app and takes a photo of the current state of the room.

[0274] The device converts the captured photo into JPEG or PNG format and sends this data to the server. Specifically, the device temporarily stores the photo file in memory and then uploads it to the server using an HTTP request.

[0275] Step 2:

[0276] Users enter their desired redecorating requirements in a form within the app, such as "Style: Scandinavian," "Budget: Under 100,000 yen," and "Favorite color: Blue."

[0277] The device encodes the entered desired conditions into a format such as JSON and sends it to the server. In this case, when the user finishes entering the conditions and presses the "Send" button, the device sends the formatted data to the server as an HTTP request.

[0278] Step 3:

[0279] Based on the received photos of the room and the desired conditions, the server uses image analysis technology to analyze the current room layout. Specifically, it uses an object detection algorithm to identify the location and type of major furniture in the room.

[0280] The server then uses an artificial intelligence model to generate images of the remodeled room that best fit the desired criteria, such as furniture and decorative items that match a Scandinavian design, a specified budget, or a blue theme. Additionally, an emotional engine analyzes the user's past responses and current emotional state and takes them into account when generating the optimal image.

[0281] Step 4:

[0282] The server sends the generated image to the device as an image file in JPEG or PNG format. During this process, the generated image is stored in memory and then sent to the device as an HTTP response.

[0283] The device displays the received image to the user and uses an emotion engine to collect the user's emotional feedback in real time, and based on this feedback, regenerates the image if necessary.

[0284] Step 5:

[0285] The server adds link information associated with each piece of furniture or appliance in the displayed image, which is the URL of the online shop that corresponds to the selected piece of furniture or appliance, and maps it to a specific area of ​​the image.

[0286] The device then displays the image with the link information attached to it again to the user. When the user taps on a specific piece of furniture or home appliance, the link information corresponding to that item is activated.

[0287] Step 6:

[0288] Users can tap on the furniture or appliance they like in the image to access the corresponding online shop page. For example, tapping on a blue sofa will open the sofa's purchase page.

[0289] Based on the link tapped, the device will launch a web browser and display the corresponding online shop page, where the user can proceed directly with the purchase.

[0290] The above is a specific processing flow of the system of the present invention. This system allows users to receive optimal room redecorating suggestions that reflect their own emotions, resulting in a more satisfying shopping experience.

[0291] Example 2

[0292] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0293] Current room redecorating suggestion systems do not take into account the user's emotions when making suggestions, making it difficult for users to receive redecorating suggestions that best suit their desired conditions. Furthermore, since they are unable to reflect emotional feedback in real time, there are issues with the accuracy of suggestions and the level of satisfaction. Furthermore, the online purchasing process for suggested furniture and home appliances is complicated, hindering the smooth purchase process.

[0294] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0295] In this invention, the server includes means for inputting an image of the current state of the space taken by the user, means for inputting the user's desired conditions, means for generating an image of the space after redecorating using an artificial intelligence model based on the input image and the desired conditions, means for displaying the generated image on the user's terminal, means for analyzing the user's emotions in real time and reflecting the analysis in the generated image, means for adding link information related to items in the displayed image, and means for the user to access the online sales page of the corresponding item by selecting the link information in the image. This enables optimal redecorating suggestions that take the user's emotions into consideration and a smooth online purchasing process.

[0296] A "user" is an individual or group who wishes to use the system to redecorate their room.

[0297] A "current state image of a space" is image data taken by a user that shows the current state of a room, office, or the like.

[0298] "Desired conditions" is information indicating conditions such as the style, budget, and color desired by the user for redecorating.

[0299] An "artificial intelligence model" is an algorithm or learning model that analyzes and makes suggestions based on the user's desired conditions and current image.

[0300] An "image" is a visual suggestion of what a space might look like after redecorating, generated by an artificial intelligence model.

[0301] A "user terminal" is a device operated by a user, such as a smartphone, tablet, or PC.

[0302] "Emotion analysis means" refers to technology that analyzes a user's facial expressions and tone of voice to grasp the user's emotional state in real time.

[0303] "Link information" refers to the URLs and product information of online shops related to the furniture and home appliances in the image.

[0304] An "online sales page" is a web page for purchasing products that a user accesses by selecting link information within an image.

[0305] "Items" refers to items that are placed in a space, such as furniture and home appliances.

[0306] This invention is a system that proposes optimal room redecorating suggestions taking into account the user's emotions. The system includes multiple hardware and software components, allowing the user to receive high-quality room redecorating suggestions.

[0307] Hardware and Software Configuration

[0308] Server: A high-performance computer responsible for data processing and running AI models, including object detection algorithms and sentiment analysis engines.

[0309] Device: A device such as a smartphone, tablet, or PC that allows users to operate the interface. It is equipped with a camera and microphone with real-time emotion analysis capabilities.

[0310] Application software: This is the software that allows the user to operate the interface. It includes a form for inputting desired conditions, an image upload function, and a function for displaying the generated images.

[0311] Explanation of system processing

[0312] 1. Upload a photo of the current state of the room

[0313] The user launches the app and takes a photo of the current state of the room. The photo is saved in the device's temporary memory. The device then converts the photo into JPEG or PNG format and sends it to the server using an HTTP request.

[0314] 2. Enter your desired conditions

[0315] Users input their desired redecorating requirements into a form within the app. For example, "Style: Scandinavian," "Budget: Within 100,000 yen," and "Favorite color: Blue." The input data is encoded into JSON format by the device and sent to the server as an HTTP POST request.

[0316] 3. Redecorating image generation using AI and emotion engine

[0317] The server receives the photos and desired conditions sent by the user. It then uses image analysis technology to analyze the current layout of the room. This uses object detection algorithms such as YOLO. Based on the analyzed data and desired conditions, a generative AI model generates an image of the room after it has been redecorated. Furthermore, an emotion engine analyzes the user's emotions in real time and reflects them to make optimal suggestions.

[0318] 4. Displaying images and emotional feedback

[0319] The server sends the generated image to the device. The device displays the image to the user and simultaneously collects emotional feedback. Emotional feedback is collected by analyzing the user's facial expressions and tone of voice, and is sent to the server as needed. The server may regenerate the image based on this feedback.

[0320] 5. Linking furniture and appliances

[0321] The server adds link information related to each item in the generated image, including the URL of the online shop. The device then displays the image again to the user, including the link information. When the user taps on a specific item, the link information corresponding to that item is activated.

[0322] 6. Purchasing furniture and appliances

[0323] Users can tap on an item they like in the image to access the corresponding online shop page. The device will launch a web browser based on the link they tapped and display the corresponding online shop page. The user can then proceed with the purchase directly from this page.

[0324] Specific examples

[0325] Here are some examples:

[0326] 1. Upload a photo of the current state of the room

[0327] A user opens the app, takes a photo of their living room, and presses the "upload" button.

[0328] The device sends the captured photo (example.jpg) to the server.

[0329] 2. Enter your desired conditions

[0330] Users enter their desired conditions, such as "Nordic style," "budget under 100,000 yen," and "blue as the main color," into a form within the app.

[0331] The device sends the desired conditions to the server in JSON format ({"style": "Nordic style", "budget": 100000, "color": "blue"}).

[0332] 3. Redecorating image generation using AI and emotion engine

[0333] The server analyzes the received photos and desired conditions, and generates an image of the redecorated home using an AI model and emotion engine.

[0334] Prompt Sentence Examples

[0335] Here are some example prompts to input to a generative AI model:

[0336] "When a user uploads a photo of their current living room and enters desired conditions such as 'Scandinavian style,' 'budget within 100,000 yen,' and 'blue as the main color,' please analyze the current layout of the room and generate a proposed image of what the room will look like after redecorating. Please also take into account the user's past reaction data and current facial expression analysis to optimize the proposal."

[0337] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0338] Step 1:

[0339] The user launches the app and takes a photo of the current state of the space. The photo is saved in the device's internal memory. The device then converts the photo to JPEG or PNG format. An example of this conversion process is to use an image format conversion library. The converted image data is sent to the server using an HTTP request (POST request). The device receives the photo data as input data, performs format conversion as data processing, and generates format-converted image data as output data.

[0340] Step 2:

[0341] The user enters the desired conditions for the makeover into a form within the app. For example, they specify "Style: Nordic style," "Budget: Under 100,000 yen," and "Favorite color: Blue." This input form is displayed on the user's device. The device encodes the entered conditions into JSON format. Specifically, the input data is converted into JSON data such as {"style": "Nordic style," "budget": 100000, "color": "blue"}. This encoded data is sent to the server as an HTTP POST request. The device receives the desired conditions as input data, encodes them into JSON format as data processing, and generates the encoded data as output data.

[0342] Step 3:

[0343] The server uses image analysis technology to analyze the current layout of the space based on the received current photos and desired conditions. Here, it uses object detection algorithms such as YOLO to identify the location and type of major furniture in the room. This process receives the current photos and desired conditions as input data, performs object detection as data processing, and outputs information on the location and type of furniture in the room. Next, the server uses an AI model to generate an image of the space after redecorating. During this generation process, an image that optimally reflects the input desired conditions is generated based on a pre-trained model. The generated image data is obtained as output.

[0344] Step 4:

[0345] The device uses an emotion analysis engine to analyze the user's emotions in real time. Specifically, the user's facial expressions and tone of voice are captured through a camera and microphone, and analyzed using an emotion analysis algorithm. This generates information about the user's emotional state. The generated image is then sent as emotional feedback to the server. The device receives facial expressions and voice as input data, processes the data, performs emotion analysis, and outputs data about the user's emotional state. The server receives this and regenerates the image as needed.

[0346] Step 5:

[0347] The server adds related link information to each item in the generated image. Specifically, it maps the URL information of the corresponding online shop for each piece of furniture or home appliance to a specific area of ​​the image. The input data for this process are the generated image and link information, and the link information is added as data processing, and an image with the link information added is generated as output. The terminal displays this image with the link information again to the user.

[0348] Step 6:

[0349] Users tap on furniture or home appliances they like in the image to access the corresponding online shop page. For example, tapping on a blue sofa opens the sofa's purchase page. Based on the tapped link information, the device launches the default web browser and displays the corresponding online shop page. The user can then proceed with the purchase directly on this page. The device receives the user's tap operation as input data, processes the data by launching a browser based on the link information, and outputs the online shop page.

[0350] (Application example 2)

[0351] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0352] Conventional room redecorating systems have difficulty making optimal suggestions based on a user's individual emotions and preferences, and can only provide general suggestions. As a result, some users are often left dissatisfied, which tends to reduce their motivation to purchase. Furthermore, when selecting furniture in a store, users are unable to virtually try out different furniture arrangements, which can result in results that are far removed from the user's expectations and imagination. There is a need to solve these issues, provide optimal room redecorating suggestions based on the user's individual emotions and preferences, and improve the shopping experience in physical stores.

[0353] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0354] In this invention, the server includes means for inputting an image of the current state of the room taken by the user, means for inputting the user's desired redecorating conditions, means for generating an image of the redecorated room using an artificial intelligence model based on the input image and the desired conditions, means for displaying the generated image on the user terminal, means for analyzing the user's emotions and optimizing the image, means for adding link information for products related to furniture or home appliances in the displayed image, and means for the user to access an online sales page for the relevant product by selecting the link information in the image. This allows the user to receive optimal redecorating suggestions based on their emotions and desires, and also allows them to try out virtual furniture arrangements in a physical store, which increases their desire to purchase and provides a highly satisfying shopping experience.

[0355] An "image of the current state of the room taken by the user" is an image showing the current state of the room taken by the user using a device such as a smartphone or camera.

[0356] "User's desired redecorating conditions" is information entered by the user regarding conditions such as style, budget, and color for the room redecorating desired by the user.

[0357] The "artificial intelligence model" is a model trained using machine learning algorithms to generate an image of a room after redecorating based on image data and conditions.

[0358] The "generated image" is a virtual image of the room after redecorating, created using an artificial intelligence model based on an image of the current room and the desired conditions.

[0359] A "user terminal" refers to an electronic device operated by a user, such as a smartphone, tablet, or PC.

[0360] "Means for analyzing user emotions and optimizing images" refers to means for analyzing the user's facial expressions and reactions using emotion analysis technology, and readjusting the generated image based on the results.

[0361] "Link information for furniture or home appliance-related products" is information such as the URL of an online sales page associated with the furniture or home appliance displayed in the generated image.

[0362] An "online sales page" is a web page where you can purchase products such as furniture and home appliances over the Internet.

[0363] This invention is a system that takes into account the user's emotions and makes optimal room redecorating suggestions. Users take photos of the current state of the room and input their desired conditions, and AI and an emotion analysis engine work together to generate an image of the room after redecorating, allowing them to purchase furniture and home appliances online based on that image.

[0364] The server has a means for inputting images of the current state of the room taken by the user and a means for inputting the user's desired redecorating conditions, and a means for generating an image of the room after redecorating using an artificial intelligence model based on the input images and desired conditions.

[0365] The generated image is displayed on the user's device, which can be a smartphone, tablet, PC, or other electronic device, allowing the user to check the generated image.

[0366] The server also has the means to analyze the user's emotions and optimize the images. The Emotion API is used for emotion analysis, and the generated images are adjusted based on the user's facial expressions and reactions. For example, the server determines whether the user is satisfied with the displayed image and changes the suggestions accordingly.

[0367] The furniture and home appliances in the generated images have links to online sales pages attached, so that when a user selects a particular piece of furniture or home appliance, they can access the online sales page for that product. This link information is achieved by the server adding the URL of the product related to the displayed image.

[0368] The following hardware and software are used to run the programs in this system:

[0369] Hardware:

[0370] User devices (smartphones, tablets, PCs)

[0371] Cloud Server

[0372] software:

[0373] Image analysis: OpenCV

[0374] Emotion analysis: EmotionAPI

[0375] Artificial intelligence model: Keras / TENSORFLOW (registered trademark)

[0376] Data communication: HTTP request

[0377] Mobile App Development: React Native

[0378] As a concrete example, suppose a user takes a photo of the current state of a room and inputs desired conditions such as "Scandinavian style," "budget within 100,000 yen," and "blue as the main color." In this case, an example of a prompt would be as follows:

[0379] "Based on photos of a room you have taken, please generate an image of a redecorating proposal with a Scandinavian design and a blue theme, costing less than 100,000 yen. Please also take into account the user's emotional feedback to make the best proposal."

[0380] As a result, this invention is a system that can provide users with optimal redecorating suggestions that reflect their individual desires and feelings, and achieve a highly satisfying purchasing experience.

[0381] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0382] Step 1:

[0383] The user launches the app and takes a photo of the current state of the room. The device converts the photo into JPEG or PNG format and sends it to the server using an HTTP request. Specifically, the photo file is stored in temporary memory and sent to an API endpoint for uploading to the server.

[0384] Input: A photo of the current state of the room taken by the user

[0385] Output: A photo file in JPEG or PNG format is sent to the server.

[0386] Step 2:

[0387] The user enters desired conditions for the redecoration in a form within the app. For example, "Style: Scandinavian," "Budget: Under 100,000 yen," "Favorite color: Blue," etc. The device encodes these conditions into JSON format and sends it to the server using an HTTP request.

[0388] Input: Redecorating requirements entered by the user

[0389] Output: JSON formatted preference data is sent to the server

[0390] Step 3:

[0391] The server uses OpenCV to analyze the current room layout based on the received room photos and desired conditions, and specifically, uses an object detection algorithm to identify the location and type of major furniture in the room.

[0392] Input: JPEG or PNG photo files, desired conditions data in JSON format

[0393] Output: Data showing the location and type of furniture in the room

[0394] Step 4:

[0395] The server uses an artificial intelligence model to generate an image of the redecorated room that best matches the desired criteria. Using a model trained with Keras / TensorFlow, it suggests furniture and decorative items that fit the room design, the specified budget, and the blue theme.

[0396] Input: Data showing the location and type of furniture in the room, and desired conditions data in JSON format

[0397] Output: A room image generated based on your desired conditions

[0398] Step 5:

[0399] The server uses an emotion analysis engine to analyze the user's emotions. It analyzes the user's facial expressions using the Emotion API and optimizes the generated image based on the results. For example, if the user is not satisfied with the displayed image, it will be regenerated.

[0400] Input: Real-time facial expression images of the user, generated room images

[0401] Output: User sentiment analysis results, room images regenerated if necessary

[0402] Step 6:

[0403] The server sends the generated image to the terminal as an image file in JPEG or PNG format.

[0404] Input: Room image generated based on desired conditions

[0405] Output: A JPEG or PNG image file is sent to the user's device.

[0406] Step 7:

[0407] The device displays the received image to the user and uses an emotion engine to collect the user's emotional feedback in real time, and based on this feedback, regenerates the image if necessary.

[0408] Input: JPEG or PNG image files, real-time user emotion data

[0409] Output: Request for regeneration based on emotional feedback or confirmation of image display

[0410] Step 8:

[0411] The server adds link information associated with each piece of furniture or appliance in the displayed image, which is the URL of the online shop that corresponds to the selected piece of furniture or appliance, and maps it to a specific area of ​​the image.

[0412] Input: Image file in JPEG or PNG format

[0413] Output: Image file with link information added

[0414] Step 9:

[0415] The device displays image photos with link information attached to them to the user, and when the user taps on a specific piece of furniture or home appliance, the link information corresponding to that item is activated.

[0416] Input: Image file with link information

[0417] Output: Display the corresponding online sales page based on tap input

[0418] Step 10:

[0419] Users can tap on the furniture or appliance they like in the image to access the corresponding online shop page. For example, tapping on a blue sofa will open the sofa's purchase page.

[0420] Input: User tap input

[0421] Output: Online store page display of the product

[0422] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0423] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0424] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0425] [Second embodiment]

[0426] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0427] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0428] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0429] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0430] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0431] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0432] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0433] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0434] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0435] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0436] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0437] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0438] This invention is a system that makes it easy to rearrange a room. Users take photos of the current state of the room and input their desired conditions, and AI generates an image of the room after rearranging, allowing them to purchase furniture and home appliances in that image online. The program of the system of the present invention and its specific processing content are described below.

[0439] System program and specific processing

[0440] Step 1: Upload a photo of the current room

[0441] The user launches the app and takes a photo of the current state of the room.

[0442] The device converts the captured photo into JPEG or PNG format and sends this data to the server.

[0443] Step 2: Enter your desired criteria

[0444] Users enter their room style and desired conditions (e.g., Nordic style, budget within 100,000 yen, blue as the main color) in a form within the app.

[0445] The terminal encodes the entered desired conditions into JSON format or similar and sends them to the server.

[0446] Step 3: AI-generated redecorating image

[0447] Based on the received photos of the room and the desired conditions, the server uses image analysis technology and artificial intelligence models (such as deep learning models) to generate an image of the room after redecorating.

[0448] The server selects furniture and home appliances from a database that meet the user's desired conditions and incorporates them into the generated image.

[0449] Step 4: Displaying the image

[0450] The server sends the generated image photograph to the terminal.

[0451] The device displays the received image to the user, allowing the user to visually check the atmosphere of the room after the makeover.

[0452] Step 5: Link furniture and appliances

[0453] The server adds link information to online shops related to each piece of furniture or home appliance in the image photo.

[0454] The terminal displays the image photograph with the link information added again to the user, allowing the user to tap it.

[0455] Step 6: Buy furniture and appliances

[0456] Users tap on the furniture or home appliance they like and access the online shop page.

[0457] The terminal opens the page of the relevant online shop, and the user can complete the purchase procedure.

[0458] Specific examples

[0459] Here are some examples:

[0460] 1. Step 1: Upload a photo of your current room

[0461] The user opens the app, takes a photo of their living room, and presses the "upload" button within the app.

[0462] The device sends the captured photo (example.jpg) to the server.

[0463] 2. Step 2: Enter your desired conditions

[0464] Users enter their desired conditions, such as "Nordic style," "budget under 100,000 yen," and "blue as the main color," into a form within the app.

[0465] The device sends the desired conditions to the server in JSON format ({"style": "Nordic style", "budget": 100000, "color": "blue"}).

[0466] 3. Step 3: AI-generated redecorating image

[0467] The server analyzes the received photos and desired conditions, and uses an AI model to generate an image of the redecorated home.

[0468] The server selects images from a database that include Scandinavian-style furniture and blue accents and incorporates them into the photo.

[0469] 4. Step 4: Displaying the image

[0470] The server sends the generated image photo (new_living_room.jpg) to the terminal.

[0471] The terminal displays this image photo to the user, allowing the user to check the atmosphere of the living room after the makeover.

[0472] 5. Step 5: Linking furniture and appliances

[0473] The server adds link information related to the furniture and home appliances in the image.

[0474] The device redisplays the linked image photo to the user.

[0475] 6. Step 6: Buy furniture and appliances

[0476] The user taps on the blue sofa to access the online shop page.

[0477] The device will open the relevant shop page (e.g. https: / / example.com / blue-sofa) and the user can proceed with the purchase.

[0478] This invention allows users to rearrange their rooms reliably and easily, and helps them create more comfortable and ideal rooms.

[0479] The processing flow will be explained below.

[0480] Step 1:

[0481] The user launches the app and takes a photo of the current state of the room.

[0482] The device converts the captured photo into JPEG or PNG format and sends this data to the server. Specifically, the device stores the photo file in temporary memory and uploads it to the server using an HTTP request.

[0483] Step 2:

[0484] Users enter their desired redecorating requirements in a form within the app, such as "Style: Scandinavian," "Budget: Under 100,000 yen," and "Favorite color: Blue."

[0485] The device encodes the entered desired conditions into a format such as JSON and sends it to the server. In this case, when the user finishes entering the conditions and presses the "Send" button, the device sends the formatted data to the server as an HTTP request.

[0486] Step 3:

[0487] Based on the received photos of the room and the desired conditions, the server uses image analysis technology to analyze the current room layout. Specifically, it uses an object detection algorithm to identify the location and type of major furniture in the room.

[0488] The server then uses an artificial intelligence model (e.g., a deep learning model) to generate images of the redecorated room that best fit the desired criteria, suggesting furniture and decorative items with a Scandinavian design, within a specified budget, and a blue theme, for example.

[0489] Step 4:

[0490] The server sends the generated redecorating image to the device as an image file in JPEG or PNG format. During this process, the generated image is stored in memory and then sent to the device as an HTTP response.

[0491] The device displays the received image to the user, who can then visually confirm the atmosphere of the room after the makeover.

[0492] Step 5:

[0493] The server adds link information associated with each piece of furniture or appliance in the displayed image, which is the URL of the online shop that corresponds to the selected piece of furniture or appliance, and maps it to a specific area of ​​the image.

[0494] The device then displays the image with the link information attached to it again to the user. When the user taps on a specific piece of furniture or home appliance, the link information corresponding to that item is activated.

[0495] Step 6:

[0496] Users can tap on the furniture or appliance they like in the image to access the corresponding online shop page. For example, tapping on a blue sofa will open the sofa's purchase page.

[0497] Based on the link tapped, the device will launch a web browser and display the corresponding online shop page, where the user can proceed directly with the purchase.

[0498] The above is a specific processing flow in the system of the present invention.

[0499] Example 1

[0500] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0501] The traditional redecorating process is extremely complicated, requiring users to spend time and effort selecting and arranging furniture and appliances. It's also difficult for users to visualize the final image, often resulting in an unsatisfactory redecorating. Furthermore, purchasing furniture and appliances requires a separate process, making it inconvenient. To solve these issues, a system is needed that allows users to easily visualize the redecorating image and immediately purchase the necessary furniture and appliances.

[0502] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0503] In this invention, the server includes means for inputting an image of the current state of the room taken by the user, means for inputting the user's desired redecorating conditions, means for converting the input image into JPEG or PNG format and sending it to the server, means for encoding the desired conditions into JSON format and sending it to the server, means for generating an image of the redecorated room using an artificial intelligence model based on the input image and the desired conditions, means for selecting furniture and home appliances from a database based on the specified conditions and incorporating them into the generated image, means for sending the generated image to the user terminal and displaying it, means for adding link information for products related to the furniture or home appliances in the displayed image, and means for the user to access the online sales page for the corresponding product by selecting the link information in the image. This allows the user to easily visually check the redecorating image and immediately purchase the necessary furniture and home appliances.

[0504] "User" refers to a person who uses the system to rearrange a room.

[0505] "Server" refers to a central processing unit that processes data sent by the user and generates an image of the room after the makeover.

[0506] "Terminal" refers to an electronic device such as a computer or smartphone that a user uses to access the system.

[0507] "Means for converting images into JPEG or PNG format" refers to the process by which a user can convert a photo of a room into a standard image format.

[0508] "Means of encoding the desired conditions into JSON format" refers to the process of converting the desired conditions for the redecoration entered by the user into a JSON format that is easy for a computer to understand.

[0509] "Artificial intelligence model" refers to an algorithm or computational model that uses deep learning or machine learning to analyze data and generate new information.

[0510] "Image analysis" refers to the techniques and processes used to analyze input images and understand their contents.

[0511] "Furniture and home appliances" refers to items that a user should install or select when redecorating a room.

[0512] "Link information" refers to URLs to online shops and information pages for products related to the furniture and home appliances in the image.

[0513] "Online sales page" refers to a web page where users can purchase furniture or home appliances of their choice.

[0514] "Generated image" refers to a visual image of the room after redecorating, generated by an artificial intelligence model based on the user's desired conditions and a photo of the room's current state.

[0515] "Database" refers to an information storage device that the system uses to manage and store information about furniture and home appliances, as well as user preferences.

[0516] "Means for displaying" refers to the function by which a server or terminal visually presents generated images and information to a user.

[0517] This invention is a system that makes it easy to rearrange a room. A user takes a photo of the current state of the room and inputs desired conditions. A generative AI model generates an image of the room after rearranging, and furniture and home appliances displayed in the image can be purchased online. The program of the system of the present invention and its specific processing content are described below.

[0518] System configuration and program processing

[0519] Upload a photo of the current state of the room

[0520] The user launches the app and takes a photo of the current state of the room. The user then presses the "Upload" button to send the photo to the server. The device converts the photo into JPEG or PNG format and uploads it to the server via the Internet.

[0521] Enter your desired conditions

[0522] The user enters their room style and desired conditions in the form within the app. The user presses the "Submit" button to send the desired conditions to the server. The device encodes the entered desired conditions into JSON format and sends it to the server.

[0523] AI-based redecorating image generation

[0524] The server uses image analysis technology and a generative AI model to generate an image of the redecorated room based on the received photos of the room and the desired conditions. The server first analyzes the received photos using image analysis software (e.g., OpenCV) to recognize the room's layout and existing furniture. Next, based on the desired conditions, a generative AI model (e.g., Stable Diffusion) is used to generate a new Nordic-style layout. Furniture and decorations that match the conditions are selected from a database (e.g., PostgresSQL) for the generated image, and the final image is created.

[0525] Displaying image photos

[0526] The server sends the generated image to the device and displays it to the user. The generated image is transferred to the device using the Internet Protocol (HTTP), and the device displays the received image in an image viewer within the app to provide visual feedback to the user.

[0527] Linking furniture and appliances

[0528] The server adds link information for online shops related to each piece of furniture or home appliance in the image photo, and the device displays the image photo with the added link information again to the user. The server retrieves the URL of the online shop for the relevant furniture or home appliance from the database and embeds it in the image. The image with the added link information is sent again to the device, which displays it to the user.

[0529] Purchasing furniture and appliances

[0530] Users can tap on the furniture or home appliance they like to access the online shop page. When users tap on the furniture icon in the displayed image, the device will open the URL of the corresponding online shop in a browser or in-app browser, where they can proceed with the purchase.

[0531] Hardware and software used

[0532] The system uses the following hardware and software:

[0533] Hardware: User devices (smartphones, tablets, personal computers), servers

[0534] Software: Generative AI models (e.g., Stable Diffusion), image analysis software (e.g., OpenCV), databases (e.g., PostgresSQL), network protocols (e.g., HTTP)

[0535] Specific operation example:

[0536] Below are some specific examples and examples of prompts for the generative AI model.

[0537] Specific examples

[0538] 1. Upload a photo of the current state of the room

[0539] A user opens the app, takes a photo of their living room, and presses the "upload" button.

[0540] The device converts the captured photo (example.jpg) into JPEG format and sends it to the server.

[0541] 2. Enter your desired conditions

[0542] Users enter desired conditions such as "Nordic style," "budget under 100,000 yen," and "blue as the main color" into the form within the app and press the "Submit" button.

[0543] The device sends the desired conditions to the server in JSON format ({"style": "Nordic style", "budget": 100000, "color": "blue"}).

[0544] 3. AI-based redecorating image generation

[0545] The server analyzes the received photos and desired conditions using image analysis software, and generates an image of the redecorated space using a generative AI model. The server then selects furniture from its database, including Scandinavian-style furniture and blue accents, and incorporates them into the photos.

[0546] Prompt Sentence Examples

[0547] Below is an example of a prompt sentence to input to the generative AI model.

[0548] markdown

[0549] I'm thinking of creating a system to make it easier to redecorate a room. I'd like you to generate an image of a living room that contains the following criteria:

[0550] Current room photo: example.jpg

[0551] Desired style: Scandinavian style

[0552] Budget: Under 100,000 yen

[0553] Color: Blue

[0554] This image should include appropriate furniture and appliances, with links to purchase each item online.

[0555] This invention allows users to easily rearrange their rooms and helps them create their ideal rooms.

[0556] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0557] Step 1:

[0558] Upload a photo of the current state of the room

[0559] The user launches the app, takes a photo of the current state of the room, presses the "Upload" button, and saves the photo (e.g., example.jpg) on ​​the device.

[0560] Input: Room photo

[0561] How it works: The device converts the captured photo into JPEG or PNG format and uploads it to a server over the Internet. The device sends this image data using an HTTP request.

[0562] Output: Image file saved on the server

[0563] Step 2:

[0564] Enter your desired conditions

[0565] Users input their room style and desired conditions into a form within the app (e.g., "Scandinavian style," "budget within 100,000 yen," "blue as the main color"), and then press the "Submit" button to save the entered desired conditions on their device.

[0566] Input: Desired conditions (e.g., {"style": "Nordic style", "budget": 100000, "color": "blue"})

[0567] Specific operation: The terminal encodes the input information into JSON format and sends it to the server using an HTTP request.

[0568] Output: JSON data of desired conditions saved on the server

[0569] Step 3:

[0570] AI-based redecorating image generation

[0571] Based on the received photos of the room and the desired conditions, the server analyzes the photos using image analysis technology (e.g., OpenCV) and generates an image of the room after redecorating using a generative AI model (e.g., Stable Diffusion).

[0572] Input: JSON data of room photos and desired conditions

[0573] How it works: The server uses image analysis software to recognize the room's layout and existing furniture. It then uses an AI model to generate a new Nordic-inspired layout based on the desired settings. It then selects furniture and appliances from a database that match the settings and creates the final image.

[0574] Output: Image of the room after redecorating (e.g., new_living_room.jpg)

[0575] Step 4:

[0576] Displaying image photos

[0577] The server sends the generated image photo (e.g., new_living_room.jpg) to the terminal and displays it to the user.

[0578] Input: Image of the room after redecorating

[0579] Specific operation: The server sends image data to the device using the HTTP protocol. The device displays the received image in the image viewer within the app, allowing the user to visually confirm it.

[0580] Output: An image that can be viewed by the user

[0581] Step 5:

[0582] Linking furniture and appliances

[0583] The server adds link information for online shops related to each piece of furniture or home appliance in the image photo and sends it to the terminal.

[0584] Input: Image of the remodeled room, link to online shop

[0585] Specific operation: The server retrieves the URL of the online shop for the relevant furniture or home appliance from the database and embeds it in the image. The image with the link information added is then sent back to the terminal, which displays it to the user.

[0586] Output: Image with link information added

[0587] Step 6:

[0588] Purchasing furniture and appliances

[0589] Users tap on the furniture or home appliance they like and access the online shop page.

[0590] Input: Image photo with link information added

[0591] Specific operation: When the user taps on a furniture icon in the displayed image, the device will open the URL of the corresponding online store in the browser or in-app browser, where the user can proceed with the purchase.

[0592] Output: Display of online shop purchase page

[0593] (Application example 1)

[0594] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0595] Conventional redecorating support systems have had problems such as differences between the image the user actually sees and the resulting redecorating result, and the time and effort required to properly find the furniture and home appliances desired. Another issue is the lack of effective advertising methods to encourage the purchase of furniture and home appliances to be used in redecorating.

[0596] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0597] In this invention, the server includes means for inputting an image of the current state of the room photographed by the user, means for inputting the user's desired redecorating conditions, means for generating an image of the redecorated room using an artificial intelligence model based on the input image and the desired conditions, means for displaying the generated image on the user terminal, means for adding link information for products related to furniture or home appliances in the displayed image, means for the user to access an online sales page for the product by selecting the link information in the image, and means for displaying advertisements related to the furniture or home appliances in the generated image. This allows the user to easily purchase the furniture or home appliance they want while checking a more realistic image of the redecorated state, and further increases their desire to purchase through effective advertisements.

[0598] "Input means" refers to a method or interface for providing the user with the captured image and desired conditions for the system.

[0599] An "artificial intelligence model" is an algorithm that analyzes data, makes predictions, and generates an image of the room after it has been redecorated.

[0600] "Display means" refers to the method or technique for visually presenting the generated image on the user's terminal.

[0601] The "means for adding link information" is a technology that adds links to online sales pages related to the furniture and home appliances in the generated image to the image.

[0602] The "means for accessing" refers to a method or system in which a user can select link information in an image to move to a web page that offers the corresponding product.

[0603] The "means for displaying advertisements" refers to a method or technology for associating advertisements for furniture or home appliances within the generated image and presenting them to the user.

[0604] A system for realizing this invention includes a means for inputting an image of the current state of a room taken by a user and transmitting the user's desired redecorating conditions to a server. The server uses an artificial intelligence model to generate an image of the room after redecorating based on the received image and desired conditions, and transmits the generated image to the user's terminal.

[0605] The user terminal has a means for displaying the transmitted image and adding link information for products related to the furniture or home appliances in the displayed image. When the user selects the link information in the image, the user accesses the online sales page for the corresponding product. In addition, the server has a means for displaying advertisements for furniture or home appliances in the generated image, so the user can see advertisements for products used in redecorating.

[0606] Hardware and software used

[0607] Hardware: Smartphone used by the user

[0608] Software: Python, PIL (Python Imaging Library), requests library, artificial intelligence model (deep learning model)

[0609] Data processing and calculation

[0610] The server uses an AI model to analyze the image data of the current state of the room received from the user and generates an image of the redecorated room based on the desired conditions entered. This image incorporates furniture and home appliances that meet the desired conditions. The generated image is sent to the user's device, and link and advertising information is added. The user's device displays the generated image, along with the link and advertising information, allowing the user to access the product page selected.

[0611] Specific examples

[0612] As a concrete example, the process will be shown below when a user inputs conditions such as "Scandinavian style," "budget within 100,000 yen," and "based on blue." The user takes a photo and uploads it to the server, then inputs the set conditions. The server uses an AI model to generate an image of the redecorated home, selects furniture and appliances that take into account the interior style, budget, and color conditions, and sends the image to the user's device. An advertising link is added to this image, allowing the user to view it and purchase the product.

[0613] Example prompt sentence:

[0614] "Generate an image of how to redecorate the room in this photo in a Nordic style, with a budget of 100,000 yen or less and a blue color scheme."

[0615] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0616] Step 1:

[0617] The user takes a photo of the current state of the room and uploads the image to the device via the app. The device converts the image into JPEG or PNG format and sends this data to the server. The input is the photo of the current state of the room, and the output is the image data sent to the server.

[0618] Step 2:

[0619] The user enters desired redecorating conditions. The user enters conditions such as style, budget, and color (e.g., "Scandinavian style," "under 100,000 yen," and "blue") into a form within the app. The device encodes the entered desired conditions into JSON format and sends it to the server. The input is the desired conditions, and the output is the JSON-formatted data sent to the server.

[0620] Step 3:

[0621] The server uses an artificial intelligence model (deep learning model) to generate an image of the redecorated room based on the received images and desired conditions. Using image analysis technology and an AI model, the input image data is analyzed, and furniture and home appliances that meet the user's desired conditions are selected from a database to generate the image. The input is image data and desired condition data in JSON format, and the output is the generated image data.

[0622] Step 4:

[0623] The server sends the generated image photograph to the terminal. The terminal displays the received image photograph to the user, allowing the user to visually confirm the atmosphere of the room after the makeover. The input is the image data sent from the server, and the output is the image photograph displayed on the user's terminal.

[0624] Step 5:

[0625] The server adds link information for products related to the furniture and home appliances in the generated image. It associates an online shop link with each piece of furniture or home appliance in the image and sends it back to the terminal. The terminal then displays the image photo with the added link information to the user again, allowing the user to tap it. The input is the generated image data, and the output is the image photo with the added link information.

[0626] Step 6:

[0627] The user taps on the furniture or home appliance they like to access the online shop page. The device opens the corresponding online shop page, allowing the user to complete the purchase procedure. The input is the user's tap operation, and the output is the online shop's purchase page.

[0628] Step 7:

[0629] The server also displays advertisements for furniture and home appliances within the generated image. By retrieving advertising information from a database and incorporating it into the generated image, users can see advertisements for the products used in the makeover. The input is the generated image data and advertising information, and the output is an image with the advertisement displayed.

[0630] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0631] This invention is a system that optimally proposes room redecorating while taking into account the user's emotions. When the user takes a photo of the current state of the room and inputs their desired conditions, AI and an emotion engine work together to generate an image of the room after redecorating, and then enables online purchasing of furniture and home appliances based on that image. The program of the system of the present invention and its specific processing content are described below.

[0632] System program and specific processing

[0633] Step 1: Upload a photo of the current room

[0634] The user launches the app and takes a photo of the current state of the room.

[0635] The device converts the captured photo into JPEG or PNG format and sends this data to the server. Specifically, the device stores the photo file in temporary memory and uploads it to the server using an HTTP request.

[0636] Step 2: Enter your desired criteria

[0637] Users enter their desired redecorating requirements in a form within the app, such as "Style: Scandinavian," "Budget: Under 100,000 yen," and "Favorite color: Blue."

[0638] The device encodes the entered desired conditions into a format such as JSON and sends it to the server. In this case, when the user finishes entering the conditions and presses the "Send" button, the device sends the formatted data to the server as an HTTP request.

[0639] Step 3: Redecorating image generation using AI and emotion engine

[0640] Based on the received photos of the room and the desired conditions, the server uses image analysis technology to analyze the current room layout. Specifically, it uses an object detection algorithm to identify the location and type of major furniture in the room.

[0641] The server then uses an artificial intelligence model to generate images of the remodeled room that best fit the desired criteria, such as furniture and decorative items that match a Scandinavian design, a specified budget, or a blue theme. Additionally, an emotional engine analyzes the user's past responses and current emotional state and takes them into account when generating the optimal image.

[0642] Step 4: Displaying images and emotional feedback

[0643] The server sends the generated image to the device as a JPEG or PNG image file. During this process, the generated image is stored in memory and sent to the device in an HTTP response.

[0644] The device displays the received image to the user and uses an emotion engine to collect the user's emotional feedback in real time, and based on this feedback, regenerates the image if necessary.

[0645] Step 5: Link furniture and appliances

[0646] The server adds link information associated with each piece of furniture or appliance in the displayed image, which is the URL of the online shop that corresponds to the selected piece of furniture or appliance, and maps it to a specific area of ​​the image.

[0647] The device then displays the image with the link information attached to it again to the user. When the user taps on a specific piece of furniture or home appliance, the link information corresponding to that item is activated.

[0648] Step 6: Buy furniture and appliances

[0649] Users can tap on the furniture or appliance they like in the image to access the corresponding online shop page. For example, tapping on a blue sofa will open the sofa's purchase page.

[0650] Based on the link tapped, the device will launch a web browser and display the corresponding online shop page, where the user can proceed directly with the purchase.

[0651] Specific examples

[0652] Here are some examples:

[0653] 1. Step 1: Upload a photo of your current room

[0654] The user opens the app, takes a photo of their living room, and presses the "upload" button within the app.

[0655] The device sends the captured photo (example.jpg) to the server.

[0656] 2. Step 2: Enter your desired conditions

[0657] Users enter their desired conditions, such as "Nordic style," "budget under 100,000 yen," and "blue as the main color," into a form within the app.

[0658] The device sends the desired conditions to the server in JSON format ({"style": "Nordic style", "budget": 100000, "color": "blue"}).

[0659] 3. Step 3: Redecorating image generation using AI and emotion engine

[0660] The server analyzes the received photos and desired conditions, and generates an image of the redecoration using an AI model and emotion engine. It combines past user reaction data with current facial expression analysis to make the ideal proposal.

[0661] 4. Step 4: Displaying images and emotional feedback

[0662] The server sends the generated image photo (new_living_room.jpg) to the terminal.

[0663] The device displays this image to the user, while the emotion engine simultaneously analyzes the user's real-time reactions.

[0664] 5. Step 5: Linking furniture and appliances

[0665] The server adds link information about the furniture and home appliances in the image.

[0666] The device will display an image photo with a link to the user, allowing them to tap the link.

[0667] 6. Step 6: Buy furniture and appliances

[0668] Users tap to select furniture or home appliances and access the online shop page.

[0669] The terminal opens the linked web page, and the user completes the purchase procedure.

[0670] This system allows users to receive optimal room redecorating suggestions that reflect their own emotions, resulting in a more satisfying shopping experience.

[0671] The processing flow will be explained below.

[0672] Step 1:

[0673] The user launches the app and takes a photo of the current state of the room.

[0674] The device converts the captured photo into JPEG or PNG format and sends this data to the server. Specifically, the device temporarily stores the photo file in memory and then uploads it to the server using an HTTP request.

[0675] Step 2:

[0676] Users enter their desired redecorating requirements in a form within the app, such as "Style: Scandinavian," "Budget: Under 100,000 yen," and "Favorite color: Blue."

[0677] The device encodes the entered desired conditions into a format such as JSON and sends it to the server. In this case, when the user finishes entering the conditions and presses the "Send" button, the device sends the formatted data to the server as an HTTP request.

[0678] Step 3:

[0679] Based on the received photos of the room and the desired conditions, the server uses image analysis technology to analyze the current room layout. Specifically, it uses an object detection algorithm to identify the location and type of major furniture in the room.

[0680] The server then uses an artificial intelligence model to generate images of the remodeled room that best fit the desired criteria, such as furniture and decorative items that match a Scandinavian design, a specified budget, or a blue theme. Additionally, an emotional engine analyzes the user's past responses and current emotional state and takes them into account when generating the optimal image.

[0681] Step 4:

[0682] The server sends the generated image to the device as an image file in JPEG or PNG format. During this process, the generated image is stored in memory and then sent to the device as an HTTP response.

[0683] The device displays the received image to the user and uses an emotion engine to collect the user's emotional feedback in real time, and based on this feedback, regenerates the image if necessary.

[0684] Step 5:

[0685] The server adds link information associated with each piece of furniture or appliance in the displayed image, which is the URL of the online shop that corresponds to the selected piece of furniture or appliance, and maps it to a specific area of ​​the image.

[0686] The device then displays the image with the link information attached to it again to the user. When the user taps on a specific piece of furniture or home appliance, the link information corresponding to that item is activated.

[0687] Step 6:

[0688] Users can tap on the furniture or appliance they like in the image to access the corresponding online shop page. For example, tapping on a blue sofa will open the sofa's purchase page.

[0689] Based on the link tapped, the device will launch a web browser and display the corresponding online shop page, where the user can proceed directly with the purchase.

[0690] The above is a specific processing flow of the system of the present invention. This system allows users to receive optimal room redecorating suggestions that reflect their own emotions, resulting in a more satisfying shopping experience.

[0691] Example 2

[0692] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0693] Current room redecorating suggestion systems do not take into account the user's emotions when making suggestions, making it difficult for users to receive redecorating suggestions that best suit their desired conditions. Furthermore, since they are unable to reflect emotional feedback in real time, there are issues with the accuracy of suggestions and the level of satisfaction. Furthermore, the online purchasing process for suggested furniture and home appliances is complicated, hindering the smooth purchase process.

[0694] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0695] In this invention, the server includes means for inputting an image of the current state of the space taken by the user, means for inputting the user's desired conditions, means for generating an image of the space after redecorating using an artificial intelligence model based on the input image and the desired conditions, means for displaying the generated image on the user's terminal, means for analyzing the user's emotions in real time and reflecting the analysis in the generated image, means for adding link information related to items in the displayed image, and means for the user to access the online sales page of the corresponding item by selecting the link information in the image. This enables optimal redecorating suggestions that take the user's emotions into consideration and a smooth online purchasing process.

[0696] A "user" is an individual or group who wishes to use the system to redecorate their room.

[0697] A "current state image of a space" is image data taken by a user that shows the current state of a room, office, or the like.

[0698] "Desired conditions" is information indicating conditions such as the style, budget, and color desired by the user for redecorating.

[0699] An "artificial intelligence model" is an algorithm or learning model that analyzes and makes suggestions based on the user's desired conditions and current image.

[0700] An "image" is a visual suggestion of what a space might look like after redecorating, generated by an artificial intelligence model.

[0701] A "user terminal" is a device operated by a user, such as a smartphone, tablet, or PC.

[0702] "Emotion analysis means" refers to technology that analyzes a user's facial expressions and tone of voice to grasp the user's emotional state in real time.

[0703] "Link information" refers to the URLs and product information of online shops related to the furniture and home appliances in the image.

[0704] An "online sales page" is a web page for purchasing products that a user accesses by selecting link information within an image.

[0705] "Items" refers to items that are placed in a space, such as furniture and home appliances.

[0706] This invention is a system that proposes optimal room redecorating suggestions taking into account the user's emotions. The system includes multiple hardware and software components, allowing the user to receive high-quality room redecorating suggestions.

[0707] Hardware and Software Configuration

[0708] Server: A high-performance computer responsible for data processing and running AI models, including object detection algorithms and sentiment analysis engines.

[0709] Device: A device such as a smartphone, tablet, or PC that allows users to operate the interface. It is equipped with a camera and microphone with real-time emotion analysis capabilities.

[0710] Application software: This is the software that allows the user to operate the interface. It includes a form for inputting desired conditions, an image upload function, and a function for displaying the generated images.

[0711] Explanation of system processing

[0712] 1. Upload a photo of the current state of the room

[0713] The user launches the app and takes a photo of the current state of the room. The photo is saved in the device's temporary memory. The device then converts the photo into JPEG or PNG format and sends it to the server using an HTTP request.

[0714] 2. Enter your desired conditions

[0715] Users input their desired redecorating requirements into a form within the app. For example, "Style: Scandinavian," "Budget: Within 100,000 yen," and "Favorite color: Blue." The input data is encoded into JSON format by the device and sent to the server as an HTTP POST request.

[0716] 3. Redecorating image generation using AI and emotion engine

[0717] The server receives the photos and desired conditions sent by the user. It then uses image analysis technology to analyze the current layout of the room. This uses object detection algorithms such as YOLO. Based on the analyzed data and desired conditions, a generative AI model generates an image of the room after it has been redecorated. Furthermore, an emotion engine analyzes the user's emotions in real time and reflects them to make optimal suggestions.

[0718] 4. Displaying images and emotional feedback

[0719] The server sends the generated image to the device. The device displays the image to the user and simultaneously collects emotional feedback. Emotional feedback is collected by analyzing the user's facial expressions and tone of voice, and is sent to the server as needed. The server may regenerate the image based on this feedback.

[0720] 5. Linking furniture and appliances

[0721] The server adds link information related to each item in the generated image, including the URL of the online shop. The device then displays the image again to the user, including the link information. When the user taps on a specific item, the link information corresponding to that item is activated.

[0722] 6. Purchasing furniture and appliances

[0723] Users can tap on an item they like in the image to access the corresponding online shop page. The device will launch a web browser based on the link they tapped and display the corresponding online shop page. The user can then proceed with the purchase directly from this page.

[0724] Specific examples

[0725] Here are some examples:

[0726] 1. Upload a photo of the current state of the room

[0727] A user opens the app, takes a photo of their living room, and presses the "upload" button.

[0728] The device sends the captured photo (example.jpg) to the server.

[0729] 2. Enter your desired conditions

[0730] Users enter their desired conditions, such as "Nordic style," "budget under 100,000 yen," and "blue as the main color," into a form within the app.

[0731] The device sends the desired conditions to the server in JSON format ({"style": "Nordic style", "budget": 100000, "color": "blue"}).

[0732] 3. Redecorating image generation using AI and emotion engine

[0733] The server analyzes the received photos and desired conditions, and generates an image of the redecorated home using an AI model and emotion engine.

[0734] Prompt Sentence Examples

[0735] Here are some example prompts to input to a generative AI model:

[0736] "When a user uploads a photo of their current living room and enters desired conditions such as 'Scandinavian style,' 'budget within 100,000 yen,' and 'blue as the main color,' please analyze the current layout of the room and generate a proposed image of what the room will look like after redecorating. Please also take into account the user's past reaction data and current facial expression analysis to optimize the proposal."

[0737] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0738] Step 1:

[0739] The user launches the app and takes a photo of the current state of the space. The photo is saved in the device's internal memory. The device then converts the photo to JPEG or PNG format. An example of this conversion process is to use an image format conversion library. The converted image data is sent to the server using an HTTP request (POST request). The device receives the photo data as input data, performs format conversion as data processing, and generates format-converted image data as output data.

[0740] Step 2:

[0741] The user enters the desired conditions for the makeover into a form within the app. For example, they specify "Style: Nordic style," "Budget: Under 100,000 yen," and "Favorite color: Blue." This input form is displayed on the user's device. The device encodes the entered conditions into JSON format. Specifically, the input data is converted into JSON data such as {"style": "Nordic style," "budget": 100000, "color": "blue"}. This encoded data is sent to the server as an HTTP POST request. The device receives the desired conditions as input data, encodes them into JSON format as data processing, and generates the encoded data as output data.

[0742] Step 3:

[0743] The server uses image analysis technology to analyze the current layout of the space based on the received current photos and desired conditions. Here, it uses object detection algorithms such as YOLO to identify the location and type of major furniture in the room. This process receives the current photos and desired conditions as input data, performs object detection as data processing, and outputs information on the location and type of furniture in the room. Next, the server uses an AI model to generate an image of the space after redecorating. During this generation process, an image that optimally reflects the input desired conditions is generated based on a pre-trained model. The generated image data is obtained as output.

[0744] Step 4:

[0745] The device uses an emotion analysis engine to analyze the user's emotions in real time. Specifically, the user's facial expressions and tone of voice are captured through a camera and microphone, and analyzed using an emotion analysis algorithm. This generates information about the user's emotional state. The generated image is then sent as emotional feedback to the server. The device receives facial expressions and voice as input data, processes the data, performs emotion analysis, and outputs data about the user's emotional state. The server receives this and regenerates the image as needed.

[0746] Step 5:

[0747] The server adds related link information to each item in the generated image. Specifically, it maps the URL information of the corresponding online shop for each piece of furniture or home appliance to a specific area of ​​the image. The input data for this process are the generated image and link information, and the link information is added as data processing, and an image with the link information added is generated as output. The terminal displays this image with the link information again to the user.

[0748] Step 6:

[0749] Users tap on furniture or home appliances they like in the image to access the corresponding online shop page. For example, tapping on a blue sofa opens the sofa's purchase page. Based on the tapped link information, the device launches the default web browser and displays the corresponding online shop page. The user can then proceed with the purchase directly on this page. The device receives the user's tap operation as input data, processes the data by launching a browser based on the link information, and outputs the online shop page.

[0750] (Application example 2)

[0751] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0752] Conventional room redecorating systems have difficulty making optimal suggestions based on a user's individual emotions and preferences, and can only provide general suggestions. As a result, some users are often left dissatisfied, which tends to reduce their motivation to purchase. Furthermore, when selecting furniture in a store, users are unable to virtually try out different furniture arrangements, which can result in results that are far removed from the user's expectations and imagination. There is a need to solve these issues, provide optimal room redecorating suggestions based on the user's individual emotions and preferences, and improve the shopping experience in physical stores.

[0753] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0754] In this invention, the server includes means for inputting an image of the current state of the room taken by the user, means for inputting the user's desired redecorating conditions, means for generating an image of the redecorated room using an artificial intelligence model based on the input image and the desired conditions, means for displaying the generated image on the user terminal, means for analyzing the user's emotions and optimizing the image, means for adding link information for products related to furniture or home appliances in the displayed image, and means for the user to access an online sales page for the relevant product by selecting the link information in the image. This allows the user to receive optimal redecorating suggestions based on their emotions and desires, and also allows them to try out virtual furniture arrangements in a physical store, which increases their desire to purchase and provides a highly satisfying shopping experience.

[0755] An "image of the current state of the room taken by the user" is an image showing the current state of the room taken by the user using a device such as a smartphone or camera.

[0756] "User's desired redecorating conditions" is information entered by the user regarding conditions such as style, budget, and color for the room redecorating desired by the user.

[0757] The "artificial intelligence model" is a model trained using machine learning algorithms to generate an image of a room after redecorating based on image data and conditions.

[0758] The "generated image" is a virtual image of the room after redecorating, created using an artificial intelligence model based on an image of the current room and the desired conditions.

[0759] A "user terminal" refers to an electronic device operated by a user, such as a smartphone, tablet, or PC.

[0760] "Means for analyzing user emotions and optimizing images" refers to means for analyzing the user's facial expressions and reactions using emotion analysis technology, and readjusting the generated image based on the results.

[0761] "Link information for furniture or home appliance-related products" is information such as the URL of an online sales page associated with the furniture or home appliance displayed in the generated image.

[0762] An "online sales page" is a web page where you can purchase products such as furniture and home appliances over the Internet.

[0763] This invention is a system that takes into account the user's emotions and makes optimal room redecorating suggestions. Users take photos of the current state of the room and input their desired conditions, and AI and an emotion analysis engine work together to generate an image of the room after redecorating, allowing them to purchase furniture and home appliances online based on that image.

[0764] The server has a means for inputting images of the current state of the room taken by the user and a means for inputting the user's desired redecorating conditions, and a means for generating an image of the room after redecorating using an artificial intelligence model based on the input images and desired conditions.

[0765] The generated image is displayed on the user's device, which can be a smartphone, tablet, PC, or other electronic device, allowing the user to check the generated image.

[0766] The server also has the means to analyze the user's emotions and optimize the images. The Emotion API is used for emotion analysis, and the generated images are adjusted based on the user's facial expressions and reactions. For example, the server determines whether the user is satisfied with the displayed image and changes the suggestions accordingly.

[0767] The furniture and home appliances in the generated images have links to online sales pages attached, so that when a user selects a particular piece of furniture or home appliance, they can access the online sales page for that product. This link information is achieved by the server adding the URL of the product related to the displayed image.

[0768] The following hardware and software are used to run the programs in this system:

[0769] Hardware:

[0770] User devices (smartphones, tablets, PCs)

[0771] Cloud Server

[0772] software:

[0773] Image analysis: OpenCV

[0774] Emotion analysis: EmotionAPI

[0775] Artificial intelligence model: Keras / TensorFlow

[0776] Data communication: HTTP request

[0777] Mobile App Development: React Native

[0778] As a concrete example, suppose a user takes a photo of the current state of a room and inputs desired conditions such as "Scandinavian style," "budget within 100,000 yen," and "blue as the main color." In this case, an example of a prompt would be as follows:

[0779] "Based on photos of a room you have taken, please generate an image of a redecorating proposal with a Scandinavian design and a blue theme, costing less than 100,000 yen. Please also take into account the user's emotional feedback to make the best proposal."

[0780] As a result, this invention is a system that can provide users with optimal redecorating suggestions that reflect their individual desires and feelings, and achieve a highly satisfying purchasing experience.

[0781] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0782] Step 1:

[0783] The user launches the app and takes a photo of the current state of the room. The device converts the photo into JPEG or PNG format and sends it to the server using an HTTP request. Specifically, the photo file is stored in temporary memory and sent to an API endpoint for uploading to the server.

[0784] Input: A photo of the current state of the room taken by the user

[0785] Output: A photo file in JPEG or PNG format is sent to the server.

[0786] Step 2:

[0787] The user enters desired conditions for the redecoration in a form within the app. For example, "Style: Scandinavian," "Budget: Under 100,000 yen," "Favorite color: Blue," etc. The device encodes these conditions into JSON format and sends it to the server using an HTTP request.

[0788] Input: Redecorating requirements entered by the user

[0789] Output: JSON formatted preference data is sent to the server

[0790] Step 3:

[0791] The server uses OpenCV to analyze the current room layout based on the received room photos and desired conditions, and specifically, uses an object detection algorithm to identify the location and type of major furniture in the room.

[0792] Input: JPEG or PNG photo files, desired conditions data in JSON format

[0793] Output: Data showing the location and type of furniture in the room

[0794] Step 4:

[0795] The server uses an artificial intelligence model to generate an image of the redecorated room that best matches the desired criteria. Using a model trained with Keras / TensorFlow, it suggests furniture and decorative items that fit the room design, the specified budget, and the blue theme.

[0796] Input: Data showing the location and type of furniture in the room, and desired conditions data in JSON format

[0797] Output: A room image generated based on your desired conditions

[0798] Step 5:

[0799] The server uses an emotion analysis engine to analyze the user's emotions. It analyzes the user's facial expressions using the Emotion API and optimizes the generated image based on the results. For example, if the user is not satisfied with the displayed image, it will be regenerated.

[0800] Input: Real-time facial expression images of the user, generated room images

[0801] Output: User sentiment analysis results, room images regenerated if necessary

[0802] Step 6:

[0803] The server sends the generated image to the terminal as an image file in JPEG or PNG format.

[0804] Input: Room image generated based on desired conditions

[0805] Output: A JPEG or PNG image file is sent to the user's device.

[0806] Step 7:

[0807] The device displays the received image to the user and uses an emotion engine to collect the user's emotional feedback in real time, and based on this feedback, regenerates the image if necessary.

[0808] Input: JPEG or PNG image files, real-time user emotion data

[0809] Output: Request for regeneration based on emotional feedback or confirmation of image display

[0810] Step 8:

[0811] The server adds link information associated with each piece of furniture or appliance in the displayed image, which is the URL of the online shop that corresponds to the selected piece of furniture or appliance, and maps it to a specific area of ​​the image.

[0812] Input: Image file in JPEG or PNG format

[0813] Output: Image file with link information added

[0814] Step 9:

[0815] The device displays image photos with link information attached to them to the user, and when the user taps on a specific piece of furniture or home appliance, the link information corresponding to that item is activated.

[0816] Input: Image file with link information

[0817] Output: Display the corresponding online sales page based on tap input

[0818] Step 10:

[0819] Users can tap on the furniture or appliance they like in the image to access the corresponding online shop page. For example, tapping on a blue sofa will open the sofa's purchase page.

[0820] Input: User tap input

[0821] Output: Online store page display of the product

[0822] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0823] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0824] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0825] [Third embodiment]

[0826] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0827] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0828] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0829] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0830] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0831] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0832] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0833] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0834] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0835] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0836] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0837] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0838] This invention is a system that makes it easy to rearrange a room. Users take photos of the current state of the room and input their desired conditions, and AI generates an image of the room after rearranging, allowing them to purchase furniture and home appliances in that image online. The program of the system of the present invention and its specific processing content are described below.

[0839] System program and specific processing

[0840] Step 1: Upload a photo of the current room

[0841] The user launches the app and takes a photo of the current state of the room.

[0842] The device converts the captured photo into JPEG or PNG format and sends this data to the server.

[0843] Step 2: Enter your desired criteria

[0844] Users enter their room style and desired conditions (e.g., Nordic style, budget within 100,000 yen, blue as the main color) in a form within the app.

[0845] The terminal encodes the entered desired conditions into JSON format or similar and sends them to the server.

[0846] Step 3: AI-generated redecorating image

[0847] Based on the received photos of the room and the desired conditions, the server uses image analysis technology and artificial intelligence models (such as deep learning models) to generate an image of the room after redecorating.

[0848] The server selects furniture and home appliances from a database that meet the user's desired conditions and incorporates them into the generated image.

[0849] Step 4: Displaying the image

[0850] The server sends the generated image photograph to the terminal.

[0851] The device displays the received image to the user, allowing the user to visually check the atmosphere of the room after the makeover.

[0852] Step 5: Link furniture and appliances

[0853] The server adds link information to online shops related to each piece of furniture or home appliance in the image photo.

[0854] The terminal displays the image photograph with the link information added again to the user, allowing the user to tap it.

[0855] Step 6: Buy furniture and appliances

[0856] Users tap on the furniture or home appliance they like and access the online shop page.

[0857] The terminal opens the page of the relevant online shop, and the user can complete the purchase procedure.

[0858] Specific examples

[0859] Here are some examples:

[0860] 1. Step 1: Upload a photo of your current room

[0861] The user opens the app, takes a photo of their living room, and presses the "upload" button within the app.

[0862] The device sends the captured photo (example.jpg) to the server.

[0863] 2. Step 2: Enter your desired conditions

[0864] Users enter their desired conditions, such as "Nordic style," "budget under 100,000 yen," and "blue as the main color," into a form within the app.

[0865] The device sends the desired conditions to the server in JSON format ({"style": "Nordic style", "budget": 100000, "color": "blue"}).

[0866] 3. Step 3: AI-generated redecorating image

[0867] The server analyzes the received photos and desired conditions, and uses an AI model to generate an image of the redecorated home.

[0868] The server selects images from a database that include Scandinavian-style furniture and blue accents and incorporates them into the photo.

[0869] 4. Step 4: Displaying the image

[0870] The server sends the generated image photo (new_living_room.jpg) to the terminal.

[0871] The terminal displays this image photo to the user, allowing the user to check the atmosphere of the living room after the makeover.

[0872] 5. Step 5: Linking furniture and appliances

[0873] The server adds link information related to the furniture and home appliances in the image.

[0874] The device redisplays the linked image photo to the user.

[0875] 6. Step 6: Buy furniture and appliances

[0876] The user taps on the blue sofa to access the online shop page.

[0877] The device will open the relevant shop page (e.g. https: / / example.com / blue-sofa) and the user can proceed with the purchase.

[0878] This invention allows users to rearrange their rooms reliably and easily, and helps them create more comfortable and ideal rooms.

[0879] The processing flow will be explained below.

[0880] Step 1:

[0881] The user launches the app and takes a photo of the current state of the room.

[0882] The device converts the captured photo into JPEG or PNG format and sends this data to the server. Specifically, the device stores the photo file in temporary memory and uploads it to the server using an HTTP request.

[0883] Step 2:

[0884] Users enter their desired redecorating requirements in a form within the app, such as "Style: Scandinavian," "Budget: Under 100,000 yen," and "Favorite color: Blue."

[0885] The device encodes the entered desired conditions into a format such as JSON and sends it to the server. In this case, when the user finishes entering the conditions and presses the "Send" button, the device sends the formatted data to the server as an HTTP request.

[0886] Step 3:

[0887] Based on the received photos of the room and the desired conditions, the server uses image analysis technology to analyze the current room layout. Specifically, it uses an object detection algorithm to identify the location and type of major furniture in the room.

[0888] The server then uses an artificial intelligence model (e.g., a deep learning model) to generate images of the redecorated room that best fit the desired criteria, suggesting furniture and decorative items with a Scandinavian design, within a specified budget, and a blue theme, for example.

[0889] Step 4:

[0890] The server sends the generated redecorating image to the device as an image file in JPEG or PNG format. During this process, the generated image is stored in memory and then sent to the device as an HTTP response.

[0891] The device displays the received image to the user, who can then visually confirm the atmosphere of the room after the makeover.

[0892] Step 5:

[0893] The server adds link information associated with each piece of furniture or appliance in the displayed image, which is the URL of the online shop that corresponds to the selected piece of furniture or appliance, and maps it to a specific area of ​​the image.

[0894] The device then displays the image with the link information attached to it again to the user. When the user taps on a specific piece of furniture or home appliance, the link information corresponding to that item is activated.

[0895] Step 6:

[0896] Users can tap on the furniture or appliance they like in the image to access the corresponding online shop page. For example, tapping on a blue sofa will open the sofa's purchase page.

[0897] Based on the link tapped, the device will launch a web browser and display the corresponding online shop page, where the user can proceed directly with the purchase.

[0898] The above is a specific processing flow in the system of the present invention.

[0899] Example 1

[0900] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0901] The traditional redecorating process is extremely complicated, requiring users to spend time and effort selecting and arranging furniture and appliances. It's also difficult for users to visualize the final image, often resulting in an unsatisfactory redecorating. Furthermore, purchasing furniture and appliances requires a separate process, making it inconvenient. To solve these issues, a system is needed that allows users to easily visualize the redecorating image and immediately purchase the necessary furniture and appliances.

[0902] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0903] In this invention, the server includes means for inputting an image of the current state of the room taken by the user, means for inputting the user's desired redecorating conditions, means for converting the input image into JPEG or PNG format and sending it to the server, means for encoding the desired conditions into JSON format and sending it to the server, means for generating an image of the redecorated room using an artificial intelligence model based on the input image and the desired conditions, means for selecting furniture and home appliances from a database based on the specified conditions and incorporating them into the generated image, means for sending the generated image to the user terminal and displaying it, means for adding link information for products related to the furniture or home appliances in the displayed image, and means for the user to access the online sales page for the corresponding product by selecting the link information in the image. This allows the user to easily visually check the redecorating image and immediately purchase the necessary furniture and home appliances.

[0904] "User" refers to a person who uses the system to rearrange a room.

[0905] "Server" refers to a central processing unit that processes data sent by the user and generates an image of the room after the makeover.

[0906] "Terminal" refers to an electronic device such as a computer or smartphone that a user uses to access the system.

[0907] "Means for converting images into JPEG or PNG format" refers to the process by which a user can convert a photo of a room into a standard image format.

[0908] "Means of encoding the desired conditions into JSON format" refers to the process of converting the desired conditions for the redecoration entered by the user into a JSON format that is easy for a computer to understand.

[0909] "Artificial intelligence model" refers to an algorithm or computational model that uses deep learning or machine learning to analyze data and generate new information.

[0910] "Image analysis" refers to the techniques and processes used to analyze input images and understand their contents.

[0911] "Furniture and home appliances" refers to items that a user should install or select when redecorating a room.

[0912] "Link information" refers to URLs to online shops and information pages for products related to the furniture and home appliances in the image.

[0913] "Online sales page" refers to a web page where users can purchase furniture or home appliances of their choice.

[0914] "Generated image" refers to a visual image of the room after redecorating, generated by an artificial intelligence model based on the user's desired conditions and a photo of the room's current state.

[0915] "Database" refers to an information storage device that the system uses to manage and store information about furniture and home appliances, as well as user preferences.

[0916] "Means for displaying" refers to the function by which a server or terminal visually presents generated images and information to a user.

[0917] This invention is a system that makes it easy to rearrange a room. A user takes a photo of the current state of the room and inputs desired conditions. A generative AI model generates an image of the room after rearranging, and furniture and home appliances displayed in the image can be purchased online. The program of the system of the present invention and its specific processing content are described below.

[0918] System configuration and program processing

[0919] Upload a photo of the current state of the room

[0920] The user launches the app and takes a photo of the current state of the room. The user then presses the "Upload" button to send the photo to the server. The device converts the photo into JPEG or PNG format and uploads it to the server via the Internet.

[0921] Enter your desired conditions

[0922] The user enters their room style and desired conditions in the form within the app. The user presses the "Submit" button to send the desired conditions to the server. The device encodes the entered desired conditions into JSON format and sends it to the server.

[0923] AI-based redecorating image generation

[0924] The server uses image analysis technology and a generative AI model to generate an image of the redecorated room based on the received photos of the room and the desired conditions. The server first analyzes the received photos using image analysis software (e.g., OpenCV) to recognize the room's layout and existing furniture. Next, based on the desired conditions, a generative AI model (e.g., Stable Diffusion) is used to generate a new Nordic-style layout. Furniture and decorations that match the conditions are selected from a database (e.g., PostgresSQL) for the generated image, and the final image is created.

[0925] Displaying image photos

[0926] The server sends the generated image to the device and displays it to the user. The generated image is transferred to the device using the Internet Protocol (HTTP), and the device displays the received image in an image viewer within the app to provide visual feedback to the user.

[0927] Linking furniture and appliances

[0928] The server adds link information for online shops related to each piece of furniture or home appliance in the image photo, and the device displays the image photo with the added link information again to the user. The server retrieves the URL of the online shop for the relevant furniture or home appliance from the database and embeds it in the image. The image with the added link information is sent again to the device, which displays it to the user.

[0929] Purchasing furniture and appliances

[0930] Users can tap on the furniture or home appliance they like to access the online shop page. When users tap on the furniture icon in the displayed image, the device will open the URL of the corresponding online shop in a browser or in-app browser, where they can proceed with the purchase.

[0931] Hardware and software used

[0932] The system uses the following hardware and software:

[0933] Hardware: User devices (smartphones, tablets, personal computers), servers

[0934] Software: Generative AI models (e.g., Stable Diffusion), image analysis software (e.g., OpenCV), databases (e.g., PostgresSQL), network protocols (e.g., HTTP)

[0935] Specific operation example:

[0936] Below are some specific examples and examples of prompts for the generative AI model.

[0937] Specific examples

[0938] 1. Upload a photo of the current state of the room

[0939] A user opens the app, takes a photo of their living room, and presses the "upload" button.

[0940] The device converts the captured photo (example.jpg) into JPEG format and sends it to the server.

[0941] 2. Enter your desired conditions

[0942] Users enter desired conditions such as "Nordic style," "budget under 100,000 yen," and "blue as the main color" into the form within the app and press the "Submit" button.

[0943] The device sends the desired conditions to the server in JSON format ({"style": "Nordic style", "budget": 100000, "color": "blue"}).

[0944] 3. AI-based redecorating image generation

[0945] The server analyzes the received photos and desired conditions using image analysis software, and generates an image of the redecorated space using a generative AI model. The server then selects furniture from its database, including Scandinavian-style furniture and blue accents, and incorporates them into the photos.

[0946] Prompt Sentence Examples

[0947] Below is an example of a prompt sentence to input to the generative AI model.

[0948] markdown

[0949] I'm thinking of creating a system to make it easier to redecorate a room. I'd like you to generate an image of a living room that contains the following criteria:

[0950] Current room photo: example.jpg

[0951] Desired style: Scandinavian style

[0952] Budget: Under 100,000 yen

[0953] Color: Blue

[0954] This image should include appropriate furniture and appliances, with links to purchase each item online.

[0955] This invention allows users to easily rearrange their rooms and helps them create their ideal rooms.

[0956] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0957] Step 1:

[0958] Upload a photo of the current state of the room

[0959] The user launches the app, takes a photo of the current state of the room, presses the "Upload" button, and saves the photo (e.g., example.jpg) on ​​the device.

[0960] Input: Room photo

[0961] How it works: The device converts the captured photo into JPEG or PNG format and uploads it to a server over the Internet. The device sends this image data using an HTTP request.

[0962] Output: Image file saved on the server

[0963] Step 2:

[0964] Enter your desired conditions

[0965] Users input their room style and desired conditions into a form within the app (e.g., "Scandinavian style," "budget within 100,000 yen," "blue as the main color"), and then press the "Submit" button to save the entered desired conditions on their device.

[0966] Input: Desired conditions (e.g., {"style": "Nordic style", "budget": 100000, "color": "blue"})

[0967] Specific operation: The terminal encodes the input information into JSON format and sends it to the server using an HTTP request.

[0968] Output: JSON data of desired conditions saved on the server

[0969] Step 3:

[0970] AI-based redecorating image generation

[0971] Based on the received photos of the room and the desired conditions, the server analyzes the photos using image analysis technology (e.g., OpenCV) and generates an image of the room after redecorating using a generative AI model (e.g., Stable Diffusion).

[0972] Input: JSON data of room photos and desired conditions

[0973] How it works: The server uses image analysis software to recognize the room's layout and existing furniture. It then uses an AI model to generate a new Nordic-inspired layout based on the desired settings. It then selects furniture and appliances from a database that match the settings and creates the final image.

[0974] Output: Image of the room after redecorating (e.g., new_living_room.jpg)

[0975] Step 4:

[0976] Displaying image photos

[0977] The server sends the generated image photo (e.g., new_living_room.jpg) to the terminal and displays it to the user.

[0978] Input: Image of the room after redecorating

[0979] Specific operation: The server sends image data to the device using the HTTP protocol. The device displays the received image in the image viewer within the app, allowing the user to visually confirm it.

[0980] Output: An image that can be viewed by the user

[0981] Step 5:

[0982] Linking furniture and appliances

[0983] The server adds link information for online shops related to each piece of furniture or home appliance in the image photo and sends it to the terminal.

[0984] Input: Image of the remodeled room, link to online shop

[0985] Specific operation: The server retrieves the URL of the online shop for the relevant furniture or home appliance from the database and embeds it in the image. The image with the link information added is then sent back to the terminal, which displays it to the user.

[0986] Output: Image with link information added

[0987] Step 6:

[0988] Purchasing furniture and appliances

[0989] Users tap on the furniture or home appliance they like and access the online shop page.

[0990] Input: Image photo with link information added

[0991] Specific operation: When the user taps on a furniture icon in the displayed image, the device will open the URL of the corresponding online store in the browser or in-app browser, where the user can proceed with the purchase.

[0992] Output: Display of online shop purchase page

[0993] (Application example 1)

[0994] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0995] Conventional redecorating support systems have had problems such as differences between the image the user actually sees and the resulting redecorating result, and the time and effort required to properly find the furniture and home appliances desired. Another issue is the lack of effective advertising methods to encourage the purchase of furniture and home appliances to be used in redecorating.

[0996] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0997] In this invention, the server includes means for inputting an image of the current state of the room photographed by the user, means for inputting the user's desired redecorating conditions, means for generating an image of the redecorated room using an artificial intelligence model based on the input image and the desired conditions, means for displaying the generated image on the user terminal, means for adding link information for products related to furniture or home appliances in the displayed image, means for the user to access an online sales page for the product by selecting the link information in the image, and means for displaying advertisements related to the furniture or home appliances in the generated image. This allows the user to easily purchase the furniture or home appliance they want while checking a more realistic image of the redecorated state, and further increases their desire to purchase through effective advertisements.

[0998] "Input means" refers to a method or interface for providing the user with the captured image and desired conditions for the system.

[0999] An "artificial intelligence model" is an algorithm that analyzes data, makes predictions, and generates an image of the room after it has been redecorated.

[1000] "Display means" refers to the method or technique for visually presenting the generated image on the user's terminal.

[1001] The "means for adding link information" is a technology that adds links to online sales pages related to the furniture and home appliances in the generated image to the image.

[1002] The "means for accessing" refers to a method or system in which a user can select link information in an image to move to a web page that offers the corresponding product.

[1003] The "means for displaying advertisements" refers to a method or technology for associating advertisements for furniture or home appliances within the generated image and presenting them to the user.

[1004] A system for realizing this invention includes a means for inputting an image of the current state of a room taken by a user and transmitting the user's desired redecorating conditions to a server. The server uses an artificial intelligence model to generate an image of the room after redecorating based on the received image and desired conditions, and transmits the generated image to the user's terminal.

[1005] The user terminal has a means for displaying the transmitted image and adding link information for products related to the furniture or home appliances in the displayed image. When the user selects the link information in the image, the user accesses the online sales page for the corresponding product. In addition, the server has a means for displaying advertisements for furniture or home appliances in the generated image, so the user can see advertisements for products used in redecorating.

[1006] Hardware and software used

[1007] Hardware: Smartphone used by the user

[1008] Software: Python, PIL (Python Imaging Library), requests library, artificial intelligence model (deep learning model)

[1009] Data processing and calculation

[1010] The server uses an AI model to analyze the image data of the current state of the room received from the user and generates an image of the redecorated room based on the desired conditions entered. This image incorporates furniture and home appliances that meet the desired conditions. The generated image is sent to the user's device, and link and advertising information is added. The user's device displays the generated image, along with the link and advertising information, allowing the user to access the product page selected.

[1011] Specific examples

[1012] As a concrete example, the process will be shown below when a user inputs conditions such as "Scandinavian style," "budget within 100,000 yen," and "based on blue." The user takes a photo and uploads it to the server, then inputs the set conditions. The server uses an AI model to generate an image of the redecorated home, selects furniture and appliances that take into account the interior style, budget, and color conditions, and sends the image to the user's device. An advertising link is added to this image, allowing the user to view it and purchase the product.

[1013] Example prompt sentence:

[1014] "Generate an image of how to redecorate the room in this photo in a Nordic style, with a budget of 100,000 yen or less and a blue color scheme."

[1015] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1016] Step 1:

[1017] The user takes a photo of the current state of the room and uploads the image to the device via the app. The device converts the image into JPEG or PNG format and sends this data to the server. The input is the photo of the current state of the room, and the output is the image data sent to the server.

[1018] Step 2:

[1019] The user enters desired redecorating conditions. The user enters conditions such as style, budget, and color (e.g., "Scandinavian style," "under 100,000 yen," and "blue") into a form within the app. The device encodes the entered desired conditions into JSON format and sends it to the server. The input is the desired conditions, and the output is the JSON-formatted data sent to the server.

[1020] Step 3:

[1021] The server uses an artificial intelligence model (deep learning model) to generate an image of the redecorated room based on the received images and desired conditions. Using image analysis technology and an AI model, the input image data is analyzed, and furniture and home appliances that meet the user's desired conditions are selected from a database to generate the image. The input is image data and desired condition data in JSON format, and the output is the generated image data.

[1022] Step 4:

[1023] The server sends the generated image photograph to the terminal. The terminal displays the received image photograph to the user, allowing the user to visually confirm the atmosphere of the room after the makeover. The input is the image data sent from the server, and the output is the image photograph displayed on the user's terminal.

[1024] Step 5:

[1025] The server adds link information for products related to the furniture and home appliances in the generated image. It associates an online shop link with each piece of furniture or home appliance in the image and sends it back to the terminal. The terminal then displays the image photo with the added link information to the user again, allowing the user to tap it. The input is the generated image data, and the output is the image photo with the added link information.

[1026] Step 6:

[1027] The user taps on the furniture or home appliance they like to access the online shop page. The device opens the corresponding online shop page, allowing the user to complete the purchase procedure. The input is the user's tap operation, and the output is the online shop's purchase page.

[1028] Step 7:

[1029] The server also displays advertisements for furniture and home appliances within the generated image. By retrieving advertising information from a database and incorporating it into the generated image, users can see advertisements for the products used in the makeover. The input is the generated image data and advertising information, and the output is an image with the advertisement displayed.

[1030] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1031] This invention is a system that optimally proposes room redecorating while taking into account the user's emotions. When the user takes a photo of the current state of the room and inputs their desired conditions, AI and an emotion engine work together to generate an image of the room after redecorating, and then enables online purchasing of furniture and home appliances based on that image. The program of the system of the present invention and its specific processing content are described below.

[1032] System program and specific processing

[1033] Step 1: Upload a photo of the current room

[1034] The user launches the app and takes a photo of the current state of the room.

[1035] The device converts the captured photo into JPEG or PNG format and sends this data to the server. Specifically, the device stores the photo file in temporary memory and uploads it to the server using an HTTP request.

[1036] Step 2: Enter your desired criteria

[1037] Users enter their desired redecorating requirements in a form within the app, such as "Style: Scandinavian," "Budget: Under 100,000 yen," and "Favorite color: Blue."

[1038] The device encodes the entered desired conditions into a format such as JSON and sends it to the server. In this case, when the user finishes entering the conditions and presses the "Send" button, the device sends the formatted data to the server as an HTTP request.

[1039] Step 3: Redecorating image generation using AI and emotion engine

[1040] Based on the received photos of the room and the desired conditions, the server uses image analysis technology to analyze the current room layout. Specifically, it uses an object detection algorithm to identify the location and type of major furniture in the room.

[1041] The server then uses an artificial intelligence model to generate images of the remodeled room that best fit the desired criteria, such as furniture and decorative items that match a Scandinavian design, a specified budget, or a blue theme. Additionally, an emotional engine analyzes the user's past responses and current emotional state and takes them into account when generating the optimal image.

[1042] Step 4: Displaying images and emotional feedback

[1043] The server sends the generated image to the device as a JPEG or PNG image file. During this process, the generated image is stored in memory and sent to the device in an HTTP response.

[1044] The device displays the received image to the user and uses an emotion engine to collect the user's emotional feedback in real time, and based on this feedback, regenerates the image if necessary.

[1045] Step 5: Link furniture and appliances

[1046] The server adds link information associated with each piece of furniture or appliance in the displayed image, which is the URL of the online shop that corresponds to the selected piece of furniture or appliance, and maps it to a specific area of ​​the image.

[1047] The device then displays the image with the link information attached to it again to the user. When the user taps on a specific piece of furniture or home appliance, the link information corresponding to that item is activated.

[1048] Step 6: Buy furniture and appliances

[1049] Users can tap on the furniture or appliance they like in the image to access the corresponding online shop page. For example, tapping on a blue sofa will open the sofa's purchase page.

[1050] Based on the link tapped, the device will launch a web browser and display the corresponding online shop page, where the user can proceed directly with the purchase.

[1051] Specific examples

[1052] Here are some examples:

[1053] 1. Step 1: Upload a photo of your current room

[1054] The user opens the app, takes a photo of their living room, and presses the "upload" button within the app.

[1055] The device sends the captured photo (example.jpg) to the server.

[1056] 2. Step 2: Enter your desired conditions

[1057] Users enter their desired conditions, such as "Nordic style," "budget under 100,000 yen," and "blue as the main color," into a form within the app.

[1058] The device sends the desired conditions to the server in JSON format ({"style": "Nordic style", "budget": 100000, "color": "blue"}).

[1059] 3. Step 3: Redecorating image generation using AI and emotion engine

[1060] The server analyzes the received photos and desired conditions, and generates an image of the redecoration using an AI model and emotion engine. It combines past user reaction data with current facial expression analysis to make the ideal proposal.

[1061] 4. Step 4: Displaying images and emotional feedback

[1062] The server sends the generated image photo (new_living_room.jpg) to the terminal.

[1063] The device displays this image to the user, while the emotion engine simultaneously analyzes the user's real-time reactions.

[1064] 5. Step 5: Linking furniture and appliances

[1065] The server adds link information about the furniture and home appliances in the image.

[1066] The device will display an image photo with a link to the user, allowing them to tap the link.

[1067] 6. Step 6: Buy furniture and appliances

[1068] Users tap to select furniture or home appliances and access the online shop page.

[1069] The terminal opens the linked web page, and the user completes the purchase procedure.

[1070] This system allows users to receive optimal room redecorating suggestions that reflect their own emotions, resulting in a more satisfying shopping experience.

[1071] The processing flow will be explained below.

[1072] Step 1:

[1073] The user launches the app and takes a photo of the current state of the room.

[1074] The device converts the captured photo into JPEG or PNG format and sends this data to the server. Specifically, the device temporarily stores the photo file in memory and then uploads it to the server using an HTTP request.

[1075] Step 2:

[1076] Users enter their desired redecorating requirements in a form within the app, such as "Style: Scandinavian," "Budget: Under 100,000 yen," and "Favorite color: Blue."

[1077] The device encodes the entered desired conditions into a format such as JSON and sends it to the server. In this case, when the user finishes entering the conditions and presses the "Send" button, the device sends the formatted data to the server as an HTTP request.

[1078] Step 3:

[1079] Based on the received photos of the room and the desired conditions, the server uses image analysis technology to analyze the current room layout. Specifically, it uses an object detection algorithm to identify the location and type of major furniture in the room.

[1080] The server then uses an artificial intelligence model to generate images of the remodeled room that best fit the desired criteria, such as furniture and decorative items that match a Scandinavian design, a specified budget, or a blue theme. Additionally, an emotional engine analyzes the user's past responses and current emotional state and takes them into account when generating the optimal image.

[1081] Step 4:

[1082] The server sends the generated image to the device as an image file in JPEG or PNG format. During this process, the generated image is stored in memory and then sent to the device as an HTTP response.

[1083] The device displays the received image to the user and uses an emotion engine to collect the user's emotional feedback in real time, and based on this feedback, regenerates the image if necessary.

[1084] Step 5:

[1085] The server adds link information associated with each piece of furniture or appliance in the displayed image, which is the URL of the online shop that corresponds to the selected piece of furniture or appliance, and maps it to a specific area of ​​the image.

[1086] The device then displays the image with the link information attached to it again to the user. When the user taps on a specific piece of furniture or home appliance, the link information corresponding to that item is activated.

[1087] Step 6:

[1088] Users can tap on the furniture or appliance they like in the image to access the corresponding online shop page. For example, tapping on a blue sofa will open the sofa's purchase page.

[1089] Based on the link tapped, the device will launch a web browser and display the corresponding online shop page, where the user can proceed directly with the purchase.

[1090] The above is a specific processing flow of the system of the present invention. This system allows users to receive optimal room redecorating suggestions that reflect their own emotions, resulting in a more satisfying shopping experience.

[1091] Example 2

[1092] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1093] Current room redecorating suggestion systems do not take into account the user's emotions when making suggestions, making it difficult for users to receive redecorating suggestions that best suit their desired conditions. Furthermore, since they are unable to reflect emotional feedback in real time, there are issues with the accuracy of suggestions and the level of satisfaction. Furthermore, the online purchasing process for suggested furniture and home appliances is complicated, hindering the smooth purchase process.

[1094] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1095] In this invention, the server includes means for inputting an image of the current state of the space taken by the user, means for inputting the user's desired conditions, means for generating an image of the space after redecorating using an artificial intelligence model based on the input image and the desired conditions, means for displaying the generated image on the user's terminal, means for analyzing the user's emotions in real time and reflecting the analysis in the generated image, means for adding link information related to items in the displayed image, and means for the user to access the online sales page of the corresponding item by selecting the link information in the image. This enables optimal redecorating suggestions that take the user's emotions into consideration and a smooth online purchasing process.

[1096] A "user" is an individual or group who wishes to use the system to redecorate their room.

[1097] A "current state image of a space" is image data taken by a user that shows the current state of a room, office, or the like.

[1098] "Desired conditions" is information indicating conditions such as the style, budget, and color desired by the user for redecorating.

[1099] An "artificial intelligence model" is an algorithm or learning model that analyzes and makes suggestions based on the user's desired conditions and current image.

[1100] An "image" is a visual suggestion of what a space might look like after redecorating, generated by an artificial intelligence model.

[1101] A "user terminal" is a device operated by a user, such as a smartphone, tablet, or PC.

[1102] "Emotion analysis means" refers to technology that analyzes a user's facial expressions and tone of voice to grasp the user's emotional state in real time.

[1103] "Link information" refers to the URLs and product information of online shops related to the furniture and home appliances in the image.

[1104] An "online sales page" is a web page for purchasing products that a user accesses by selecting link information within an image.

[1105] "Items" refers to items that are placed in a space, such as furniture and home appliances.

[1106] This invention is a system that proposes optimal room redecorating suggestions taking into account the user's emotions. The system includes multiple hardware and software components, allowing the user to receive high-quality room redecorating suggestions.

[1107] Hardware and Software Configuration

[1108] Server: A high-performance computer responsible for data processing and running AI models, including object detection algorithms and sentiment analysis engines.

[1109] Device: A device such as a smartphone, tablet, or PC that allows users to operate the interface. It is equipped with a camera and microphone with real-time emotion analysis capabilities.

[1110] Application software: This is the software that allows the user to operate the interface. It includes a form for inputting desired conditions, an image upload function, and a function for displaying the generated images.

[1111] Explanation of system processing

[1112] 1. Upload a photo of the current state of the room

[1113] The user launches the app and takes a photo of the current state of the room. The photo is saved in the device's temporary memory. The device then converts the photo into JPEG or PNG format and sends it to the server using an HTTP request.

[1114] 2. Enter your desired conditions

[1115] Users input their desired redecorating requirements into a form within the app. For example, "Style: Scandinavian," "Budget: Within 100,000 yen," and "Favorite color: Blue." The input data is encoded into JSON format by the device and sent to the server as an HTTP POST request.

[1116] 3. Redecorating image generation using AI and emotion engine

[1117] The server receives the photos and desired conditions sent by the user. It then uses image analysis technology to analyze the current layout of the room. This uses object detection algorithms such as YOLO. Based on the analyzed data and desired conditions, a generative AI model generates an image of the room after it has been redecorated. Furthermore, an emotion engine analyzes the user's emotions in real time and reflects them to make optimal suggestions.

[1118] 4. Displaying images and emotional feedback

[1119] The server sends the generated image to the device. The device displays the image to the user and simultaneously collects emotional feedback. Emotional feedback is collected by analyzing the user's facial expressions and tone of voice, and is sent to the server as needed. The server may regenerate the image based on this feedback.

[1120] 5. Linking furniture and appliances

[1121] The server adds link information related to each item in the generated image, including the URL of the online shop. The device then displays the image again to the user, including the link information. When the user taps on a specific item, the link information corresponding to that item is activated.

[1122] 6. Purchasing furniture and appliances

[1123] Users can tap on an item they like in the image to access the corresponding online shop page. The device will launch a web browser based on the link they tapped and display the corresponding online shop page. The user can then proceed with the purchase directly from this page.

[1124] Specific examples

[1125] Here are some examples:

[1126] 1. Upload a photo of the current state of the room

[1127] A user opens the app, takes a photo of their living room, and presses the "upload" button.

[1128] The device sends the captured photo (example.jpg) to the server.

[1129] 2. Enter your desired conditions

[1130] Users enter their desired conditions, such as "Nordic style," "budget under 100,000 yen," and "blue as the main color," into a form within the app.

[1131] The device sends the desired conditions to the server in JSON format ({"style": "Nordic style", "budget": 100000, "color": "blue"}).

[1132] 3. Redecorating image generation using AI and emotion engine

[1133] The server analyzes the received photos and desired conditions, and generates an image of the redecorated home using an AI model and emotion engine.

[1134] Prompt Sentence Examples

[1135] Here are some example prompts to input to a generative AI model:

[1136] "When a user uploads a photo of their current living room and enters desired conditions such as 'Scandinavian style,' 'budget within 100,000 yen,' and 'blue as the main color,' please analyze the current layout of the room and generate a proposed image of what the room will look like after redecorating. Please also take into account the user's past reaction data and current facial expression analysis to optimize the proposal."

[1137] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1138] Step 1:

[1139] The user launches the app and takes a photo of the current state of the space. The photo is saved in the device's internal memory. The device then converts the photo to JPEG or PNG format. An example of this conversion process is to use an image format conversion library. The converted image data is sent to the server using an HTTP request (POST request). The device receives the photo data as input data, performs format conversion as data processing, and generates format-converted image data as output data.

[1140] Step 2:

[1141] The user enters the desired conditions for the makeover into a form within the app. For example, they specify "Style: Nordic style," "Budget: Under 100,000 yen," and "Favorite color: Blue." This input form is displayed on the user's device. The device encodes the entered conditions into JSON format. Specifically, the input data is converted into JSON data such as {"style": "Nordic style," "budget": 100000, "color": "blue"}. This encoded data is sent to the server as an HTTP POST request. The device receives the desired conditions as input data, encodes them into JSON format as data processing, and generates the encoded data as output data.

[1142] Step 3:

[1143] The server uses image analysis technology to analyze the current layout of the space based on the received current photos and desired conditions. Here, it uses object detection algorithms such as YOLO to identify the location and type of major furniture in the room. This process receives the current photos and desired conditions as input data, performs object detection as data processing, and outputs information on the location and type of furniture in the room. Next, the server uses an AI model to generate an image of the space after redecorating. During this generation process, an image that optimally reflects the input desired conditions is generated based on a pre-trained model. The generated image data is obtained as output.

[1144] Step 4:

[1145] The device uses an emotion analysis engine to analyze the user's emotions in real time. Specifically, the user's facial expressions and tone of voice are captured through a camera and microphone, and analyzed using an emotion analysis algorithm. This generates information about the user's emotional state. The generated image is then sent as emotional feedback to the server. The device receives facial expressions and voice as input data, processes the data, performs emotion analysis, and outputs data about the user's emotional state. The server receives this and regenerates the image as needed.

[1146] Step 5:

[1147] The server adds related link information to each item in the generated image. Specifically, it maps the URL information of the corresponding online shop for each piece of furniture or home appliance to a specific area of ​​the image. The input data for this process are the generated image and link information, and the link information is added as data processing, and an image with the link information added is generated as output. The terminal displays this image with the link information again to the user.

[1148] Step 6:

[1149] Users tap on furniture or home appliances they like in the image to access the corresponding online shop page. For example, tapping on a blue sofa opens the sofa's purchase page. Based on the tapped link information, the device launches the default web browser and displays the corresponding online shop page. The user can then proceed with the purchase directly on this page. The device receives the user's tap operation as input data, processes the data by launching a browser based on the link information, and outputs the online shop page.

[1150] (Application example 2)

[1151] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1152] Conventional room redecorating systems have difficulty making optimal suggestions based on a user's individual emotions and preferences, and can only provide general suggestions. As a result, some users are often left dissatisfied, which tends to reduce their motivation to purchase. Furthermore, when selecting furniture in a store, users are unable to virtually try out different furniture arrangements, which can result in results that are far removed from the user's expectations and imagination. There is a need to solve these issues, provide optimal room redecorating suggestions based on the user's individual emotions and preferences, and improve the shopping experience in physical stores.

[1153] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1154] In this invention, the server includes means for inputting an image of the current state of the room taken by the user, means for inputting the user's desired redecorating conditions, means for generating an image of the redecorated room using an artificial intelligence model based on the input image and the desired conditions, means for displaying the generated image on the user terminal, means for analyzing the user's emotions and optimizing the image, means for adding link information for products related to furniture or home appliances in the displayed image, and means for the user to access an online sales page for the relevant product by selecting the link information in the image. This allows the user to receive optimal redecorating suggestions based on their emotions and desires, and also allows them to try out virtual furniture arrangements in a physical store, which increases their desire to purchase and provides a highly satisfying shopping experience.

[1155] An "image of the current state of the room taken by the user" is an image showing the current state of the room taken by the user using a device such as a smartphone or camera.

[1156] "User's desired redecorating conditions" is information entered by the user regarding conditions such as style, budget, and color for the room redecorating desired by the user.

[1157] The "artificial intelligence model" is a model trained using machine learning algorithms to generate an image of a room after redecorating based on image data and conditions.

[1158] The "generated image" is a virtual image of the room after redecorating, created using an artificial intelligence model based on an image of the current room and the desired conditions.

[1159] A "user terminal" refers to an electronic device operated by a user, such as a smartphone, tablet, or PC.

[1160] "Means for analyzing user emotions and optimizing images" refers to means for analyzing the user's facial expressions and reactions using emotion analysis technology, and readjusting the generated image based on the results.

[1161] "Link information for furniture or home appliance-related products" is information such as the URL of an online sales page associated with the furniture or home appliance displayed in the generated image.

[1162] An "online sales page" is a web page where you can purchase products such as furniture and home appliances over the Internet.

[1163] This invention is a system that takes into account the user's emotions and makes optimal room redecorating suggestions. Users take photos of the current state of the room and input their desired conditions, and AI and an emotion analysis engine work together to generate an image of the room after redecorating, allowing them to purchase furniture and home appliances online based on that image.

[1164] The server has a means for inputting images of the current state of the room taken by the user and a means for inputting the user's desired redecorating conditions, and a means for generating an image of the room after redecorating using an artificial intelligence model based on the input images and desired conditions.

[1165] The generated image is displayed on the user's device, which can be a smartphone, tablet, PC, or other electronic device, allowing the user to check the generated image.

[1166] The server also has the means to analyze the user's emotions and optimize the images. The Emotion API is used for emotion analysis, and the generated images are adjusted based on the user's facial expressions and reactions. For example, the server determines whether the user is satisfied with the displayed image and changes the suggestions accordingly.

[1167] The furniture and home appliances in the generated images have links to online sales pages attached, so that when a user selects a particular piece of furniture or home appliance, they can access the online sales page for that product. This link information is achieved by the server adding the URL of the product related to the displayed image.

[1168] The following hardware and software are used to run the programs in this system:

[1169] Hardware:

[1170] User devices (smartphones, tablets, PCs)

[1171] Cloud Server

[1172] software:

[1173] Image analysis: OpenCV

[1174] Emotion analysis: EmotionAPI

[1175] Artificial intelligence model: Keras / TensorFlow

[1176] Data communication: HTTP request

[1177] Mobile App Development: React Native

[1178] As a concrete example, suppose a user takes a photo of the current state of a room and inputs desired conditions such as "Scandinavian style," "budget within 100,000 yen," and "blue as the main color." In this case, an example of a prompt would be as follows:

[1179] "Based on photos of a room you have taken, please generate an image of a redecorating proposal with a Scandinavian design and a blue theme, costing less than 100,000 yen. Please also take into account the user's emotional feedback to make the best proposal."

[1180] As a result, this invention is a system that can provide users with optimal redecorating suggestions that reflect their individual desires and feelings, and achieve a highly satisfying purchasing experience.

[1181] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1182] Step 1:

[1183] The user launches the app and takes a photo of the current state of the room. The device converts the photo into JPEG or PNG format and sends it to the server using an HTTP request. Specifically, the photo file is stored in temporary memory and sent to an API endpoint for uploading to the server.

[1184] Input: A photo of the current state of the room taken by the user

[1185] Output: A photo file in JPEG or PNG format is sent to the server.

[1186] Step 2:

[1187] The user enters desired conditions for the redecoration in a form within the app. For example, "Style: Scandinavian," "Budget: Under 100,000 yen," "Favorite color: Blue," etc. The device encodes these conditions into JSON format and sends it to the server using an HTTP request.

[1188] Input: Redecorating requirements entered by the user

[1189] Output: JSON formatted preference data is sent to the server

[1190] Step 3:

[1191] The server uses OpenCV to analyze the current room layout based on the received room photos and desired conditions, and specifically, uses an object detection algorithm to identify the location and type of major furniture in the room.

[1192] Input: JPEG or PNG photo files, desired conditions data in JSON format

[1193] Output: Data showing the location and type of furniture in the room

[1194] Step 4:

[1195] The server uses an artificial intelligence model to generate an image of the redecorated room that best matches the desired criteria. Using a model trained with Keras / TensorFlow, it suggests furniture and decorative items that fit the room design, the specified budget, and the blue theme.

[1196] Input: Data showing the location and type of furniture in the room, and desired conditions data in JSON format

[1197] Output: A room image generated based on your desired conditions

[1198] Step 5:

[1199] The server uses an emotion analysis engine to analyze the user's emotions. It analyzes the user's facial expressions using the Emotion API and optimizes the generated image based on the results. For example, if the user is not satisfied with the displayed image, it will be regenerated.

[1200] Input: Real-time facial expression images of the user, generated room images

[1201] Output: User sentiment analysis results, room images regenerated if necessary

[1202] Step 6:

[1203] The server sends the generated image to the terminal as an image file in JPEG or PNG format.

[1204] Input: Room image generated based on desired conditions

[1205] Output: A JPEG or PNG image file is sent to the user's device.

[1206] Step 7:

[1207] The device displays the received image to the user and uses an emotion engine to collect the user's emotional feedback in real time, and based on this feedback, regenerates the image if necessary.

[1208] Input: JPEG or PNG image files, real-time user emotion data

[1209] Output: Request for regeneration based on emotional feedback or confirmation of image display

[1210] Step 8:

[1211] The server adds link information associated with each piece of furniture or appliance in the displayed image, which is the URL of the online shop that corresponds to the selected piece of furniture or appliance, and maps it to a specific area of ​​the image.

[1212] Input: Image file in JPEG or PNG format

[1213] Output: Image file with link information added

[1214] Step 9:

[1215] The device displays image photos with link information attached to them to the user, and when the user taps on a specific piece of furniture or home appliance, the link information corresponding to that item is activated.

[1216] Input: Image file with link information

[1217] Output: Display the corresponding online sales page based on tap input

[1218] Step 10:

[1219] Users can tap on the furniture or appliance they like in the image to access the corresponding online shop page. For example, tapping on a blue sofa will open the sofa's purchase page.

[1220] Input: User tap input

[1221] Output: Online store page display of the product

[1222] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1223] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1224] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1225] [Fourth embodiment]

[1226] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1227] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1228] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1229] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1230] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1231] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1232] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1233] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1234] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1235] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1236] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1237] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1238] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1239] This invention is a system that makes it easy to rearrange a room. Users take photos of the current state of the room and input their desired conditions, and AI generates an image of the room after rearranging, allowing them to purchase furniture and home appliances in that image online. The program of the system of the present invention and its specific processing content are described below.

[1240] System program and specific processing

[1241] Step 1: Upload a photo of the current room

[1242] The user launches the app and takes a photo of the current state of the room.

[1243] The device converts the captured photo into JPEG or PNG format and sends this data to the server.

[1244] Step 2: Enter your desired criteria

[1245] Users enter their room style and desired conditions (e.g., Nordic style, budget within 100,000 yen, blue as the main color) in a form within the app.

[1246] The terminal encodes the entered desired conditions into JSON format or similar and sends them to the server.

[1247] Step 3: AI-generated redecorating image

[1248] Based on the received photos of the room and the desired conditions, the server uses image analysis technology and artificial intelligence models (such as deep learning models) to generate an image of the room after redecorating.

[1249] The server selects furniture and home appliances from a database that meet the user's desired conditions and incorporates them into the generated image.

[1250] Step 4: Displaying the image

[1251] The server sends the generated image photograph to the terminal.

[1252] The device displays the received image to the user, allowing the user to visually check the atmosphere of the room after the makeover.

[1253] Step 5: Link furniture and appliances

[1254] The server adds link information to online shops related to each piece of furniture or home appliance in the image photo.

[1255] The terminal displays the image photograph with the link information added again to the user, allowing the user to tap it.

[1256] Step 6: Buy furniture and appliances

[1257] Users tap on the furniture or home appliance they like and access the online shop page.

[1258] The terminal opens the page of the relevant online shop, and the user can complete the purchase procedure.

[1259] Specific examples

[1260] Here are some examples:

[1261] 1. Step 1: Upload a photo of your current room

[1262] The user opens the app, takes a photo of their living room, and presses the "upload" button within the app.

[1263] The device sends the captured photo (example.jpg) to the server.

[1264] 2. Step 2: Enter your desired conditions

[1265] Users enter their desired conditions, such as "Nordic style," "budget under 100,000 yen," and "blue as the main color," into a form within the app.

[1266] The device sends the desired conditions to the server in JSON format ({"style": "Nordic style", "budget": 100000, "color": "blue"}).

[1267] 3. Step 3: AI-generated redecorating image

[1268] The server analyzes the received photos and desired conditions, and uses an AI model to generate an image of the redecorated home.

[1269] The server selects images from a database that include Scandinavian-style furniture and blue accents and incorporates them into the photo.

[1270] 4. Step 4: Displaying the image

[1271] The server sends the generated image photo (new_living_room.jpg) to the terminal.

[1272] The terminal displays this image photo to the user, allowing the user to check the atmosphere of the living room after the makeover.

[1273] 5. Step 5: Linking furniture and appliances

[1274] The server adds link information related to the furniture and home appliances in the image.

[1275] The device redisplays the linked image photo to the user.

[1276] 6. Step 6: Buy furniture and appliances

[1277] The user taps on the blue sofa to access the online shop page.

[1278] The device will open the relevant shop page (e.g. https: / / example.com / blue-sofa) and the user can proceed with the purchase.

[1279] This invention allows users to rearrange their rooms reliably and easily, and helps them create more comfortable and ideal rooms.

[1280] The processing flow will be explained below.

[1281] Step 1:

[1282] The user launches the app and takes a photo of the current state of the room.

[1283] The device converts the captured photo into JPEG or PNG format and sends this data to the server. Specifically, the device stores the photo file in temporary memory and uploads it to the server using an HTTP request.

[1284] Step 2:

[1285] Users enter their desired redecorating requirements in a form within the app, such as "Style: Scandinavian," "Budget: Under 100,000 yen," and "Favorite color: Blue."

[1286] The device encodes the entered desired conditions into a format such as JSON and sends it to the server. In this case, when the user finishes entering the conditions and presses the "Send" button, the device sends the formatted data to the server as an HTTP request.

[1287] Step 3:

[1288] Based on the received photos of the room and the desired conditions, the server uses image analysis technology to analyze the current room layout. Specifically, it uses an object detection algorithm to identify the location and type of major furniture in the room.

[1289] The server then uses an artificial intelligence model (e.g., a deep learning model) to generate images of the redecorated room that best fit the desired criteria, suggesting furniture and decorative items with a Scandinavian design, within a specified budget, and a blue theme, for example.

[1290] Step 4:

[1291] The server sends the generated redecorating image to the device as an image file in JPEG or PNG format. During this process, the generated image is stored in memory and then sent to the device as an HTTP response.

[1292] The device displays the received image to the user, who can then visually confirm the atmosphere of the room after the makeover.

[1293] Step 5:

[1294] The server adds link information associated with each piece of furniture or appliance in the displayed image, which is the URL of the online shop that corresponds to the selected piece of furniture or appliance, and maps it to a specific area of ​​the image.

[1295] The device then displays the image with the link information attached to it again to the user. When the user taps on a specific piece of furniture or home appliance, the link information corresponding to that item is activated.

[1296] Step 6:

[1297] Users can tap on the furniture or appliance they like in the image to access the corresponding online shop page. For example, tapping on a blue sofa will open the sofa's purchase page.

[1298] Based on the link tapped, the device will launch a web browser and display the corresponding online shop page, where the user can proceed directly with the purchase.

[1299] The above is a specific processing flow in the system of the present invention.

[1300] Example 1

[1301] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1302] The traditional redecorating process is extremely complicated, requiring users to spend time and effort selecting and arranging furniture and appliances. It's also difficult for users to visualize the final image, often resulting in an unsatisfactory redecorating. Furthermore, purchasing furniture and appliances requires a separate process, making it inconvenient. To solve these issues, a system is needed that allows users to easily visualize the redecorating image and immediately purchase the necessary furniture and appliances.

[1303] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1304] In this invention, the server includes means for inputting an image of the current state of the room taken by the user, means for inputting the user's desired redecorating conditions, means for converting the input image into JPEG or PNG format and sending it to the server, means for encoding the desired conditions into JSON format and sending it to the server, means for generating an image of the redecorated room using an artificial intelligence model based on the input image and the desired conditions, means for selecting furniture and home appliances from a database based on the specified conditions and incorporating them into the generated image, means for sending the generated image to the user terminal and displaying it, means for adding link information for products related to the furniture or home appliances in the displayed image, and means for the user to access the online sales page for the corresponding product by selecting the link information in the image. This allows the user to easily visually check the redecorating image and immediately purchase the necessary furniture and home appliances.

[1305] "User" refers to a person who uses the system to rearrange a room.

[1306] "Server" refers to a central processing unit that processes data sent by the user and generates an image of the room after the makeover.

[1307] "Terminal" refers to an electronic device such as a computer or smartphone that a user uses to access the system.

[1308] "Means for converting images into JPEG or PNG format" refers to the process by which a user can convert a photo of a room into a standard image format.

[1309] "Means of encoding the desired conditions into JSON format" refers to the process of converting the desired conditions for the redecoration entered by the user into a JSON format that is easy for a computer to understand.

[1310] "Artificial intelligence model" refers to an algorithm or computational model that uses deep learning or machine learning to analyze data and generate new information.

[1311] "Image analysis" refers to the techniques and processes used to analyze input images and understand their contents.

[1312] "Furniture and home appliances" refers to items that a user should install or select when redecorating a room.

[1313] "Link information" refers to URLs to online shops and information pages for products related to the furniture and home appliances in the image.

[1314] "Online sales page" refers to a web page where users can purchase furniture or home appliances of their choice.

[1315] "Generated image" refers to a visual image of the room after redecorating, generated by an artificial intelligence model based on the user's desired conditions and a photo of the room's current state.

[1316] "Database" refers to an information storage device that the system uses to manage and store information about furniture and home appliances, as well as user preferences.

[1317] "Means for displaying" refers to the function by which a server or terminal visually presents generated images and information to a user.

[1318] This invention is a system that makes it easy to rearrange a room. A user takes a photo of the current state of the room and inputs desired conditions. A generative AI model generates an image of the room after rearranging, and furniture and home appliances displayed in the image can be purchased online. The program of the system of the present invention and its specific processing content are described below.

[1319] System configuration and program processing

[1320] Upload a photo of the current state of the room

[1321] The user launches the app and takes a photo of the current state of the room. The user then presses the "Upload" button to send the photo to the server. The device converts the photo into JPEG or PNG format and uploads it to the server via the Internet.

[1322] Enter your desired conditions

[1323] The user enters their room style and desired conditions in the form within the app. The user presses the "Submit" button to send the desired conditions to the server. The device encodes the entered desired conditions into JSON format and sends it to the server.

[1324] AI-based redecorating image generation

[1325] The server uses image analysis technology and a generative AI model to generate an image of the redecorated room based on the received photos of the room and the desired conditions. The server first analyzes the received photos using image analysis software (e.g., OpenCV) to recognize the room's layout and existing furniture. Next, based on the desired conditions, a generative AI model (e.g., Stable Diffusion) is used to generate a new Nordic-style layout. Furniture and decorations that match the conditions are selected from a database (e.g., PostgresSQL) for the generated image, and the final image is created.

[1326] Displaying image photos

[1327] The server sends the generated image to the device and displays it to the user. The generated image is transferred to the device using the Internet Protocol (HTTP), and the device displays the received image in an image viewer within the app to provide visual feedback to the user.

[1328] Linking furniture and appliances

[1329] The server adds link information for online shops related to each piece of furniture or home appliance in the image photo, and the device displays the image photo with the added link information again to the user. The server retrieves the URL of the online shop for the relevant furniture or home appliance from the database and embeds it in the image. The image with the added link information is sent again to the device, which displays it to the user.

[1330] Purchasing furniture and appliances

[1331] Users can tap on the furniture or home appliance they like to access the online shop page. When users tap on the furniture icon in the displayed image, the device will open the URL of the corresponding online shop in a browser or in-app browser, where they can proceed with the purchase.

[1332] Hardware and software used

[1333] The system uses the following hardware and software:

[1334] Hardware: User devices (smartphones, tablets, personal computers), servers

[1335] Software: Generative AI models (e.g., Stable Diffusion), image analysis software (e.g., OpenCV), databases (e.g., PostgresSQL), network protocols (e.g., HTTP)

[1336] Specific operation example:

[1337] Below are some specific examples and examples of prompts for the generative AI model.

[1338] Specific examples

[1339] 1. Upload a photo of the current state of the room

[1340] A user opens the app, takes a photo of their living room, and presses the "upload" button.

[1341] The device converts the captured photo (example.jpg) into JPEG format and sends it to the server.

[1342] 2. Enter your desired conditions

[1343] Users enter desired conditions such as "Nordic style," "budget under 100,000 yen," and "blue as the main color" into the form within the app and press the "Submit" button.

[1344] The device sends the desired conditions to the server in JSON format ({"style": "Nordic style", "budget": 100000, "color": "blue"}).

[1345] 3. AI-based redecorating image generation

[1346] The server analyzes the received photos and desired conditions using image analysis software, and generates an image of the redecorated space using a generative AI model. The server then selects furniture from its database, including Scandinavian-style furniture and blue accents, and incorporates them into the photos.

[1347] Prompt Sentence Examples

[1348] Below is an example of a prompt sentence to input to the generative AI model.

[1349] markdown

[1350] I'm thinking of creating a system to make it easier to redecorate a room. I'd like you to generate an image of a living room that contains the following criteria:

[1351] Current room photo: example.jpg

[1352] Desired style: Scandinavian style

[1353] Budget: Under 100,000 yen

[1354] Color: Blue

[1355] This image should include appropriate furniture and appliances, with links to purchase each item online.

[1356] This invention allows users to easily rearrange their rooms and helps them create their ideal rooms.

[1357] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1358] Step 1:

[1359] Upload a photo of the current state of the room

[1360] The user launches the app, takes a photo of the current state of the room, presses the "Upload" button, and saves the photo (e.g., example.jpg) on ​​the device.

[1361] Input: Room photo

[1362] How it works: The device converts the captured photo into JPEG or PNG format and uploads it to a server over the Internet. The device sends this image data using an HTTP request.

[1363] Output: Image file saved on the server

[1364] Step 2:

[1365] Enter your desired conditions

[1366] Users input their room style and desired conditions into a form within the app (e.g., "Scandinavian style," "budget within 100,000 yen," "blue as the main color"), and then press the "Submit" button to save the entered desired conditions on their device.

[1367] Input: Desired conditions (e.g., {"style": "Nordic style", "budget": 100000, "color": "blue"})

[1368] Specific operation: The terminal encodes the input information into JSON format and sends it to the server using an HTTP request.

[1369] Output: JSON data of desired conditions saved on the server

[1370] Step 3:

[1371] AI-based redecorating image generation

[1372] Based on the received photos of the room and the desired conditions, the server analyzes the photos using image analysis technology (e.g., OpenCV) and generates an image of the room after redecorating using a generative AI model (e.g., Stable Diffusion).

[1373] Input: JSON data of room photos and desired conditions

[1374] How it works: The server uses image analysis software to recognize the room's layout and existing furniture. It then uses an AI model to generate a new Nordic-inspired layout based on the desired settings. It then selects furniture and appliances from a database that match the settings and creates the final image.

[1375] Output: Image of the room after redecorating (e.g., new_living_room.jpg)

[1376] Step 4:

[1377] Displaying image photos

[1378] The server sends the generated image photo (e.g., new_living_room.jpg) to the terminal and displays it to the user.

[1379] Input: Image of the room after redecorating

[1380] Specific operation: The server sends image data to the device using the HTTP protocol. The device displays the received image in the image viewer within the app, allowing the user to visually confirm it.

[1381] Output: An image that can be viewed by the user

[1382] Step 5:

[1383] Linking furniture and appliances

[1384] The server adds link information for online shops related to each piece of furniture or home appliance in the image photo and sends it to the terminal.

[1385] Input: Image of the remodeled room, link to online shop

[1386] Specific operation: The server retrieves the URL of the online shop for the relevant furniture or home appliance from the database and embeds it in the image. The image with the link information added is then sent back to the terminal, which displays it to the user.

[1387] Output: Image with link information added

[1388] Step 6:

[1389] Purchasing furniture and appliances

[1390] Users tap on the furniture or home appliance they like and access the online shop page.

[1391] Input: Image photo with link information added

[1392] Specific operation: When the user taps on a furniture icon in the displayed image, the device will open the URL of the corresponding online store in the browser or in-app browser, where the user can proceed with the purchase.

[1393] Output: Display of online shop purchase page

[1394] (Application example 1)

[1395] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1396] Conventional redecorating support systems have had problems such as differences between the image the user actually sees and the resulting redecorating result, and the time and effort required to properly find the furniture and home appliances desired. Another issue is the lack of effective advertising methods to encourage the purchase of furniture and home appliances to be used in redecorating.

[1397] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1398] In this invention, the server includes means for inputting an image of the current state of the room photographed by the user, means for inputting the user's desired redecorating conditions, means for generating an image of the redecorated room using an artificial intelligence model based on the input image and the desired conditions, means for displaying the generated image on the user terminal, means for adding link information for products related to furniture or home appliances in the displayed image, means for the user to access an online sales page for the product by selecting the link information in the image, and means for displaying advertisements related to the furniture or home appliances in the generated image. This allows the user to easily purchase the furniture or home appliance they want while checking a more realistic image of the redecorated state, and further increases their desire to purchase through effective advertisements.

[1399] "Input means" refers to a method or interface for providing the user with the captured image and desired conditions for the system.

[1400] An "artificial intelligence model" is an algorithm that analyzes data, makes predictions, and generates an image of the room after it has been redecorated.

[1401] "Display means" refers to the method or technique for visually presenting the generated image on the user's terminal.

[1402] The "means for adding link information" is a technology that adds links to online sales pages related to the furniture and home appliances in the generated image to the image.

[1403] The "means for accessing" refers to a method or system in which a user can select link information in an image to move to a web page that offers the corresponding product.

[1404] The "means for displaying advertisements" refers to a method or technology for associating advertisements for furniture or home appliances within the generated image and presenting them to the user.

[1405] A system for realizing this invention includes a means for inputting an image of the current state of a room taken by a user and transmitting the user's desired redecorating conditions to a server. The server uses an artificial intelligence model to generate an image of the room after redecorating based on the received image and desired conditions, and transmits the generated image to the user's terminal.

[1406] The user terminal has a means for displaying the transmitted image and adding link information for products related to the furniture or home appliances in the displayed image. When the user selects the link information in the image, the user accesses the online sales page for the corresponding product. In addition, the server has a means for displaying advertisements for furniture or home appliances in the generated image, so the user can see advertisements for products used in redecorating.

[1407] Hardware and software used

[1408] Hardware: Smartphone used by the user

[1409] Software: Python, PIL (Python Imaging Library), requests library, artificial intelligence model (deep learning model)

[1410] Data processing and calculation

[1411] The server uses an AI model to analyze the image data of the current state of the room received from the user and generates an image of the redecorated room based on the desired conditions entered. This image incorporates furniture and home appliances that meet the desired conditions. The generated image is sent to the user's device, and link and advertising information is added. The user's device displays the generated image, along with the link and advertising information, allowing the user to access the product page selected.

[1412] Specific examples

[1413] As a concrete example, the process will be shown below when a user inputs conditions such as "Scandinavian style," "budget within 100,000 yen," and "based on blue." The user takes a photo and uploads it to the server, then inputs the set conditions. The server uses an AI model to generate an image of the redecorated home, selects furniture and appliances that take into account the interior style, budget, and color conditions, and sends the image to the user's device. An advertising link is added to this image, allowing the user to view it and purchase the product.

[1414] Example prompt sentence:

[1415] "Generate an image of how to redecorate the room in this photo in a Nordic style, with a budget of 100,000 yen or less and a blue color scheme."

[1416] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1417] Step 1:

[1418] The user takes a photo of the current state of the room and uploads the image to the device via the app. The device converts the image into JPEG or PNG format and sends this data to the server. The input is the photo of the current state of the room, and the output is the image data sent to the server.

[1419] Step 2:

[1420] The user enters desired redecorating conditions. The user enters conditions such as style, budget, and color (e.g., "Scandinavian style," "under 100,000 yen," and "blue") into a form within the app. The device encodes the entered desired conditions into JSON format and sends it to the server. The input is the desired conditions, and the output is the JSON-formatted data sent to the server.

[1421] Step 3:

[1422] The server uses an artificial intelligence model (deep learning model) to generate an image of the redecorated room based on the received images and desired conditions. Using image analysis technology and an AI model, the input image data is analyzed, and furniture and home appliances that meet the user's desired conditions are selected from a database to generate the image. The input is image data and desired condition data in JSON format, and the output is the generated image data.

[1423] Step 4:

[1424] The server sends the generated image photograph to the terminal. The terminal displays the received image photograph to the user, allowing the user to visually confirm the atmosphere of the room after the makeover. The input is the image data sent from the server, and the output is the image photograph displayed on the user's terminal.

[1425] Step 5:

[1426] The server adds link information for products related to the furniture and home appliances in the generated image. It associates an online shop link with each piece of furniture or home appliance in the image and sends it back to the terminal. The terminal then displays the image photo with the added link information to the user again, allowing the user to tap it. The input is the generated image data, and the output is the image photo with the added link information.

[1427] Step 6:

[1428] The user taps on the furniture or home appliance they like to access the online shop page. The device opens the corresponding online shop page, allowing the user to complete the purchase procedure. The input is the user's tap operation, and the output is the online shop's purchase page.

[1429] Step 7:

[1430] The server also displays advertisements for furniture and home appliances within the generated image. By retrieving advertising information from a database and incorporating it into the generated image, users can see advertisements for the products used in the makeover. The input is the generated image data and advertising information, and the output is an image with the advertisement displayed.

[1431] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1432] This invention is a system that optimally proposes room redecorating while taking into account the user's emotions. When the user takes a photo of the current state of the room and inputs their desired conditions, AI and an emotion engine work together to generate an image of the room after redecorating, and then enables online purchasing of furniture and home appliances based on that image. The program of the system of the present invention and its specific processing content are described below.

[1433] System program and specific processing

[1434] Step 1: Upload a photo of the current room

[1435] The user launches the app and takes a photo of the current state of the room.

[1436] The device converts the captured photo into JPEG or PNG format and sends this data to the server. Specifically, the device stores the photo file in temporary memory and uploads it to the server using an HTTP request.

[1437] Step 2: Enter your desired criteria

[1438] Users enter their desired redecorating requirements in a form within the app, such as "Style: Scandinavian," "Budget: Under 100,000 yen," and "Favorite color: Blue."

[1439] The device encodes the entered desired conditions into a format such as JSON and sends it to the server. In this case, when the user finishes entering the conditions and presses the "Send" button, the device sends the formatted data to the server as an HTTP request.

[1440] Step 3: Redecorating image generation using AI and emotion engine

[1441] Based on the received photos of the room and the desired conditions, the server uses image analysis technology to analyze the current room layout. Specifically, it uses an object detection algorithm to identify the location and type of major furniture in the room.

[1442] The server then uses an artificial intelligence model to generate images of the remodeled room that best fit the desired criteria, such as furniture and decorative items that match a Scandinavian design, a specified budget, or a blue theme. Additionally, an emotional engine analyzes the user's past responses and current emotional state and takes them into account when generating the optimal image.

[1443] Step 4: Displaying images and emotional feedback

[1444] The server sends the generated image to the device as a JPEG or PNG image file. During this process, the generated image is stored in memory and sent to the device in an HTTP response.

[1445] The device displays the received image to the user and uses an emotion engine to collect the user's emotional feedback in real time, and based on this feedback, regenerates the image if necessary.

[1446] Step 5: Link furniture and appliances

[1447] The server adds link information associated with each piece of furniture or appliance in the displayed image, which is the URL of the online shop that corresponds to the selected piece of furniture or appliance, and maps it to a specific area of ​​the image.

[1448] The device then displays the image with the link information attached to it again to the user. When the user taps on a specific piece of furniture or home appliance, the link information corresponding to that item is activated.

[1449] Step 6: Buy furniture and appliances

[1450] Users can tap on the furniture or appliance they like in the image to access the corresponding online shop page. For example, tapping on a blue sofa will open the sofa's purchase page.

[1451] Based on the link tapped, the device will launch a web browser and display the corresponding online shop page, where the user can proceed directly with the purchase.

[1452] Specific examples

[1453] Here are some examples:

[1454] 1. Step 1: Upload a photo of your current room

[1455] The user opens the app, takes a photo of their living room, and presses the "upload" button within the app.

[1456] The device sends the captured photo (example.jpg) to the server.

[1457] 2. Step 2: Enter your desired conditions

[1458] Users enter their desired conditions, such as "Nordic style," "budget under 100,000 yen," and "blue as the main color," into a form within the app.

[1459] The device sends the desired conditions to the server in JSON format ({"style": "Nordic style", "budget": 100000, "color": "blue"}).

[1460] 3. Step 3: Redecorating image generation using AI and emotion engine

[1461] The server analyzes the received photos and desired conditions, and generates an image of the redecoration using an AI model and emotion engine. It combines past user reaction data with current facial expression analysis to make the ideal proposal.

[1462] 4. Step 4: Displaying images and emotional feedback

[1463] The server sends the generated image photo (new_living_room.jpg) to the terminal.

[1464] The device displays this image to the user, while the emotion engine simultaneously analyzes the user's real-time reactions.

[1465] 5. Step 5: Linking furniture and appliances

[1466] The server adds link information about the furniture and home appliances in the image.

[1467] The device will display an image photo with a link to the user, allowing them to tap the link.

[1468] 6. Step 6: Buy furniture and appliances

[1469] Users tap to select furniture or home appliances and access the online shop page.

[1470] The terminal opens the linked web page, and the user completes the purchase procedure.

[1471] This system allows users to receive optimal room redecorating suggestions that reflect their own emotions, resulting in a more satisfying shopping experience.

[1472] The processing flow will be explained below.

[1473] Step 1:

[1474] The user launches the app and takes a photo of the current state of the room.

[1475] The device converts the captured photo into JPEG or PNG format and sends this data to the server. Specifically, the device temporarily stores the photo file in memory and then uploads it to the server using an HTTP request.

[1476] Step 2:

[1477] Users enter their desired redecorating requirements in a form within the app, such as "Style: Scandinavian," "Budget: Under 100,000 yen," and "Favorite color: Blue."

[1478] The device encodes the entered desired conditions into a format such as JSON and sends it to the server. In this case, when the user finishes entering the conditions and presses the "Send" button, the device sends the formatted data to the server as an HTTP request.

[1479] Step 3:

[1480] Based on the received photos of the room and the desired conditions, the server uses image analysis technology to analyze the current room layout. Specifically, it uses an object detection algorithm to identify the location and type of major furniture in the room.

[1481] The server then uses an artificial intelligence model to generate images of the remodeled room that best fit the desired criteria, such as furniture and decorative items that match a Scandinavian design, a specified budget, or a blue theme. Additionally, an emotional engine analyzes the user's past responses and current emotional state and takes them into account when generating the optimal image.

[1482] Step 4:

[1483] The server sends the generated image to the device as an image file in JPEG or PNG format. During this process, the generated image is stored in memory and then sent to the device as an HTTP response.

[1484] The device displays the received image to the user and uses an emotion engine to collect the user's emotional feedback in real time, and based on this feedback, regenerates the image if necessary.

[1485] Step 5:

[1486] The server adds link information associated with each piece of furniture or appliance in the displayed image, which is the URL of the online shop that corresponds to the selected piece of furniture or appliance, and maps it to a specific area of ​​the image.

[1487] The device then displays the image with the link information attached to it again to the user. When the user taps on a specific piece of furniture or home appliance, the link information corresponding to that item is activated.

[1488] Step 6:

[1489] Users can tap on the furniture or appliance they like in the image to access the corresponding online shop page. For example, tapping on a blue sofa will open the sofa's purchase page.

[1490] Based on the link tapped, the device will launch a web browser and display the corresponding online shop page, where the user can proceed directly with the purchase.

[1491] The above is a specific processing flow of the system of the present invention. This system allows users to receive optimal room redecorating suggestions that reflect their own emotions, resulting in a more satisfying shopping experience.

[1492] Example 2

[1493] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1494] Current room redecorating suggestion systems do not take into account the user's emotions when making suggestions, making it difficult for users to receive redecorating suggestions that best suit their desired conditions. Furthermore, since they are unable to reflect emotional feedback in real time, there are issues with the accuracy of suggestions and the level of satisfaction. Furthermore, the online purchasing process for suggested furniture and home appliances is complicated, hindering the smooth purchase process.

[1495] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1496] In this invention, the server includes means for inputting an image of the current state of the space taken by the user, means for inputting the user's desired conditions, means for generating an image of the space after redecorating using an artificial intelligence model based on the input image and the desired conditions, means for displaying the generated image on the user's terminal, means for analyzing the user's emotions in real time and reflecting the analysis in the generated image, means for adding link information related to items in the displayed image, and means for the user to access the online sales page of the corresponding item by selecting the link information in the image. This enables optimal redecorating suggestions that take the user's emotions into consideration and a smooth online purchasing process.

[1497] A "user" is an individual or group who wishes to use the system to redecorate their room.

[1498] A "current state image of a space" is image data taken by a user that shows the current state of a room, office, or the like.

[1499] "Desired conditions" is information indicating conditions such as the style, budget, and color desired by the user for redecorating.

[1500] An "artificial intelligence model" is an algorithm or learning model that analyzes and makes suggestions based on the user's desired conditions and current image.

[1501] An "image" is a visual suggestion of what a space might look like after redecorating, generated by an artificial intelligence model.

[1502] A "user terminal" is a device operated by a user, such as a smartphone, tablet, or PC.

[1503] "Emotion analysis means" refers to technology that analyzes a user's facial expressions and tone of voice to grasp the user's emotional state in real time.

[1504] "Link information" refers to the URLs and product information of online shops related to the furniture and home appliances in the image.

[1505] An "online sales page" is a web page for purchasing products that a user accesses by selecting link information within an image.

[1506] "Items" refers to items that are placed in a space, such as furniture and home appliances.

[1507] This invention is a system that proposes optimal room redecorating suggestions taking into account the user's emotions. The system includes multiple hardware and software components, allowing the user to receive high-quality room redecorating suggestions.

[1508] Hardware and Software Configuration

[1509] Server: A high-performance computer responsible for data processing and running AI models, including object detection algorithms and sentiment analysis engines.

[1510] Device: A device such as a smartphone, tablet, or PC that allows users to operate the interface. It is equipped with a camera and microphone with real-time emotion analysis capabilities.

[1511] Application software: This is the software that allows the user to operate the interface. It includes a form for inputting desired conditions, an image upload function, and a function for displaying the generated images.

[1512] Explanation of system processing

[1513] 1. Upload a photo of the current state of the room

[1514] The user launches the app and takes a photo of the current state of the room. The photo is saved in the device's temporary memory. The device then converts the photo into JPEG or PNG format and sends it to the server using an HTTP request.

[1515] 2. Enter your desired conditions

[1516] Users input their desired redecorating requirements into a form within the app. For example, "Style: Scandinavian," "Budget: Within 100,000 yen," and "Favorite color: Blue." The input data is encoded into JSON format by the device and sent to the server as an HTTP POST request.

[1517] 3. Redecorating image generation using AI and emotion engine

[1518] The server receives the photos and desired conditions sent by the user. It then uses image analysis technology to analyze the current layout of the room. This uses object detection algorithms such as YOLO. Based on the analyzed data and desired conditions, a generative AI model generates an image of the room after it has been redecorated. Furthermore, an emotion engine analyzes the user's emotions in real time and reflects them to make optimal suggestions.

[1519] 4. Displaying images and emotional feedback

[1520] The server sends the generated image to the device. The device displays the image to the user and simultaneously collects emotional feedback. Emotional feedback is collected by analyzing the user's facial expressions and tone of voice, and is sent to the server as needed. The server may regenerate the image based on this feedback.

[1521] 5. Linking furniture and appliances

[1522] The server adds link information related to each item in the generated image, including the URL of the online shop. The device then displays the image again to the user, including the link information. When the user taps on a specific item, the link information corresponding to that item is activated.

[1523] 6. Purchasing furniture and appliances

[1524] Users can tap on an item they like in the image to access the corresponding online shop page. The device will launch a web browser based on the link they tapped and display the corresponding online shop page. The user can then proceed with the purchase directly from this page.

[1525] Specific examples

[1526] Here are some examples:

[1527] 1. Upload a photo of the current state of the room

[1528] A user opens the app, takes a photo of their living room, and presses the "upload" button.

[1529] The device sends the captured photo (example.jpg) to the server.

[1530] 2. Enter your desired conditions

[1531] Users enter their desired conditions, such as "Nordic style," "budget under 100,000 yen," and "blue as the main color," into a form within the app.

[1532] The device sends the desired conditions to the server in JSON format ({"style": "Nordic style", "budget": 100000, "color": "blue"}).

[1533] 3. Redecorating image generation using AI and emotion engine

[1534] The server analyzes the received photos and desired conditions, and generates an image of the redecorated home using an AI model and emotion engine.

[1535] Prompt Sentence Examples

[1536] Here are some example prompts to input to a generative AI model:

[1537] "When a user uploads a photo of their current living room and enters desired conditions such as 'Scandinavian style,' 'budget within 100,000 yen,' and 'blue as the main color,' please analyze the current layout of the room and generate a proposed image of what the room will look like after redecorating. Please also take into account the user's past reaction data and current facial expression analysis to optimize the proposal."

[1538] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1539] Step 1:

[1540] The user launches the app and takes a photo of the current state of the space. The photo is saved in the device's internal memory. The device then converts the photo to JPEG or PNG format. An example of this conversion process is to use an image format conversion library. The converted image data is sent to the server using an HTTP request (POST request). The device receives the photo data as input data, performs format conversion as data processing, and generates format-converted image data as output data.

[1541] Step 2:

[1542] The user enters the desired conditions for the makeover into a form within the app. For example, they specify "Style: Nordic style," "Budget: Under 100,000 yen," and "Favorite color: Blue." This input form is displayed on the user's device. The device encodes the entered conditions into JSON format. Specifically, the input data is converted into JSON data such as {"style": "Nordic style," "budget": 100000, "color": "blue"}. This encoded data is sent to the server as an HTTP POST request. The device receives the desired conditions as input data, encodes them into JSON format as data processing, and generates the encoded data as output data.

[1543] Step 3:

[1544] The server uses image analysis technology to analyze the current layout of the space based on the received current photos and desired conditions. Here, it uses object detection algorithms such as YOLO to identify the location and type of major furniture in the room. This process receives the current photos and desired conditions as input data, performs object detection as data processing, and outputs information on the location and type of furniture in the room. Next, the server uses an AI model to generate an image of the space after redecorating. During this generation process, an image that optimally reflects the input desired conditions is generated based on a pre-trained model. The generated image data is obtained as output.

[1545] Step 4:

[1546] The device uses an emotion analysis engine to analyze the user's emotions in real time. Specifically, the user's facial expressions and tone of voice are captured through a camera and microphone, and analyzed using an emotion analysis algorithm. This generates information about the user's emotional state. The generated image is then sent as emotional feedback to the server. The device receives facial expressions and voice as input data, processes the data, performs emotion analysis, and outputs data about the user's emotional state. The server receives this and regenerates the image as needed.

[1547] Step 5:

[1548] The server adds related link information to each item in the generated image. Specifically, it maps the URL information of the corresponding online shop for each piece of furniture or home appliance to a specific area of ​​the image. The input data for this process are the generated image and link information, and the link information is added as data processing, and an image with the link information added is generated as output. The terminal displays this image with the link information again to the user.

[1549] Step 6:

[1550] Users tap on furniture or home appliances they like in the image to access the corresponding online shop page. For example, tapping on a blue sofa opens the sofa's purchase page. Based on the tapped link information, the device launches the default web browser and displays the corresponding online shop page. The user can then proceed with the purchase directly on this page. The device receives the user's tap operation as input data, processes the data by launching a browser based on the link information, and outputs the online shop page.

[1551] (Application example 2)

[1552] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1553] Conventional room redecorating systems have difficulty making optimal suggestions based on a user's individual emotions and preferences, and can only provide general suggestions. As a result, some users are often left dissatisfied, which tends to reduce their motivation to purchase. Furthermore, when selecting furniture in a store, users are unable to virtually try out different furniture arrangements, which can result in results that are far removed from the user's expectations and imagination. There is a need to solve these issues, provide optimal room redecorating suggestions based on the user's individual emotions and preferences, and improve the shopping experience in physical stores.

[1554] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1555] In this invention, the server includes means for inputting an image of the current state of the room taken by the user, means for inputting the user's desired redecorating conditions, means for generating an image of the redecorated room using an artificial intelligence model based on the input image and the desired conditions, means for displaying the generated image on the user terminal, means for analyzing the user's emotions and optimizing the image, means for adding link information for products related to furniture or home appliances in the displayed image, and means for the user to access an online sales page for the relevant product by selecting the link information in the image. This allows the user to receive optimal redecorating suggestions based on their emotions and desires, and also allows them to try out virtual furniture arrangements in a physical store, which increases their desire to purchase and provides a highly satisfying shopping experience.

[1556] An "image of the current state of the room taken by the user" is an image showing the current state of the room taken by the user using a device such as a smartphone or camera.

[1557] "User's desired redecorating conditions" is information entered by the user regarding conditions such as style, budget, and color for the room redecorating desired by the user.

[1558] The "artificial intelligence model" is a model trained using machine learning algorithms to generate an image of a room after redecorating based on image data and conditions.

[1559] The "generated image" is a virtual image of the room after redecorating, created using an artificial intelligence model based on an image of the current room and the desired conditions.

[1560] A "user terminal" refers to an electronic device operated by a user, such as a smartphone, tablet, or PC.

[1561] "Means for analyzing user emotions and optimizing images" refers to means for analyzing the user's facial expressions and reactions using emotion analysis technology, and readjusting the generated image based on the results.

[1562] "Link information for furniture or home appliance-related products" is information such as the URL of an online sales page associated with the furniture or home appliance displayed in the generated image.

[1563] An "online sales page" is a web page where you can purchase products such as furniture and home appliances over the Internet.

[1564] This invention is a system that takes into account the user's emotions and makes optimal room redecorating suggestions. Users take photos of the current state of the room and input their desired conditions, and AI and an emotion analysis engine work together to generate an image of the room after redecorating, allowing them to purchase furniture and home appliances online based on that image.

[1565] The server has a means for inputting images of the current state of the room taken by the user and a means for inputting the user's desired redecorating conditions, and a means for generating an image of the room after redecorating using an artificial intelligence model based on the input images and desired conditions.

[1566] The generated image is displayed on the user's device, which can be a smartphone, tablet, PC, or other electronic device, allowing the user to check the generated image.

[1567] The server also has the means to analyze the user's emotions and optimize the images. The Emotion API is used for emotion analysis, and the generated images are adjusted based on the user's facial expressions and reactions. For example, the server determines whether the user is satisfied with the displayed image and changes the suggestions accordingly.

[1568] The furniture and home appliances in the generated images have links to online sales pages attached, so that when a user selects a particular piece of furniture or home appliance, they can access the online sales page for that product. This link information is achieved by the server adding the URL of the product related to the displayed image.

[1569] The following hardware and software are used to run the programs in this system:

[1570] Hardware:

[1571] User devices (smartphones, tablets, PCs)

[1572] Cloud Server

[1573] software:

[1574] Image analysis: OpenCV

[1575] Emotion analysis: EmotionAPI

[1576] Artificial intelligence model: Keras / TensorFlow

[1577] Data communication: HTTP request

[1578] Mobile App Development: React Native

[1579] As a concrete example, suppose a user takes a photo of the current state of a room and inputs desired conditions such as "Scandinavian style," "budget within 100,000 yen," and "blue as the main color." In this case, an example of a prompt would be as follows:

[1580] "Based on photos of a room you have taken, please generate an image of a redecorating proposal with a Scandinavian design and a blue theme, costing less than 100,000 yen. Please also take into account the user's emotional feedback to make the best proposal."

[1581] As a result, this invention is a system that can provide users with optimal redecorating suggestions that reflect their individual desires and feelings, and achieve a highly satisfying purchasing experience.

[1582] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1583] Step 1:

[1584] The user launches the app and takes a photo of the current state of the room. The device converts the photo into JPEG or PNG format and sends it to the server using an HTTP request. Specifically, the photo file is stored in temporary memory and sent to an API endpoint for uploading to the server.

[1585] Input: A photo of the current state of the room taken by the user

[1586] Output: A photo file in JPEG or PNG format is sent to the server.

[1587] Step 2:

[1588] The user enters desired conditions for the redecoration in a form within the app. For example, "Style: Scandinavian," "Budget: Under 100,000 yen," "Favorite color: Blue," etc. The device encodes these conditions into JSON format and sends it to the server using an HTTP request.

[1589] Input: Redecorating requirements entered by the user

[1590] Output: JSON formatted preference data is sent to the server

[1591] Step 3:

[1592] The server uses OpenCV to analyze the current room layout based on the received room photos and desired conditions, and specifically, uses an object detection algorithm to identify the location and type of major furniture in the room.

[1593] Input: JPEG or PNG photo files, desired conditions data in JSON format

[1594] Output: Data showing the location and type of furniture in the room

[1595] Step 4:

[1596] The server uses an artificial intelligence model to generate an image of the redecorated room that best matches the desired criteria. Using a model trained with Keras / TensorFlow, it suggests furniture and decorative items that fit the room design, the specified budget, and the blue theme.

[1597] Input: Data showing the location and type of furniture in the room, and desired conditions data in JSON format

[1598] Output: A room image generated based on your desired conditions

[1599] Step 5:

[1600] The server uses an emotion analysis engine to analyze the user's emotions. It analyzes the user's facial expressions using the Emotion API and optimizes the generated image based on the results. For example, if the user is not satisfied with the displayed image, it will be regenerated.

[1601] Input: Real-time facial expression images of the user, generated room images

[1602] Output: User sentiment analysis results, room images regenerated if necessary

[1603] Step 6:

[1604] The server sends the generated image to the terminal as an image file in JPEG or PNG format.

[1605] Input: Room image generated based on desired conditions

[1606] Output: A JPEG or PNG image file is sent to the user's device.

[1607] Step 7:

[1608] The device displays the received image to the user and uses an emotion engine to collect the user's emotional feedback in real time, and based on this feedback, regenerates the image if necessary.

[1609] Input: JPEG or PNG image files, real-time user emotion data

[1610] Output: Request for regeneration based on emotional feedback or confirmation of image display

[1611] Step 8:

[1612] The server adds link information associated with each piece of furniture or appliance in the displayed image, which is the URL of the online shop that corresponds to the selected piece of furniture or appliance, and maps it to a specific area of ​​the image.

[1613] Input: Image file in JPEG or PNG format

[1614] Output: Image file with link information added

[1615] Step 9:

[1616] The device displays image photos with link information attached to them to the user, and when the user taps on a specific piece of furniture or home appliance, the link information corresponding to that item is activated.

[1617] Input: Image file with link information

[1618] Output: Display the corresponding online sales page based on tap input

[1619] Step 10:

[1620] Users can tap on the furniture or appliance they like in the image to access the corresponding online shop page. For example, tapping on a blue sofa will open the sofa's purchase page.

[1621] Input: User tap input

[1622] Output: Online store page display of the product

[1623] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1624] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1625] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1626] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1627] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1628] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1629] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1630] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1631] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1632] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1633] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1634] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1635] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1636] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1637] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1638] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1639] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1640] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1641] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1642] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1643] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1644] The following is further disclosed regarding the above embodiment.

[1645] (Claim 1)

[1646] A means for inputting an image of the current state of the room taken by a user;

[1647] A means for inputting desired redecorating conditions of the user;

[1648] A means for generating an image of the room after redecorating using an artificial intelligence model based on the input image and desired conditions;

[1649] means for displaying the generated image on a user terminal;

[1650] means for adding link information of products related to furniture or home appliances in the displayed image;

[1651] A means for a user to access the online sales page of the corresponding product by selecting the link information in the image;

[1652] A system including:

[1653] (Claim 2)

[1654] 10. The system of claim 1, further comprising means for optimizing furniture or appliances in the generated image to conform to user preferences.

[1655] (Claim 3)

[1656] 10. The system of claim 1, further comprising means for displaying price information for a piece of furniture or an appliance when that item is selected in the displayed image.

[1657] "Example 1"

[1658] (Claim 1)

[1659] A means for inputting an image of the current state of the room taken by a user;

[1660] A means for inputting desired redecorating conditions of the user;

[1661] A means to convert the input image into JPEG or PNG format and send it to the server,

[1662] A means to encode desired conditions into JSON format and send it to the server,

[1663] A means for generating an image of the room after redecorating using an artificial intelligence model based on the input image and desired conditions;

[1664] A means to select furniture and appliances from a database based on specified criteria and incorporate them into the generated image;

[1665] means for transmitting the generated image to a user terminal and displaying the image;

[1666] means for adding link information of products related to furniture or home appliances in the displayed image;

[1667] A means for a user to access the online sales page of the corresponding product by selecting the link information in the image;

[1668] A system including:

[1669] (Claim 2)

[1670] 10. The system of claim 1, further comprising means for optimizing furniture or appliances in the generated image to conform to user preferences.

[1671] (Claim 3)

[1672] 10. The system of claim 1, further comprising means for displaying price information for a piece of furniture or an appliance when that item is selected in the displayed image.

[1673] "Application Example 1"

[1674] (Claim 1)

[1675] A means for inputting an image of the current state of the room taken by a user;

[1676] A means for inputting desired redecorating conditions of the user;

[1677] A means for generating an image of the room after redecorating using an artificial intelligence model based on the input image and desired conditions;

[1678] means for displaying the generated image on a user terminal;

[1679] means for adding link information of products related to furniture or home appliances in the displayed image;

[1680] A means for a user to access the online sales page of the corresponding product by selecting the link information in the image;

[1681] means for displaying an advertisement related to the furniture or appliance in the generated image;

[1682] A system including:

[1683] (Claim 2)

[1684] 10. The system of claim 1, further comprising means for optimizing furniture or appliances in the generated image to conform to user preferences.

[1685] (Claim 3)

[1686] 10. The system of claim 1, further comprising means for displaying price information for a piece of furniture or an appliance when that item is selected in the displayed image.

[1687] "Example 2: Combining Emotion Engines"

[1688] (Claim 1)

[1689] A means for inputting a current image of the space taken by a user;

[1690] A means for inputting user desired conditions;

[1691] A means for generating an image of the space after redecorating using an artificial intelligence model based on the input image and desired conditions;

[1692] means for displaying the generated image on a user terminal;

[1693] A means to analyze the user's emotions in real time and reflect them in image generation,

[1694] means for adding link information relating to the item in the displayed image;

[1695] A means for a user to access the online sales page of the corresponding product by selecting the link information in the image;

[1696] A system including:

[1697] (Claim 2)

[1698] 10. The system of claim 1, further comprising means for optimizing the article in the generated image to conform to user preferences.

[1699] (Claim 3)

[1700] 10. The system of claim 1, further comprising means for displaying price information for an item upon selection of the item in the displayed image.

[1701] "Application example 2 when combining emotion engines"

[1702] (Claim 1)

[1703] A means for inputting an image of the current state of the room taken by a user;

[1704] A means for inputting desired redecorating conditions of the user;

[1705] A means for generating an image of the room after redecorating using an artificial intelligence model based on the input image and desired conditions;

[1706] means for displaying the generated image on a user terminal;

[1707] A means of analyzing user emotions and optimizing images;

[1708] means for adding link information of products related to furniture or home appliances in the displayed image;

[1709] A means for a user to access the online sales page of the corresponding product by selecting the link information in the image;

[1710] A system including:

[1711] (Claim 2)

[1712] 10. The system of claim 1, further comprising means for optimizing furniture or appliances in the generated image to conform to user preferences.

[1713] (Claim 3)

[1714] 10. The system of claim 1, further comprising means for displaying price information for a piece of furniture or an appliance when that item is selected in the displayed image. [Explanation of symbols]

[1715] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. A means for inputting an image of the current state of the room taken by a user; A means for inputting desired redecorating conditions of the user; A means for generating an image of the room after redecorating using an artificial intelligence model based on the input image and desired conditions; means for displaying the generated image on a user terminal; means for adding link information of products related to furniture or home appliances in the displayed image; A means for a user to access the online sales page of the corresponding product by selecting the link information in the image; A system including:

2. 10. The system of claim 1, further comprising means for optimizing furniture or appliances in the generated image to conform to user preferences.

3. 10. The system of claim 1, further comprising means for displaying price information for a piece of furniture or home appliance when that item is selected in the displayed image.

Images