system

Abstract

We provide the system. [Solution] A means for receiving a user's facial photograph and body shape information from a user terminal, Means for storing the user's facial photograph and body shape information in a database, A means for obtaining detailed information related to the product being viewed by the user from a database, A means for calling a generative AI model that creates a generated image based on the user's facial photograph, body shape information, and product details, A means for transmitting the image generated by the aforementioned AI model to the user terminal, A system that includes this.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot performed by at least one processor, the method including: receiving a user utterance; adding the user utterance to a prompt including an instruction sentence related to an explanation of a character of the chatbot; encoding the prompt; and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In online shopping, since users cannot actually try on clothes, they often feel anxious about the size and fit when purchasing, and are often unable to make the decision to purchase. In addition, problems such as the size not fitting after purchase or not being able to wear the clothes like a model occur frequently, resulting in frequent returns and exchanges. Such problems reduce customer satisfaction and also cause additional labor and costs for the operating company in handling returns. Therefore, there is a need for a system that can easily confirm the actual wearing feeling and size feeling when users purchase clothes online.

Means for Solving the Problems

[0005] The present invention provides a system in which a user uploads their facial photograph and body shape information to an online system, and based on that, generates a realistic image of the user wearing the clothing they have selected. Specifically, the system includes means for receiving the user's facial photograph and body shape information from a user terminal, means for storing the user's facial photograph and body shape information in a database, means for obtaining detailed information related to the product the user is viewing from the database, means for calling a generation AI model that creates a generated image based on the user's facial photograph, body shape information, and product details, and means for transmitting the image generated by the generation AI model to the user terminal. As a result, users can check a realistic image of themselves wearing clothes that fit them online, alleviate anxieties about size and fit, and increase their willingness to purchase.

[0006] A "user terminal" refers to electronic devices such as computers, smartphones, and tablets that users use to connect to the internet and conduct online shopping.

[0007] A "user profile picture" is a digital image that shows the features of a user's face and is uploaded to an online system by the user.

[0008] "Body type information" refers to data that shows the user's physical characteristics, such as height, weight, and bust / waist / hip measurements.

[0009] A "server" is a central computer that receives, stores, analyzes, and processes data sent from user terminals.

[0010] A "database" is a system for systematically storing and managing multiple types of data, such as facial photographs, body type information, and product details.

[0011] "Product details" refers to information that describes the characteristics of a product, such as the material, color, and size of the clothing.

[0012] A "generative AI model" is an artificial intelligence algorithm that generates realistic images of how the product would look when worn, based on the user's facial photograph, body shape information, and product details.

[0013] An "image representation" is an image created by a generative AI model that shows a realistic visual representation of what the user's selected clothing would look like when actually worn.

[0014] A "notification" is a message that informs the user that the generated image is now viewable on their device.

[0015] An "e-commerce site" is a website that enables the purchase of goods and services via the internet. [Brief explanation of the drawing]

[0016] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of the data processing device and smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] Shows an emotion map to which a plurality of emotions are mapped. [Figure 11] It is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] It is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] It is a sequence diagram showing the processing flow of the data processing system in Example 2 when the emotion engine is combined. [Figure 14] It is a sequence diagram showing the processing flow of the data processing system in Application Example 2 when the emotion engine is combined.

Mode for Carrying Out the Invention

[0017] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0018] First, the terms used in the following description will be described.

[0019] In the following embodiments, a numbered processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), etc.

[0020] In the following embodiments, a numbered RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0021] In the following embodiments, the signed storage is one or more non-volatile storage devices that store various programs and various parameters. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes.

[0022] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0023] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0024] [First Embodiment]

[0025] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0026] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0027] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0028] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0029] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0030] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0031] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0032] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0033] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0034] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0035] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0036] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0037] This invention provides a system that allows users to check the actual fit and size of clothing when purchasing it on an online shopping site (EC site). Specific embodiments of the system are described in detail below.

[0038] System Overview

[0039] This system consists of a user terminal, a server, and a generative AI model. The user uploads their facial photo and body shape information from the terminal, which the server receives and stores. The server retrieves detailed information about the products the user is viewing and uses the generative AI model to generate realistic images of the clothes being worn. The generated images are then sent to the user terminal, allowing the user to view online how the clothes would look on them.

[0040] Program processing details

[0041] 1. User registration and initial setup

[0042] The user uploads a photo of their face and detailed body information (e.g., height, weight, bust / waist / hip measurements) from their device. The device sends the entered photo and body information to the server. The server stores the received photo and body information in a database and generates a user profile.

[0043] 2. Acquisition and analysis of product information

[0044] The user browses the product page of clothing they are considering purchasing on an e-commerce site. The server retrieves detailed information about the product the user is viewing (material, color, size, etc.) from the database. The server analyzes the retrieved product details and prepares them for input into a generating AI model.

[0045] 3. Image generation using AI models

[0046] The server prepares a dataset for creating generated images based on the user's facial photograph, body type information, and product details. The server invokes a generative AI model and inputs the prepared dataset. Based on the input data, the generative AI model generates realistic images of the user wearing the selected clothing. The generated images are temporarily stored by the server.

[0047] 4. User Feedback

[0048] The server notifies the device that the generated wearing image is complete. The device displays the notification received from the server to the user, who then navigates to the product page to view the generated image. The device receives the worn image generated in response to the request from the server and displays it on the product details page. The user reviews the generated realistic wearing image and determines whether the product suits them.

[0049] Specific example

[0050] For example, if a female user is considering purchasing a linen summer dress, she uploads a photo of her face and body measurements (e.g., height 165cm, weight 55kg, bust 85cm, waist 65cm, hips 90cm) from her device. This information is sent to and stored on the server.

[0051] Next, when the user views the product page for a linen summer dress, the server retrieves the dress's details (linen material, yellow, size information S, M, L, XL) from the database. Based on the user's facial photo, body type information, and product details, the server calls a generative AI model to generate an image of the user wearing the dress.

[0052] The generated image is sent from the server to the user's terminal, and after the user confirms the notification, they check the generated image on the product page. This allows the user to check in advance whether the item suits them and whether the size is correct. Through this entire process, users can enjoy online shopping with peace of mind.

[0053] The following describes the processing flow.

[0054] Step 1:

[0055] Users input and upload their own facial photo and body information (height, weight, bust / waist / hip measurements, etc.) from their device.

[0056] Step 2:

[0057] The device sends the entered facial photo and body shape information to the server.

[0058] Step 3:

[0059] The server stores the received facial photograph and body shape information in a database and generates a user profile.

[0060] Step 4:

[0061] The user browses the product page of clothing they are considering purchasing on the e-commerce site.

[0062] Step 5:

[0063] The server retrieves detailed information about the product the user is viewing (such as material, color, and size) from the database.

[0064] Step 6:

[0065] The server analyzes the detailed information of the acquired products and prepares a dataset to be input into the generating AI model.

[0066] Step 7:

[0067] The server invokes a generative AI model to create generated images based on the user's facial photo, body type information, and product details.

[0068] Step 8:

[0069] The generative AI model generates realistic images of how the user's selected clothing would look when worn, based on the input dataset.

[0070] Step 9:

[0071] The server temporarily stores the generated image.

[0072] Step 10:

[0073] The server notifies the terminal that the generated image is complete.

[0074] Step 11:

[0075] The device receives a notification from the server and displays a message to the user indicating that the generated image is available for viewing.

[0076] Step 12:

[0077] The user checks the notification, navigates to the product page, and views the generated image.

[0078] Step 13:

[0079] The terminal requests an image from the server and displays the received image on the product details page.

[0080] Step 14:

[0081] Users can check whether the product suits them based on the generated realistic images of them wearing it, and then make a purchase decision.

[0082] (Example 1)

[0083] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0084] In online shopping, users cannot actually try on products, making it difficult to confirm whether purchased clothing fits them. This leads to problems such as incorrect sizing, the product not looking as expected, or unsuitable materials, resulting in decreased customer satisfaction and increased returns.

[0085] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0086] In this invention, the server includes means for receiving a user's facial photograph and body shape information from a user terminal; means for storing the user's facial photograph and body shape information in a database; means for obtaining detailed information related to the product the user is viewing from the database; means for calling a generation AI model that creates a generated image based on the user's facial photograph, body shape information, and product details; means for temporarily storing the image generated by the generation AI model; means for transmitting the generated image to the user terminal; and means for notifying the user terminal that the generated image is complete. This enables users to have a realistic try-on experience when shopping online and to check whether the product fits them before purchasing.

[0087] A "user terminal" is a device used by a user to access a system and input information, and includes smartphones, tablets, and personal computers.

[0088] A "server" is a computer system that processes and stores information received from a user terminal, calls a generation AI model, and sends the generated image to the user terminal.

[0089] A "face photograph" is image data of a user's face captured by a camera, and includes the user's facial features.

[0090] "Body shape information" refers to data that represents the user's physical characteristics, including height, weight, bust size, waist size, and hip size.

[0091] A "database" is a data storage system that stores facial photographs, body type information, and product details, and retrieves and updates them as needed.

[0092] "Product details" refers to information about the material, color, size, etc., of the product the user is viewing.

[0093] A "generative AI model" is an artificial intelligence model that generates realistic images of a user wearing a product they have selected, based on the user's facial photograph, body shape information, and product details.

[0094] An "image" is a visual representation of a user-selected product being worn, created by a generative AI model.

[0095] A "notification" is a message sent to inform the user that the generated image is complete, and includes push notifications, in-app notifications, and other similar messages.

[0096] "Temporary storage" refers to the process where the server temporarily stores the generated image, which is then used to send it to the user's terminal later.

[0097] "Decoding" refers to the process by which a user's terminal converts an image received from a server into a format that can be displayed.

[0098] This invention provides a system that allows users to check the actual fit and size of clothing when purchasing it on an online shopping site. Specific embodiments of the system are described in detail below.

[0099] System Configuration

[0100] This system consists of a user terminal, a server, and a generative AI model. The user terminal is implemented using devices such as smartphones, tablets, and personal computers. The server is a computer system that centrally manages information received from the user terminal and operates the generative AI model. The generative AI model uses deep learning models such as TENSORFLOW® or StyleGAN2 using PyTorch.

[0101] Receiving and storing data

[0102] Users input their facial photograph and detailed body information (e.g., height, weight, bust / waist / hip measurements) through a dedicated application or website, and send it from their device to the server. The server stores the received facial photograph and body information in a database. MySQL® or PostgreSQL is used for the database, and data is transferred securely using security protocols (SSL / TLS).

[0103] Product information acquisition and analysis

[0104] When a user views a product page for clothing they are considering purchasing on an e-commerce site, the server retrieves detailed information about that product (material, color, size, etc.) from a database or external API. The retrieved information is then analyzed and formatted for input into a generating AI model.

[0105] Image generation using a generative AI model

[0106] The server prepares a dataset for generating images based on the user's facial photograph, body type information, and product details. It then calls a generation AI model and inputs the prepared dataset. The generation AI model generates realistic images of the user wearing the selected product. These generated images are temporarily stored by the server.

[0107] User Feedback

[0108] The server notifies the user's device that the generated wearing image is complete. The device displays the notification received from the server to the user, and the user navigates to the product page to view the generated image. The device receives the worn image generated in response to the request from the server and displays it on the product details page. The user reviews the generated realistic wearing image and decides whether the product suits them.

[0109] Specific example

[0110] For example, if a female user is considering purchasing a linen summer dress, she would upload a photo of her face and body measurements (e.g., height 165cm, weight 55kg, bust 85cm, waist 65cm, hips 90cm) from her device. This information would then be sent to and stored on the server.

[0111] Next, when the user views the product page for a linen summer dress, the server retrieves the product details (linen material, yellow, size information S, M, L, XL) from the database. Based on the user's face photo, body type information, and product details, the server calls a generative AI model to generate an image of the user wearing the dress.

[0112] The generated image is sent from the server to the user's terminal, and after the user confirms the notification, they check the generated image on the product page. This allows the user to check in advance whether the item suits them and whether the size is correct. Through this entire process, users can enjoy online shopping with peace of mind.

[0113] Example of a prompt

[0114] "Please generate an image of the user wearing a summer dress made of linen. The user's face photo and body information are as follows: Height 165cm, Weight 55kg, Bust 85cm, Waist 65cm, Hips 90cm. The dress details are linen material, yellow, size M."

[0115] In this way, this system can significantly improve the user experience in online shopping.

[0116] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0117] Step 1:

[0118] Enter and submit user information

[0119] Input: User's profile picture, body information (e.g., height, weight, bust / waist / hip measurements)

[0120] Processing: Users enter their facial photo and body type information through a dedicated application or website.

[0121] Output: Facial photo and body shape information (packet format) sent from the user terminal to the server.

[0122] The user inputs their facial photo and body type information from their device. The device converts the input information into a packet format and sends it to the server using a security protocol (SSL / TLS).

[0123] Step 2:

[0124] User information storage

[0125] Input: User's face photo and body type information

[0126] Processing: The server saves the received user information to a database. Specifically, it uses an RDBMS such as MySQL or PostgreSQL.

[0127] Output: User's facial photo and body type information stored in the database

[0128] The server stores the received facial image and body type information in a database. Once the data is confirmed to have been saved, a user profile is generated.

[0129] Step 3:

[0130] Product information acquisition

[0131] Input: URL of the product page the user is viewing

[0132] Processing: The server parses the URL and retrieves product details (material, color, size, etc.) from the database or an external API.

[0133] Output: Retrieved product details

[0134] When a user opens a product page for clothing they are interested in on an e-commerce site, the server parses the URL of that product page and retrieves the necessary product details.

[0135] Step 4:

[0136] Product Analysis

[0137] Input: Retrieved product details

[0138] Processing: The server analyzes the product details and prepares them in a format for input into the generated AI model.

[0139] Output: Product data formatted for input into the generating AI model.

[0140] The server analyzes the detailed information of the retrieved products and converts it into a format suitable for the generated AI model. This includes information such as material, color, and size.

[0141] Step 5:

[0142] Preparing the dataset

[0143] Input: User's face photo, body type information, and formatted product data.

[0144] Processing: The server creates a dataset to be input into the generated AI model.

[0145] Output: Dataset for generative AI models

[0146] The server creates an input dataset based on the user's facial photo, body type information, and product details.

[0147] Step 6:

[0148] Calling an AI Model

[0149] Input: Dataset for generated AI models

[0150] Processing: The server calls a generative AI model (e.g., StyleGAN2) and inputs the dataset.

[0151] Output: Generated image

[0152] The server invokes a generative AI model using TensorFlow or PyTorch and inputs the prepared dataset. The model generates realistic images of the user wearing the selected product.

[0153] Step 7:

[0154] Saving the generated image

[0155] Input: Generated image

[0156] Processing: The server temporarily stores the generated image. It uses cloud storage such as Amazon S3 or Google Cloud Storage.

[0157] Output: Temporarily saved image

[0158] The generated image data is securely and quickly stored in cloud storage by the server.

[0159] Step 8:

[0160] User notifications

[0161] Input: Notification that the generated image has been saved.

[0162] Processing: The server notifies the user's device that the generated image is complete. This notification is sent using push notifications or in-app notifications.

[0163] Output: Notification sent to the user

[0164] The server notifies the user that the generated image is complete on their device. This notification is sent promptly using push notifications or in-app notifications.

[0165] Step 9:

[0166] Image acquisition

[0167] Input: Notifications confirmed by the user

[0168] Processing: The terminal sends a request to the server and downloads the generated image.

[0169] Output: Image downloaded to the device

[0170] After the user terminal confirms the notification, it sends a request to the server to securely download the image.

[0171] Step 10:

[0172] Displaying images

[0173] Input: Downloaded image

[0174] Processing: The terminal decodes the received image and displays it on the product details page.

[0175] Output: Image displayed on the product details page

[0176] The user's terminal decodes the image received from the server and displays it on the product details page. The user then uses this information to determine if the product is suitable for them.

[0177] (Application Example 1)

[0178] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0179] When purchasing items online, it's difficult for users to accurately determine how the product will actually look on them and whether it fits properly. This is especially true for clothing, where there's no opportunity to try it on, often leading to disappointment after purchase. To address this problem, there's a need for a system that allows users to check how the clothing will look and fit beforehand.

[0180] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0181] In this invention, the server includes means for receiving a user's facial photograph and body shape information from a user terminal; means for storing the user's facial photograph and body shape information in a database; means for obtaining detailed information related to the product the user is viewing from the database; means for calling a generation AI model that creates a generated image based on the user's facial photograph, body shape information, and product details; means for transmitting the image generated by the generation AI model to the user terminal; and means for temporarily storing the generated image in cloud storage. This makes it possible for a user to check in real time how the clothing will look on them and whether the size will fit them before purchasing the product.

[0182] A "user terminal" refers to a computer device operated by a user using online shopping, and includes devices such as smartphones, tablets, and personal computers.

[0183] A "face photograph" is image data of a user's face, used as identification information.

[0184] "Body shape information" refers to data representing the physical dimensions of the user's body, including information such as height, weight, and bust / waist / hip measurements.

[0185] A "database" is an information system that organizes and stores large amounts of data, and it is a place where user information and product information are stored.

[0186] "Product details" refers to all information about the product, including data such as material, color, size, and price.

[0187] A "generative AI model" is a model that uses artificial intelligence techniques, such as generative adversarial networks (GANs), to generate new images from specific input data.

[0188] A "generated image" is an image created by a generative AI model, simulating what a user would look like wearing a specific product.

[0189] "Cloud storage" is a remote storage service for saving and managing data over a network, and is used to store generated images and other user data.

[0190] A "notification" is the process of transmitting specific information to a user's device, including a message informing them that a new generated image is ready.

[0191] "Display" refers to the act of visually presenting information on a user's terminal, and is a means of showing the generated image to the user.

[0192] Modes for carrying out the invention

[0193] This invention provides a system that allows users to check the actual fit and size of clothing when purchasing it online. This system primarily consists of a user terminal, a server, and a generating AI model. Specific embodiments are described below.

[0194] System Overview

[0195] 1. User registration and initial setup

[0196] First, the user uploads their facial photo and body shape information from their device. The user's device receives and processes this information and sends it to the server. The server stores the received facial photo and body shape information in a database and generates a user profile.

[0197] 2. Acquisition and analysis of product information

[0198] When a user views a product page on an e-commerce site that they are considering purchasing, the server retrieves detailed information about that product (material, color, size, etc.) from its database. The server then analyzes the retrieved product information and prepares it for input into a generating AI model.

[0199] 3. Image generation using AI models

[0200] The server prepares a dataset for generating images based on the user's facial photo, body type information, and product details. Next, the server invokes a generative AI model and inputs the pre-prepared dataset. Based on this data, the generative AI model generates realistic images of the user wearing the selected clothing. The generated images are temporarily stored in cloud storage by the server.

[0201] 4. User Feedback

[0202] The server notifies the user's device when the generated wearing image is complete. The user's device displays the notification received from the server, and the user can then go to the product page to check the generated image. This allows the user to check in advance whether the item suits them and whether the size is correct, supporting their purchase decision.

[0203] Technologies and hardware / software used

[0204] Smartphone app: Application for iOS or Android®

[0205] Server: Cloud servers such as AWS (registered trademark)

[0206] Generative AI Model: A custom model based on GAN (Generative Adversarial Network).

[0207] Cloud storage: Cloud storage services such as AWS S3

[0208] Specific example

[0209] For example, consider a case where user A wants to purchase a summer dress made of linen. First, A registers with the app and uploads a photo of their face and body measurements. This information is sent to and stored on the server. Next, when A views the product page for the linen summer dress on the e-commerce site, the server retrieves the detailed information about the dress and uses a generative AI model to generate an image of A wearing the dress. The generated image is temporarily stored in cloud storage, and A is notified when the generation is complete. A receives this notification in the app and checks the generated image of them wearing the dress. This allows A to check the size and fit before purchasing, enabling them to proceed with the purchase with confidence.

[0210] Examples of prompt statements

[0211] User A is browsing the product page for a linen summer dress. The server retrieves the dress's material and size information, and a generation AI model generates a model of User A wearing the dress based on their face photo and body shape information. The app notifies User A of the generated image, and User A confirms the image.

[0212] Thus, the present invention provides a system that can improve the user's online shopping experience and enhance the accuracy of their purchase decisions.

[0213] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0214] Step 1:

[0215] User registration and initial setup

[0216] Users input and upload their facial photo and body measurements using a smartphone app. The user's device sends this information to the server. The server stores the received facial photo and body measurements in a database and generates a user profile. This profile includes detailed information such as the user's name, facial photo, height, weight, and bust / waist / hip measurements.

[0217] Input: User's profile picture, body information (height, weight, bust / waist / hip measurements, etc.)

[0218] Output: User profile is saved to the database.

[0219] Step 2:

[0220] Product information acquisition and analysis

[0221] When a user views a product page on an e-commerce site, their device sends a request for that page to the server. The server retrieves detailed product information from its database based on the product ID. This information includes the product's material, color, and size. The server analyzes this information and prepares a dataset for input into a generating AI model.

[0222] Input: Product ID

[0223] Output: Product details (material, color, size, etc.)

[0224] Step 3:

[0225] Image generation using AI models

[0226] The server inputs the prepared dataset (user's facial photo, body type information, and product details) into the generating AI model. Based on the input data, the generating AI model creates realistic images of the user wearing the selected clothing. The generated images are temporarily stored in cloud storage by the server.

[0227] Input: Dataset (user's profile picture, body type information, product details)

[0228] Output: Generated image

[0229] Step 4:

[0230] User Feedback

[0231] The server notifies the user's device that the generation of the generated wearing image is complete. The user's device receives this notification and displays it to the user. The user opens the app and checks the generated image. Based on this image, the user decides whether the product suits them and whether to purchase it.

[0232] Input: Generated image, user terminal

[0233] Output: Display of notifications and images.

[0234] Specifically, if a user selects a linen summer dress, the server retrieves the product's material and size information, and a generation AI model generates an image based on the user's facial photo and body shape information. The generated image is then notified to the user, who then makes a purchase decision based on the image. This smooth process allows users to enjoy online shopping with peace of mind.

[0235] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0236] This invention provides a system that allows users to check the actual fit and size of clothing when purchasing it on an online shopping site (EC site), and also recognizes the user's emotional state to suggest appropriate products. Specific embodiments of the system are described in detail below.

[0237] System Overview

[0238] This system consists of a user terminal, a server, a generative AI model, and an emotion engine. The user uploads their facial photo and body shape information from the terminal, which the server receives and stores. The server retrieves detailed information about the products the user is viewing and uses the generative AI model to generate realistic images of the clothes being worn. Furthermore, the emotion engine analyzes the user's facial expressions and behavioral data, recognizes the user's emotional state, adjusts the images generated by the generative AI model based on this, and notifies the user terminal. This allows the user to receive appropriate product suggestions and view images of themselves wearing clothes online.

[0239] Program processing details

[0240] 1. User registration and initial setup

[0241] The user inputs and uploads their facial photo and detailed body information (e.g., height, weight, bust / waist / hip measurements) from their device. The device sends the entered facial photo and body information to the server. The server stores the received facial photo and body information in a database and generates a user profile.

[0242] 2. Acquisition and analysis of product information

[0243] The user browses the product page of clothing they are considering purchasing on an e-commerce site. The server retrieves detailed information about the product the user is viewing (material, color, size, etc.) from the database. The server analyzes the retrieved product details and prepares a dataset to be input into a generative AI model.

[0244] 3. Analysis using an emotion engine

[0245] The terminal collects user facial expressions and behavioral data and sends it to the server. The server invokes an emotion engine and analyzes the user's emotional state based on the data. The emotional state recognized by the emotion engine is reflected in the dataset input to the generative AI model.

[0246] 4. Image generation using AI models

[0247] The server prepares a dataset for creating generated images based on the user's facial photograph, body type information, and product details. This also includes emotional states analyzed by the emotion engine. The server invokes a generative AI model and inputs the prepared dataset. Based on the input data, the generative AI model generates realistic images of the user wearing the clothing selected by the user. The generated images are temporarily stored by the server.

[0248] 5. User Feedback

[0249] The server notifies the device that the generated wearing image is complete. The device displays the notification received from the server to the user, who then navigates to the product page to view the generated image. The device receives the worn image generated in response to the request from the server and displays it on the product details page. The user checks whether the product suits them based on the generated realistic wearing image and makes a purchase decision.

[0250] Specific example

[0251] For example, if a female user is considering purchasing a linen summer dress, she uploads a photo of her face and body measurements (e.g., height 165cm, weight 55kg, bust 85cm, waist 65cm, hips 90cm) from her device. This information is sent to and stored on the server.

[0252] Next, when a user views the product page for a linen summer dress, the server retrieves detailed information about the dress (linen material, yellow, size information S, M, L, XL) from the database. Based on the user's facial photo and body type information, as well as the product details, the server prepares a dataset to create a generated image by adding the emotional state analyzed from the user's facial expression data and behavioral data using an emotion engine.

[0253] The generated image is sent from the server to the user's terminal, and after the user confirms the notification, they check the generated image on the product page. This allows the user to check in advance whether the item suits them and whether the size is correct. Furthermore, suggestions are made according to the user's emotional state, providing a more satisfying shopping experience. Through this entire process, users can enjoy online shopping with peace of mind.

[0254] The following describes the processing flow.

[0255] Step 1:

[0256] Users input and upload their own facial photo and body information (height, weight, bust / waist / hip measurements, etc.) from their device.

[0257] Step 2:

[0258] The device sends the entered facial photo and body shape information to the server.

[0259] Step 3:

[0260] The server stores the received facial photograph and body shape information in a database and generates a user profile.

[0261] Step 4:

[0262] The user browses the product page of clothing they are considering purchasing on the e-commerce site.

[0263] Step 5:

[0264] The server retrieves detailed information (such as material, color, and size) related to the product the user is viewing from the database.

[0265] Step 6:

[0266] The device collects facial and behavioral data while the user is browsing products and sends it to the server.

[0267] Step 7:

[0268] The server invokes the emotion engine to analyze the user's emotional state based on the received facial and behavioral data. This analysis result is added to the user's profile information.

[0269] Step 8:

[0270] The server prepares a dataset for generating images based on the user's facial photograph, body type information, product details, and emotional state determined by the emotion engine.

[0271] Step 9:

[0272] The server invokes the generative AI model and inputs the prepared dataset.

[0273] Step 10:

[0274] Based on the input data, the generated AI model generates a realistic image of the user wearing the selected clothing.

[0275] Step 11:

[0276] The server temporarily stores the generated image.

[0277] Step 12:

[0278] The server notifies the terminal that the generated image is complete.

[0279] Step 13:

[0280] The terminal receives the notification from the server and displays a message indicating that the generated image can be viewed by the user.

[0281] Step 14:

[0282] The user checks the notification and moves to the product page to view the generated image.

[0283] Step 15:

[0284] [[ID=3۹]] The terminal requests the image from the server and displays the received image on the product details page.

[0285] Step 16:

[0286] The user checks whether the product suits themselves based on the generated realistic image of wearing. Also, if appropriate products are presented according to the server's proposal according to the emotional state, the user also considers that proposal.

[0287] Step 17:

[0288] If the user determines that the product suits themselves, the user decides to purchase the product.

[0289] (Example 2)

[0290] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0291] In modern online shopping, users cannot physically try on products, making it difficult to check whether an item suits them or fits before purchasing. This often leads to returns or exchanges after the product arrives due to reasons such as the item not matching expectations or being the wrong size, resulting in decreased user satisfaction. Furthermore, there is a lack of appropriate product suggestions that take into account the user's emotional state, highlighting the need for improved satisfaction.

[0292] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0293] In this invention, the server includes means for receiving a user's facial photograph and body shape information from a user terminal; means for storing the user's facial photograph and body shape information in a database; means for obtaining detailed information related to the product the user is viewing from the database; means for collecting the user's facial expression and behavior data; means for analyzing the user's emotional state based on the facial expression and behavior data; means for calling a generation AI model that creates a generated image based on the user's facial photograph, body shape information, product details, and emotional state; means for temporarily storing the image generated by the generation AI model; and means for transmitting the generated image to the user terminal. As a result, the user can not only check in advance whether the product suits them and what size it is, but also receive appropriate product suggestions according to their emotional state.

[0294] A "user terminal" is an electronic device used by a user, which is used to input the user's facial photograph and body shape information, and to communicate with the server.

[0295] A "face photograph" is image data of the user's face, providing information for the system to perform facial recognition.

[0296] "Body type information" refers to information that indicates the user's physical characteristics, such as height, weight, and bust / waist / hip measurements.

[0297] A "server" is a computing system that receives and stores data sent from user terminals, and performs data processing and analysis.

[0298] A "database" is a system that manages and stores data such as users' facial photos, body type information, and product details.

[0299] "Product details" refers to information about the product the user is viewing, including material, color, size, etc.

[0300] "Facial expression data" refers to data that shows the user's facial expressions, and is collected using sensors such as cameras.

[0301] "Behavioral data" refers to data that shows a user's online actions, such as their mouse and keyboard operations.

[0302] "Emotional analysis means" refers to a system or method for analyzing a user's facial expression data and behavioral data to identify the user's emotional state.

[0303] A "generative AI model" is an artificial intelligence model that takes various data as input and generates realistic imagery that makes it appear as if the user is wearing the product they have selected.

[0304] An "image" is a virtual image generated by a generative AI model, showing the user wearing the product they selected.

[0305] "Temporary storage" is the process of saving generated image data for a certain period of time and making it accessible as needed.

[0306] This invention provides a system that enables users to confirm the actual wearing experience and size feeling when purchasing clothes on an online shopping site (EC site), and to recognize the user's emotional state and make appropriate product recommendations. The following will explain the specific embodiments of the system in detail.

[0307] Overview of the System

[0308] This system consists of a user terminal, a server, a generation AI model, and an emotion engine.

[0309] 1. User Terminal:

[0310] The user terminal is a device for the user to input face photos and body shape information and send them to the server. Specific examples include personal computers, smartphones, and tablets. The user terminal is equipped with a web camera or a front camera, enabling the collection of expression data and the recording of behavior data.

[0311] 2. Server:

[0312] The server receives the data sent by the user and stores it in the database. Specifically, it manages face photos, body shape information, product detailed information, expression data, etc. It also plays the role of analyzing the user's emotional state using the emotion engine and passing the results to the generation AI model.

[0313] 3. Generation AI Model:

[0314] Based on the input data (the user's face photo, body shape information, product detailed information, emotional state), the generation AI model generates a realistic image of the user wearing the selected clothes. The AI models used include computer vision technology and deep learning technology. [[ID=​​​​ An emotion engine is software that analyzes facial expression data and behavioral data to recognize a user's emotional state. This engine identifies a user's emotions using, for example, facial expression recognition algorithms and machine learning models.

[0317] Specific example

[0318] For example, if a female user is considering purchasing a summer dress made of linen, the system will operate through the following steps.

[0319] 1. The user terminal allows female users to input their facial photograph and body measurements (e.g., height 165cm, weight 55kg, bust 85cm, waist 65cm, hips 90cm) and send this information to the server.

[0320] 2. The server stores the received facial photo and body shape information in a database and generates a user profile. At the same time, it retrieves information that the user is viewing a product page for a linen summer dress.

[0321] 3. The device collects emotional data from the user through facial expression data while the user is viewing the product page. This includes, for example, smiles or frowns while viewing product details.

[0322] 4. The server inputs the collected facial expression data into the emotion engine and obtains results that analyze the user's emotional state in real time. Based on the results, the emotional state is reflected in the dataset that is input into the generative AI model.

[0323] 5. The generation AI model generates a realistic image of the user wearing a linen summer dress, based on the user's facial photo, body type information, product details, and the results of the emotion engine's analysis. This image is temporarily stored on the server.

[0324] 6. The server notifies the user's device that the generated wearing image is complete. The user's device displays this notification to the user. The user can check the generated image on the product page and check in advance whether it suits them.

[0325] Example of a prompt

[0326] "Please generate a realistic image of a 165cm, 55kg woman trying on a yellow linen summer dress she is considering purchasing. Please also ensure the user has a happy expression in the image."

[0327] By inputting this prompt into the AI ​​model, it becomes possible to generate realistic images of the garment being worn based on specific scenarios.

[0328] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0329] Step 1:

[0330] The user accesses the e-commerce site, enters a photo of their face and body information (e.g., height, weight, bust / waist / hip measurements) on their profile page, and clicks the "Upload" button. The input data (face photo, height, weight, bust / waist / hip measurements) is collected by the device. The device sends this information to the server using an HTTP POST request. The server stores the received face photo and body information in a database and generates a user profile. As output, the user profile is saved in the database.

[0331] Step 2:

[0332] A user views a product page for clothing they are considering purchasing on an e-commerce site. When a user opens a product page, the server monitors this access event and obtains the ID of the product the user is viewing. Based on the obtained product ID, the server retrieves detailed information about the product (material, color, size, etc.) from the database. As output, the server retrieves the detailed product information and passes it to the next analysis step.

[0333] Step 3:

[0334] The device collects facial expression data in real time through the user's camera and simultaneously monitors behavioral data such as the user's mouse and keyboard operations. This facial expression and behavioral data is transmitted to a server via a secure channel. The server inputs the received facial expression and behavioral data into an emotion engine, which analyzes the emotional state in real time. The emotion engine identifies the emotional state from the user's facial expressions and actions and reflects the results in a generating AI model. As output, the user's emotional state is analyzed and passed on to the next generation step.

[0335] Step 4:

[0336] The server prepares a dataset of generated images based on the user's facial photograph, body type information, product details, and the results of the emotion engine's analysis. The server inputs this dataset into the generative AI model. The generative AI model uses the input data, along with prompts, to generate realistic images of the user wearing the selected clothing. The generated images are temporarily stored on the server. The output is the generated realistic image.

[0337] Step 5:

[0338] The server notifies the user's device that the generated wearing image is complete. This notification is sent in real-time via WebSocket or push notification. The device receives the notification from the server and displays a pop-up notification to the user. The user confirms the notification and is redirected to the product page. The device receives the worn image generated in response to the request from the server and displays it on the product details page. As output, the user can review the generated image and make a final purchase decision.

[0339] (Application Example 2)

[0340] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0341] Conventional online shopping systems have made it difficult for users to check the actual fit and size of clothing they are considering purchasing, leading to problems such as high return rates and low customer satisfaction. Furthermore, because product suggestions are made uniformly without considering the user's emotional state, it has been impossible to suggest products that are best suited to each individual user. The objective of this invention is to support users' purchasing decisions and improve the customer experience by allowing them to check realistic images of how clothes look when worn online and by providing appropriate product suggestions based on their emotional state.

[0342] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving the user's facial photograph and body shape information, means for storing this information in a database, means for obtaining detailed information related to the product the user is viewing, means for recognizing the user's facial expression, means for analyzing the user's emotional state, means for calling a generation AI model that creates a generated image based on the user's facial photograph, body shape information, product details, and the user's emotional state, and means for transmitting the generated image to the user terminal. As a result, the user can check images of clothes that suit them in real time and receive product suggestions that match their emotions.

[0343] A "user terminal" is a device that allows a user to input or receive information, and specific examples include smartphones and head-mounted displays.

[0344] A "face photo" is a digital image of the user's face.

[0345] "Body type information" refers to the user's physical data, such as height, weight, and bust / waist / hip measurements.

[0346] A "database" is a place where information is stored, managed, and made easily accessible.

[0347] "Product details" refers to information that describes the characteristics of the product the user is viewing, such as its material, color, and size.

[0348] A "generated image" is an image created by a generative AI model that shows what a user would look like wearing the product.

[0349] A "generative AI model" is an artificial intelligence model that uses machine learning or deep learning techniques to create generated images from input data.

[0350] "Emotional state" refers to the psychological and emotional state of a user, obtained by analyzing their facial expressions and behavioral data.

[0351] "Customizing product recommendations" means taking into account the user's emotional state and history to individually recommend the most suitable products.

[0352] This invention provides a system that allows users to check the actual fit and size of clothing when purchasing it on an online shopping site, and also recognizes the user's emotional state to suggest appropriate products. This system consists of a user terminal, a server, a generative AI model, and an emotion engine.

[0353] System Configuration

[0354] The server is configured using the following hardware and software:

[0355] Hardware: High-performance servers, database servers, storage

[0356] software:

[0357] Sentiment analysis engine: Amazon Rekognition, Microsoft® Azure® Face API

[0358] Generative AI models: DALL-E, Stable Diffusion

[0359] Databases: Amazon RDS, Firebase Firestore

[0360] User terminal

[0361] Users access the system using a smartphone or head-mounted display (HMD). The user terminal performs the following actions:

[0362] 1. Enter your facial photo and body type information:

[0363] Users take a photo of their face using their device's camera and input body information such as height, weight, and bust / waist / hip measurements.

[0364] The terminal sends this information to the server.

[0365] 2. Recognition and transmission of facial expressions:

[0366] The device's camera recognizes the user's facial expressions in real time and sends that data to the server.

[0367] server

[0368] The server performs the following actions:

[0369] 1. Information storage and analysis:

[0370] The system stores the received facial photograph and body type information in a database and generates a user profile.

[0371] Retrieve detailed information about the product the user is viewing from the database.

[0372] 2. Analysis of emotional state:

[0373] The system analyzes facial expression data to determine the user's emotional state.

[0374] 3. Creating the generated image:

[0375] Based on the user's facial photo, body type information, product details, and emotional state, a generative AI model is invoked to create generated images.

[0376] The generated images are temporarily stored on the server.

[0377] 4. Customization of notifications and product suggestions:

[0378] The generated image is sent to the user's terminal, and product suggestions are customized based on the user's emotional state.

[0379] Specific example

[0380] For example, if a user is using their smartphone to access a virtual store and is looking for a casual shirt for summer:

[0381] 1. The user takes a photo of themselves with their smartphone camera and enters their body shape information.

[0382] 2. When a user selects a casual shirt, the server retrieves the product information.

[0383] 3. While the user is browsing the product page for a shirt, the smartphone camera analyzes the user's facial expressions and determines that they are showing interest.

[0384] 4. The server displays to the user in real time an image of the product being worn, generated based on the user's facial photo, body type information, product details, and emotional state.

[0385] 5. If the user looks satisfied, the app will suggest, "Would you also like some pants or accessories to go with this shirt?"

[0386] Example of a prompt

[0387] "Enter the user's profile picture, body type information, and product details (shirt material, color, size), and based on the user's emotional state (interest level), generate a real-time image of the product being worn, which will be displayed on their smartphone."

[0388] This system allows users to see images of clothes that suit them in real time and enjoy a comfortable shopping experience while receiving product suggestions tailored to their emotions.

[0389] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0390] Step 1:

[0391] The user enters a facial photograph and body information into a smartphone or head-mounted display (HMD). The entered information includes the user's facial photograph and detailed body data (e.g., height, weight, bust / waist / hip measurements). This data is sent to a server via the device. The server stores the received data in a database and generates a user profile.

[0392] Step 2:

[0393] A user views the product page for a piece of clothing they are considering purchasing within a virtual store. The server retrieves and analyzes detailed information about the selected product (e.g., material, color, size) from the database. This information is properly configured as it will be used in subsequent processing.

[0394] Step 3:

[0395] The device uses a camera to recognize the user's facial expressions in real time. The recognized facial expression data is analyzed using an emotion analysis engine (e.g., Amazon Rekognition, Microsoft Azure Face API). The analysis results in the user's emotional state (e.g., interest, satisfaction, dissatisfaction, etc.). This emotional state data is sent to the server.

[0396] Step 4:

[0397] The server creates a dataset based on the acquired emotional state data, combining it with the user's facial photograph, body shape information, and product details. The created dataset is then input into a generative AI model (e.g., DALL-E, Stable Diffusion). Based on this input data, the generative AI model generates realistic images of the user wearing the clothes.

[0398] Step 5:

[0399] The server temporarily stores the generated image. It then sends the generated image to the user's terminal. The terminal displays the received image to the user, providing visual feedback as if the user were actually trying on the clothes.

[0400] Step 6:

[0401] The user reviews the generated image. The device then re-analyzes the user's emotional state and sends the data to the server. The server customizes product suggestions based on the latest emotional state. For example, if the user's emotional state is "satisfied," it will suggest other related products (e.g., matching pants or accessories).

[0402] Step 7:

[0403] The server sends customized product suggestions to the user's terminal. The terminal displays details of the suggested products and images of the items being worn to the user. Based on this information, the user can make a purchase decision.

[0404] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0405] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0406] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0407] [Second Embodiment]

[0408] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0409] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0410] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0411] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0412] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0413] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0414] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0415] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0416] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0417] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0418] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0419] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0420] This invention provides a system that allows users to check the actual fit and size of clothing when purchasing it on an online shopping site (EC site). Specific embodiments of the system are described in detail below.

[0421] System Overview

[0422] This system consists of a user terminal, a server, and a generative AI model. The user uploads their facial photo and body shape information from the terminal, which the server receives and stores. The server retrieves detailed information about the products the user is viewing and uses the generative AI model to generate realistic images of the clothes being worn. The generated images are then sent to the user terminal, allowing the user to view online how the clothes would look on them.

[0423] Program processing details

[0424] 1. User registration and initial setup

[0425] The user uploads a photo of their face and detailed body information (e.g., height, weight, bust / waist / hip measurements) from their device. The device sends the entered photo and body information to the server. The server stores the received photo and body information in a database and generates a user profile.

[0426] 2. Acquisition and analysis of product information

[0427] The user browses the product page of clothing they are considering purchasing on an e-commerce site. The server retrieves detailed information about the product the user is viewing (material, color, size, etc.) from the database. The server analyzes the retrieved product details and prepares them for input into a generating AI model.

[0428] 3. Image generation using AI models

[0429] The server prepares a dataset for creating generated images based on the user's facial photograph, body type information, and product details. The server invokes a generative AI model and inputs the prepared dataset. Based on the input data, the generative AI model generates realistic images of the user wearing the selected clothing. The generated images are temporarily stored by the server.

[0430] 4. User Feedback

[0431] The server notifies the device that the generated wearing image is complete. The device displays the notification received from the server to the user, who then navigates to the product page to view the generated image. The device receives the worn image generated in response to the request from the server and displays it on the product details page. The user reviews the generated realistic wearing image and determines whether the product suits them.

[0432] Specific example

[0433] For example, if a female user is considering purchasing a linen summer dress, she uploads a photo of her face and body measurements (e.g., height 165cm, weight 55kg, bust 85cm, waist 65cm, hips 90cm) from her device. This information is sent to and stored on the server.

[0434] Next, when the user views the product page for a linen summer dress, the server retrieves the dress's details (linen material, yellow, size information S, M, L, XL) from the database. Based on the user's facial photo, body type information, and product details, the server calls a generative AI model to generate an image of the user wearing the dress.

[0435] The generated image is sent from the server to the user's terminal, and after the user confirms the notification, they check the generated image on the product page. This allows the user to check in advance whether the item suits them and whether the size is correct. Through this entire process, users can enjoy online shopping with peace of mind.

[0436] The following describes the processing flow.

[0437] Step 1:

[0438] Users input and upload their own facial photo and body information (height, weight, bust / waist / hip measurements, etc.) from their device.

[0439] Step 2:

[0440] The device sends the entered facial photo and body shape information to the server.

[0441] Step 3:

[0442] The server stores the received facial photograph and body shape information in a database and generates a user profile.

[0443] Step 4:

[0444] The user browses the product page of clothing they are considering purchasing on the e-commerce site.

[0445] Step 5:

[0446] The server retrieves detailed information about the product the user is viewing (such as material, color, and size) from the database.

[0447] Step 6:

[0448] The server analyzes the detailed information of the acquired products and prepares a dataset to be input into the generating AI model.

[0449] Step 7:

[0450] The server invokes a generative AI model to create generated images based on the user's facial photo, body type information, and product details.

[0451] Step 8:

[0452] The generative AI model generates realistic images of how the user's selected clothing would look when worn, based on the input dataset.

[0453] Step 9:

[0454] The server temporarily stores the generated image.

[0455] Step 10:

[0456] The server notifies the terminal that the generated image is complete.

[0457] Step 11:

[0458] The device receives a notification from the server and displays a message to the user indicating that the generated image is available for viewing.

[0459] Step 12:

[0460] The user checks the notification, navigates to the product page, and views the generated image.

[0461] Step 13:

[0462] The terminal requests an image from the server and displays the received image on the product details page.

[0463] Step 14:

[0464] Users can check whether the product suits them based on the generated realistic images of them wearing it, and then make a purchase decision.

[0465] (Example 1)

[0466] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0467] In online shopping, users cannot actually try on products, making it difficult to confirm whether purchased clothing fits them. This leads to problems such as incorrect sizing, the product not looking as expected, or unsuitable materials, resulting in decreased customer satisfaction and increased returns.

[0468] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0469] In this invention, the server includes means for receiving a user's facial photograph and body shape information from a user terminal; means for storing the user's facial photograph and body shape information in a database; means for obtaining detailed information related to the product the user is viewing from the database; means for calling a generation AI model that creates a generated image based on the user's facial photograph, body shape information, and product details; means for temporarily storing the image generated by the generation AI model; means for transmitting the generated image to the user terminal; and means for notifying the user terminal that the generated image is complete. This enables users to have a realistic try-on experience when shopping online and to check whether the product fits them before purchasing.

[0470] A "user terminal" is a device used by a user to access a system and input information, and includes smartphones, tablets, and personal computers.

[0471] A "server" is a computer system that processes and stores information received from a user terminal, calls a generation AI model, and sends the generated image to the user terminal.

[0472] A "face photograph" is image data of a user's face captured by a camera, and includes the user's facial features.

[0473] "Body shape information" refers to data that represents the user's physical characteristics, including height, weight, bust size, waist size, and hip size.

[0474] A "database" is a data storage system that stores facial photographs, body type information, and product details, and retrieves and updates them as needed.

[0475] "Product details" refers to information about the material, color, size, etc., of the product the user is viewing.

[0476] A "generative AI model" is an artificial intelligence model that generates realistic images of a user wearing a product they have selected, based on the user's facial photograph, body shape information, and product details.

[0477] An "image" is a visual representation of a user-selected product being worn, created by a generative AI model.

[0478] A "notification" is a message sent to inform the user that the generated image is complete, and includes push notifications, in-app notifications, and other similar messages.

[0479] "Temporary storage" refers to the process where the server temporarily stores the generated image, which is then used to send it to the user's terminal later.

[0480] "Decoding" refers to the process by which a user's terminal converts an image received from a server into a format that can be displayed.

[0481] This invention provides a system that allows users to check the actual fit and size of clothing when purchasing it on an online shopping site. Specific embodiments of the system are described in detail below.

[0482] System Configuration

[0483] This system consists of user terminals, a server, and a generative AI model. User terminals are implemented using devices such as smartphones, tablets, and personal computers. The server is a computer system that centrally manages information received from user terminals and operates the generative AI model. The generative AI model uses deep learning models such as StyleGAN2, which utilizes TensorFlow or PyTorch.

[0484] Receiving and storing data

[0485] Users input their facial photo and detailed body information (e.g., height, weight, bust / waist / hip measurements) through a dedicated application or website, and send it from their device to the server. The server stores the received facial photo and body information in a database. MySQL or PostgreSQL is used for the database, and data is transferred securely using security protocols (SSL / TLS).

[0486] Product information acquisition and analysis

[0487] When a user views a product page for clothing they are considering purchasing on an e-commerce site, the server retrieves detailed information about that product (material, color, size, etc.) from a database or external API. The retrieved information is then analyzed and formatted for input into a generating AI model.

[0488] Image generation using a generative AI model

[0489] The server prepares a dataset for generating images based on the user's facial photograph, body type information, and product details. It then calls a generation AI model and inputs the prepared dataset. The generation AI model generates realistic images of the user wearing the selected product. These generated images are temporarily stored by the server.

[0490] User Feedback

[0491] The server notifies the user's device that the generated wearing image is complete. The device displays the notification received from the server to the user, and the user navigates to the product page to view the generated image. The device receives the worn image generated in response to the request from the server and displays it on the product details page. The user reviews the generated realistic wearing image and decides whether the product suits them.

[0492] Specific example

[0493] For example, if a female user is considering purchasing a linen summer dress, she would upload a photo of her face and body measurements (e.g., height 165cm, weight 55kg, bust 85cm, waist 65cm, hips 90cm) from her device. This information would then be sent to and stored on the server.

[0494] Next, when the user views the product page for a linen summer dress, the server retrieves the product details (linen material, yellow, size information S, M, L, XL) from the database. Based on the user's face photo, body type information, and product details, the server calls a generative AI model to generate an image of the user wearing the dress.

[0495] The generated image is sent from the server to the user's terminal, and after the user confirms the notification, they check the generated image on the product page. This allows the user to check in advance whether the item suits them and whether the size is correct. Through this entire process, users can enjoy online shopping with peace of mind.

[0496] Example of a prompt

[0497] "Please generate an image of the user wearing a summer dress made of linen. The user's face photo and body information are as follows: Height 165cm, Weight 55kg, Bust 85cm, Waist 65cm, Hips 90cm. The dress details are linen material, yellow, size M."

[0498] In this way, this system can significantly improve the user experience in online shopping.

[0499] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0500] Step 1:

[0501] Enter and submit user information

[0502] Input: User's profile picture, body information (e.g., height, weight, bust / waist / hip measurements)

[0503] Processing: Users enter their facial photo and body type information through a dedicated application or website.

[0504] Output: Facial photo and body shape information (packet format) sent from the user terminal to the server.

[0505] The user inputs their facial photo and body type information from their device. The device converts the input information into a packet format and sends it to the server using a security protocol (SSL / TLS).

[0506] Step 2:

[0507] User information storage

[0508] Input: User's face photo and body type information

[0509] Processing: The server saves the received user information to a database. Specifically, it uses an RDBMS such as MySQL or PostgreSQL.

[0510] Output: User's facial photo and body type information stored in the database

[0511] The server stores the received facial image and body type information in a database. Once the data is confirmed to have been saved, a user profile is generated.

[0512] Step 3:

[0513] Product information acquisition

[0514] Input: URL of the product page the user is viewing

[0515] Processing: The server parses the URL and retrieves product details (material, color, size, etc.) from the database or an external API.

[0516] Output: Retrieved product details

[0517] When a user opens a product page for clothing they are interested in on an e-commerce site, the server parses the URL of that product page and retrieves the necessary product details.

[0518] Step 4:

[0519] Product Analysis

[0520] Input: Retrieved product details

[0521] Processing: The server analyzes the product details and prepares them in a format for input into the generated AI model.

[0522] Output: Product data formatted for input into the generating AI model.

[0523] The server analyzes the detailed information of the retrieved products and converts it into a format suitable for the generated AI model. This includes information such as material, color, and size.

[0524] Step 5:

[0525] Preparing the dataset

[0526] Input: User's face photo, body type information, and formatted product data.

[0527] Processing: The server creates a dataset to be input into the generated AI model.

[0528] Output: Dataset for generative AI models

[0529] The server creates an input dataset based on the user's facial photo, body type information, and product details.

[0530] Step 6:

[0531] Calling an AI Model

[0532] Input: Dataset for generated AI models

[0533] Processing: The server calls a generative AI model (e.g., StyleGAN2) and inputs the dataset.

[0534] Output: Generated image

[0535] The server invokes a generative AI model using TensorFlow or PyTorch and inputs the prepared dataset. The model generates realistic images of the user wearing the selected product.

[0536] Step 7:

[0537] Saving the generated image

[0538] Input: Generated image

[0539] Processing: The server temporarily stores the generated image. It uses cloud storage such as Amazon S3 or Google Cloud Storage.

[0540] Output: Temporarily saved image

[0541] The generated image data is securely and quickly stored in cloud storage by the server.

[0542] Step 8:

[0543] User notifications

[0544] Input: Notification that the generated image has been saved.

[0545] Processing: The server notifies the user's device that the generated image is complete. This notification is sent using push notifications or in-app notifications.

[0546] Output: Notification sent to the user

[0547] The server notifies the user that the generated image is complete on their device. This notification is sent promptly using push notifications or in-app notifications.

[0548] Step 9:

[0549] Image acquisition

[0550] Input: Notifications confirmed by the user

[0551] Processing: The terminal sends a request to the server and downloads the generated image.

[0552] Output: Image downloaded to the device

[0553] After the user terminal confirms the notification, it sends a request to the server to securely download the image.

[0554] Step 10:

[0555] Displaying images

[0556] Input: Downloaded image

[0557] Processing: The terminal decodes the received image and displays it on the product details page.

[0558] Output: Image displayed on the product details page

[0559] The user's terminal decodes the image received from the server and displays it on the product details page. The user then uses this information to determine if the product is suitable for them.

[0560] (Application Example 1)

[0561] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0562] When purchasing items online, it's difficult for users to accurately determine how the product will actually look on them and whether it fits properly. This is especially true for clothing, where there's no opportunity to try it on, often leading to disappointment after purchase. To address this problem, there's a need for a system that allows users to check how the clothing will look and fit beforehand.

[0563] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0564] In this invention, the server includes means for receiving a user's facial photograph and body shape information from a user terminal; means for storing the user's facial photograph and body shape information in a database; means for obtaining detailed information related to the product the user is viewing from the database; means for calling a generation AI model that creates a generated image based on the user's facial photograph, body shape information, and product details; means for transmitting the image generated by the generation AI model to the user terminal; and means for temporarily storing the generated image in cloud storage. This makes it possible for a user to check in real time how the clothing will look on them and whether the size will fit them before purchasing the product.

[0565] A "user terminal" refers to a computer device operated by a user using online shopping, and includes devices such as smartphones, tablets, and personal computers.

[0566] A "face photograph" is image data of a user's face, used as identification information.

[0567] "Body shape information" refers to data representing the physical dimensions of the user's body, including information such as height, weight, and bust / waist / hip measurements.

[0568] A "database" is an information system that organizes and stores large amounts of data, and it is a place where user information and product information are stored.

[0569] "Product details" refers to all information about the product, including data such as material, color, size, and price.

[0570] A "generative AI model" is a model that uses artificial intelligence techniques, such as generative adversarial networks (GANs), to generate new images from specific input data.

[0571] A "generated image" is an image created by a generative AI model, simulating what a user would look like wearing a specific product.

[0572] "Cloud storage" is a remote storage service for saving and managing data over a network, and is used to store generated images and other user data.

[0573] A "notification" is the process of transmitting specific information to a user's device, including a message informing them that a new generated image is ready.

[0574] "Display" refers to the act of visually presenting information on a user's terminal, and is a means of showing the generated image to the user.

[0575] Modes for carrying out the invention

[0576] This invention provides a system that allows users to check the actual fit and size of clothing when purchasing it online. This system primarily consists of a user terminal, a server, and a generating AI model. Specific embodiments are described below.

[0577] System Overview

[0578] 1. User registration and initial setup

[0579] First, the user uploads their facial photo and body shape information from their device. The user's device receives and processes this information and sends it to the server. The server stores the received facial photo and body shape information in a database and generates a user profile.

[0580] 2. Acquisition and analysis of product information

[0581] When a user views a product page on an e-commerce site that they are considering purchasing, the server retrieves detailed information about that product (material, color, size, etc.) from its database. The server then analyzes the retrieved product information and prepares it for input into a generating AI model.

[0582] 3. Image generation using AI models

[0583] The server prepares a dataset for generating images based on the user's facial photo, body type information, and product details. Next, the server invokes a generative AI model and inputs the pre-prepared dataset. Based on this data, the generative AI model generates realistic images of the user wearing the selected clothing. The generated images are temporarily stored in cloud storage by the server.

[0584] 4. User Feedback

[0585] The server notifies the user's device when the generated wearing image is complete. The user's device displays the notification received from the server, and the user can then go to the product page to check the generated image. This allows the user to check in advance whether the item suits them and whether the size is correct, supporting their purchase decision.

[0586] Technologies and hardware / software used

[0587] Smartphone app: Application for iOS or Android

[0588] Server: Cloud servers such as AWS

[0589] Generative AI Model: A custom model based on GAN (Generative Adversarial Network).

[0590] Cloud storage: Cloud storage services such as AWS S3

[0591] Specific example

[0592] For example, consider a case where user A wants to purchase a summer dress made of linen. First, A registers with the app and uploads a photo of their face and body measurements. This information is sent to and stored on the server. Next, when A views the product page for the linen summer dress on the e-commerce site, the server retrieves the detailed information about the dress and uses a generative AI model to generate an image of A wearing the dress. The generated image is temporarily stored in cloud storage, and A is notified when the generation is complete. A receives this notification in the app and checks the generated image of them wearing the dress. This allows A to check the size and fit before purchasing, enabling them to proceed with the purchase with confidence.

[0593] Examples of prompt statements

[0594] User A is browsing the product page for a linen summer dress. The server retrieves the dress's material and size information, and a generation AI model generates a model of User A wearing the dress based on their face photo and body shape information. The app notifies User A of the generated image, and User A confirms the image.

[0595] Thus, the present invention provides a system that can improve the user's online shopping experience and enhance the accuracy of their purchase decisions.

[0596] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0597] Step 1:

[0598] User registration and initial setup

[0599] Users input and upload their facial photo and body measurements using a smartphone app. The user's device sends this information to the server. The server stores the received facial photo and body measurements in a database and generates a user profile. This profile includes detailed information such as the user's name, facial photo, height, weight, and bust / waist / hip measurements.

[0600] Input: User's profile picture, body information (height, weight, bust / waist / hip measurements, etc.)

[0601] Output: User profile is saved to the database.

[0602] Step 2:

[0603] Product information acquisition and analysis

[0604] When a user views a product page on an e-commerce site, their device sends a request for that page to the server. The server retrieves detailed product information from its database based on the product ID. This information includes the product's material, color, and size. The server analyzes this information and prepares a dataset for input into a generating AI model.

[0605] Input: Product ID

[0606] Output: Product details (material, color, size, etc.)

[0607] Step 3:

[0608] Image generation using AI models

[0609] The server inputs the prepared dataset (user's facial photo, body type information, and product details) into the generating AI model. Based on the input data, the generating AI model creates realistic images of the user wearing the selected clothing. The generated images are temporarily stored in cloud storage by the server.

[0610] Input: Dataset (user's profile picture, body type information, product details)

[0611] Output: Generated image

[0612] Step 4:

[0613] User Feedback

[0614] The server notifies the user's device that the generation of the generated wearing image is complete. The user's device receives this notification and displays it to the user. The user opens the app and checks the generated image. Based on this image, the user decides whether the product suits them and whether to purchase it.

[0615] Input: Generated image, user terminal

[0616] Output: Display of notifications and images.

[0617] Specifically, if a user selects a linen summer dress, the server retrieves the product's material and size information, and a generation AI model generates an image based on the user's facial photo and body shape information. The generated image is then notified to the user, who then makes a purchase decision based on the image. This smooth process allows users to enjoy online shopping with peace of mind.

[0618] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0619] This invention provides a system that allows users to check the actual fit and size of clothing when purchasing it on an online shopping site (EC site), and also recognizes the user's emotional state to suggest appropriate products. Specific embodiments of the system are described in detail below.

[0620] System Overview

[0621] This system consists of a user terminal, a server, a generative AI model, and an emotion engine. The user uploads their facial photo and body shape information from the terminal, which the server receives and stores. The server retrieves detailed information about the products the user is viewing and uses the generative AI model to generate realistic images of the clothes being worn. Furthermore, the emotion engine analyzes the user's facial expressions and behavioral data, recognizes the user's emotional state, adjusts the images generated by the generative AI model based on this, and notifies the user terminal. This allows the user to receive appropriate product suggestions and view images of themselves wearing clothes online.

[0622] Program processing details

[0623] 1. User registration and initial setup

[0624] The user inputs and uploads their facial photo and detailed body information (e.g., height, weight, bust / waist / hip measurements) from their device. The device sends the entered facial photo and body information to the server. The server stores the received facial photo and body information in a database and generates a user profile.

[0625] 2. Acquisition and analysis of product information

[0626] The user browses the product page of clothing they are considering purchasing on an e-commerce site. The server retrieves detailed information about the product the user is viewing (material, color, size, etc.) from the database. The server analyzes the retrieved product details and prepares a dataset to be input into a generative AI model.

[0627] 3. Analysis using an emotion engine

[0628] The terminal collects user facial expressions and behavioral data and sends it to the server. The server invokes an emotion engine and analyzes the user's emotional state based on the data. The emotional state recognized by the emotion engine is reflected in the dataset input to the generative AI model.

[0629] 4. Image generation using AI models

[0630] The server prepares a dataset for creating generated images based on the user's facial photograph, body type information, and product details. This also includes emotional states analyzed by the emotion engine. The server invokes a generative AI model and inputs the prepared dataset. Based on the input data, the generative AI model generates realistic images of the user wearing the clothing selected by the user. The generated images are temporarily stored by the server.

[0631] 5. User Feedback

[0632] The server notifies the device that the generated wearing image is complete. The device displays the notification received from the server to the user, who then navigates to the product page to view the generated image. The device receives the worn image generated in response to the request from the server and displays it on the product details page. The user checks whether the product suits them based on the generated realistic wearing image and makes a purchase decision.

[0633] Specific example

[0634] For example, if a female user is considering purchasing a linen summer dress, she uploads a photo of her face and body measurements (e.g., height 165cm, weight 55kg, bust 85cm, waist 65cm, hips 90cm) from her device. This information is sent to and stored on the server.

[0635] Next, when a user views the product page for a linen summer dress, the server retrieves detailed information about the dress (linen material, yellow, size information S, M, L, XL) from the database. Based on the user's facial photo and body type information, as well as the product details, the server prepares a dataset to create a generated image by adding the emotional state analyzed from the user's facial expression data and behavioral data using an emotion engine.

[0636] The generated image is sent from the server to the user's terminal, and after the user confirms the notification, they check the generated image on the product page. This allows the user to check in advance whether the item suits them and whether the size is correct. Furthermore, suggestions are made according to the user's emotional state, providing a more satisfying shopping experience. Through this entire process, users can enjoy online shopping with peace of mind.

[0637] The following describes the processing flow.

[0638] Step 1:

[0639] Users input and upload their own facial photo and body information (height, weight, bust / waist / hip measurements, etc.) from their device.

[0640] Step 2:

[0641] The device sends the entered facial photo and body shape information to the server.

[0642] Step 3:

[0643] The server stores the received facial photograph and body shape information in a database and generates a user profile.

[0644] Step 4:

[0645] The user browses the product page of clothing they are considering purchasing on the e-commerce site.

[0646] Step 5:

[0647] The server retrieves detailed information (such as material, color, and size) related to the product the user is viewing from the database.

[0648] Step 6:

[0649] The device collects facial and behavioral data while the user is browsing products and sends it to the server.

[0650] Step 7:

[0651] The server invokes the emotion engine to analyze the user's emotional state based on the received facial and behavioral data. This analysis result is added to the user's profile information.

[0652] Step 8:

[0653] The server prepares a dataset for generating images based on the user's facial photograph, body type information, product details, and emotional state determined by the emotion engine.

[0654] Step 9:

[0655] The server invokes the generative AI model and inputs the prepared dataset.

[0656] Step 10:

[0657] The generative AI model generates realistic images of the user wearing the clothes selected by the user, based on the input data.

[0658] Step 11:

[0659] The server temporarily stores the generated image.

[0660] Step 12:

[0661] The server notifies the terminal that the generated image is complete.

[0662] Step 13:

[0663] The device receives a notification from the server and displays a message to the user indicating that the generated image is available for viewing.

[0664] Step 14:

[0665] The user checks the notification, navigates to the product page, and views the generated image.

[0666] Step 15:

[0667] The terminal requests an image from the server and displays the received image on the product details page.

[0668] Step 16:

[0669] Users check whether the product suits them based on the generated realistic images of them wearing it. They also consider suggestions from the server, if appropriate products are presented based on their emotional state.

[0670] Step 17:

[0671] Users decide to purchase a product if they determine that it is suitable for them.

[0672] (Example 2)

[0673] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0674] In modern online shopping, users cannot physically try on products, making it difficult to check whether an item suits them or fits before purchasing. This often leads to returns or exchanges after the product arrives due to reasons such as the item not matching expectations or being the wrong size, resulting in decreased user satisfaction. Furthermore, there is a lack of appropriate product suggestions that take into account the user's emotional state, highlighting the need for improved satisfaction.

[0675] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0676] In this invention, the server includes means for receiving a user's facial photograph and body shape information from a user terminal; means for storing the user's facial photograph and body shape information in a database; means for obtaining detailed information related to the product the user is viewing from the database; means for collecting the user's facial expression and behavior data; means for analyzing the user's emotional state based on the facial expression and behavior data; means for calling a generation AI model that creates a generated image based on the user's facial photograph, body shape information, product details, and emotional state; means for temporarily storing the image generated by the generation AI model; and means for transmitting the generated image to the user terminal. As a result, the user can not only check in advance whether the product suits them and what size it is, but also receive appropriate product suggestions according to their emotional state.

[0677] A "user terminal" is an electronic device used by a user, which is used to input the user's facial photograph and body shape information, and to communicate with the server.

[0678] A "face photograph" is image data of the user's face, providing information for the system to perform facial recognition.

[0679] "Body type information" refers to information that indicates the user's physical characteristics, such as height, weight, and bust / waist / hip measurements.

[0680] A "server" is a computing system that receives and stores data sent from user terminals, and performs data processing and analysis.

[0681] A "database" is a system that manages and stores data such as users' facial photos, body type information, and product details.

[0682] "Product details" refers to information about the product the user is viewing, including material, color, size, etc.

[0683] "Facial expression data" refers to data that shows the user's facial expressions, and is collected using sensors such as cameras.

[0684] "Behavioral data" refers to data that shows a user's online actions, such as their mouse and keyboard operations.

[0685] "Emotional analysis means" refers to a system or method for analyzing a user's facial expression data and behavioral data to identify the user's emotional state.

[0686] A "generative AI model" is an artificial intelligence model that takes various data as input and generates realistic imagery that makes it appear as if the user is wearing the product they have selected.

[0687] An "image" is a virtual image generated by a generative AI model, showing the user wearing the product they selected.

[0688] "Temporary storage" is the process of saving generated image data for a certain period of time and making it accessible as needed.

[0689] This invention provides a system that allows users to check the actual fit and size of clothing when purchasing it on an online shopping site (EC site), and also recognizes the user's emotional state to suggest appropriate products. The following describes specific embodiments of the system in detail.

[0690] System Overview

[0691] This system consists of a user terminal, a server, a generative AI model, and an emotion engine.

[0692] 1. User terminal:

[0693] A user terminal is a device used by users to input facial photos and body shape information and send it to a server. Specific examples include personal computers, smartphones, and tablets. User terminals are equipped with webcams and front cameras, enabling the collection of facial expression data and the recording of behavioral data.

[0694] 2. Server:

[0695] The server receives data sent from users and stores it in a database. Specifically, it manages data such as facial photos, body shape information, product details, and facial expression data. It also plays a role in analyzing the user's emotional state using an emotion engine and passing the results to the generating AI model.

[0696] 3. Generative AI Models:

[0697] The generative AI model generates realistic images of the user wearing the clothing selected by the user, based on the input data (user's facial photo, body type information, product details, and emotional state). The AI ​​model used includes computer vision technology and deep learning technology.

[0698] 4. Emotional Engine:

[0699] An emotion engine is software that analyzes facial expression data and behavioral data to recognize a user's emotional state. This engine identifies a user's emotions using, for example, facial expression recognition algorithms and machine learning models.

[0700] Specific example

[0701] For example, if a female user is considering purchasing a summer dress made of linen, the system will operate through the following steps.

[0702] 1. The user terminal allows female users to input their facial photograph and body measurements (e.g., height 165cm, weight 55kg, bust 85cm, waist 65cm, hips 90cm) and send this information to the server.

[0703] 2. The server stores the received facial photo and body shape information in a database and generates a user profile. At the same time, it retrieves information that the user is viewing a product page for a linen summer dress.

[0704] 3. The device collects emotional data from the user through facial expression data while the user is viewing the product page. This includes, for example, smiles or frowns while viewing product details.

[0705] 4. The server inputs the collected facial expression data into the emotion engine and obtains results that analyze the user's emotional state in real time. Based on the results, the emotional state is reflected in the dataset that is input into the generative AI model.

[0706] 5. The generation AI model generates a realistic image of the user wearing a linen summer dress, based on the user's facial photo, body type information, product details, and the results of the emotion engine's analysis. This image is temporarily stored on the server.

[0707] 6. The server notifies the user's device that the generated wearing image is complete. The user's device displays this notification to the user. The user can check the generated image on the product page and check in advance whether it suits them.

[0708] Example of a prompt

[0709] "Please generate a realistic image of a 165cm, 55kg woman trying on a yellow linen summer dress she is considering purchasing. Please also ensure the user has a happy expression in the image."

[0710] By inputting this prompt into the AI ​​model, it becomes possible to generate realistic images of the garment being worn based on specific scenarios.

[0711] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0712] Step 1:

[0713] The user accesses the e-commerce site, enters a photo of their face and body information (e.g., height, weight, bust / waist / hip measurements) on their profile page, and clicks the "Upload" button. The input data (face photo, height, weight, bust / waist / hip measurements) is collected by the device. The device sends this information to the server using an HTTP POST request. The server stores the received face photo and body information in a database and generates a user profile. As output, the user profile is saved in the database.

[0714] Step 2:

[0715] A user views a product page for clothing they are considering purchasing on an e-commerce site. When a user opens a product page, the server monitors this access event and obtains the ID of the product the user is viewing. Based on the obtained product ID, the server retrieves detailed information about the product (material, color, size, etc.) from the database. As output, the server retrieves the detailed product information and passes it to the next analysis step.

[0716] Step 3:

[0717] The device collects facial expression data in real time through the user's camera and simultaneously monitors behavioral data such as the user's mouse and keyboard operations. This facial expression and behavioral data is transmitted to a server via a secure channel. The server inputs the received facial expression and behavioral data into an emotion engine, which analyzes the emotional state in real time. The emotion engine identifies the emotional state from the user's facial expressions and actions and reflects the results in a generating AI model. As output, the user's emotional state is analyzed and passed on to the next generation step.

[0718] Step 4:

[0719] The server prepares a dataset of generated images based on the user's facial photograph, body type information, product details, and the results of the emotion engine's analysis. The server inputs this dataset into the generative AI model. The generative AI model uses the input data, along with prompts, to generate realistic images of the user wearing the selected clothing. The generated images are temporarily stored on the server. The output is the generated realistic image.

[0720] Step 5:

[0721] The server notifies the user's device that the generated wearing image is complete. This notification is sent in real-time via WebSocket or push notification. The device receives the notification from the server and displays a pop-up notification to the user. The user confirms the notification and is redirected to the product page. The device receives the worn image generated in response to the request from the server and displays it on the product details page. As output, the user can review the generated image and make a final purchase decision.

[0722] (Application Example 2)

[0723] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0724] Conventional online shopping systems have made it difficult for users to check the actual fit and size of clothing they are considering purchasing, leading to problems such as high return rates and low customer satisfaction. Furthermore, because product suggestions are made uniformly without considering the user's emotional state, it has been impossible to suggest products that are best suited to each individual user. The objective of this invention is to support users' purchasing decisions and improve the customer experience by allowing them to check realistic images of how clothes look when worn online and by providing appropriate product suggestions based on their emotional state.

[0725] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving the user's facial photograph and body shape information, means for storing this information in a database, means for obtaining detailed information related to the product the user is viewing, means for recognizing the user's facial expression, means for analyzing the user's emotional state, means for calling a generation AI model that creates a generated image based on the user's facial photograph, body shape information, product details, and the user's emotional state, and means for transmitting the generated image to the user terminal. As a result, the user can check images of clothes that suit them in real time and receive product suggestions that match their emotions.

[0726] A "user terminal" is a device that allows a user to input or receive information, and specific examples include smartphones and head-mounted displays.

[0727] A "face photo" is a digital image of the user's face.

[0728] "Body type information" refers to the user's physical data, such as height, weight, and bust / waist / hip measurements.

[0729] A "database" is a place where information is stored, managed, and made easily accessible.

[0730] "Product details" refers to information that describes the characteristics of the product the user is viewing, such as its material, color, and size.

[0731] A "generated image" is an image created by a generative AI model that shows what a user would look like wearing the product.

[0732] A "generative AI model" is an artificial intelligence model that uses machine learning or deep learning techniques to create generated images from input data.

[0733] "Emotional state" refers to the psychological and emotional state of a user, obtained by analyzing their facial expressions and behavioral data.

[0734] "Customizing product recommendations" means taking into account the user's emotional state and history to individually recommend the most suitable products.

[0735] This invention provides a system that allows users to check the actual fit and size of clothing when purchasing it on an online shopping site, and also recognizes the user's emotional state to suggest appropriate products. This system consists of a user terminal, a server, a generative AI model, and an emotion engine.

[0736] System Configuration

[0737] The server is configured using the following hardware and software:

[0738] Hardware: High-performance servers, database servers, storage

[0739] software:

[0740] Sentiment analysis engine: Amazon Rekognition, Microsoft Azure Face API

[0741] Generative AI models: DALL-E, Stable Diffusion

[0742] Databases: Amazon RDS, Firebase Firestore

[0743] User terminal

[0744] Users access the system using a smartphone or head-mounted display (HMD). The user terminal performs the following actions:

[0745] 1. Enter your facial photo and body type information:

[0746] Users take a photo of their face using their device's camera and input body information such as height, weight, and bust / waist / hip measurements.

[0747] The terminal sends this information to the server.

[0748] 2. Recognition and transmission of facial expressions:

[0749] The device's camera recognizes the user's facial expressions in real time and sends that data to the server.

[0750] server

[0751] The server performs the following actions:

[0752] 1. Information storage and analysis:

[0753] The system stores the received facial photograph and body type information in a database and generates a user profile.

[0754] Retrieve detailed information about the product the user is viewing from the database.

[0755] 2. Analysis of emotional state:

[0756] The system analyzes facial expression data to determine the user's emotional state.

[0757] 3. Creating the generated image:

[0758] Based on the user's facial photo, body type information, product details, and emotional state, a generative AI model is invoked to create generated images.

[0759] The generated images are temporarily stored on the server.

[0760] 4. Customization of notifications and product suggestions:

[0761] The generated image is sent to the user's terminal, and product suggestions are customized based on the user's emotional state.

[0762] Specific example

[0763] For example, if a user is using their smartphone to access a virtual store and is looking for a casual shirt for summer:

[0764] 1. The user takes a photo of themselves with their smartphone camera and enters their body shape information.

[0765] 2. When a user selects a casual shirt, the server retrieves the product information.

[0766] 3. While the user is browsing the product page for a shirt, the smartphone camera analyzes the user's facial expressions and determines that they are showing interest.

[0767] 4. The server displays to the user in real time an image of the product being worn, generated based on the user's facial photo, body type information, product details, and emotional state.

[0768] 5. If the user looks satisfied, the app will suggest, "Would you also like some pants or accessories to go with this shirt?"

[0769] Example of a prompt

[0770] "Enter the user's profile picture, body type information, and product details (shirt material, color, size), and based on the user's emotional state (interest level), generate a real-time image of the product being worn, which will be displayed on their smartphone."

[0771] This system allows users to see images of clothes that suit them in real time and enjoy a comfortable shopping experience while receiving product suggestions tailored to their emotions.

[0772] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0773] Step 1:

[0774] The user enters a facial photograph and body information into a smartphone or head-mounted display (HMD). The entered information includes the user's facial photograph and detailed body data (e.g., height, weight, bust / waist / hip measurements). This data is sent to a server via the device. The server stores the received data in a database and generates a user profile.

[0775] Step 2:

[0776] A user views the product page for a piece of clothing they are considering purchasing within a virtual store. The server retrieves and analyzes detailed information about the selected product (e.g., material, color, size) from the database. This information is properly configured as it will be used in subsequent processing.

[0777] Step 3:

[0778] The device uses a camera to recognize the user's facial expressions in real time. The recognized facial expression data is analyzed using an emotion analysis engine (e.g., Amazon Rekognition, Microsoft Azure Face API). The analysis results in the user's emotional state (e.g., interest, satisfaction, dissatisfaction, etc.). This emotional state data is sent to the server.

[0779] Step 4:

[0780] The server creates a dataset based on the acquired emotional state data, combining it with the user's facial photograph, body shape information, and product details. The created dataset is then input into a generative AI model (e.g., DALL-E, Stable Diffusion). Based on this input data, the generative AI model generates realistic images of the user wearing the clothes.

[0781] Step 5:

[0782] The server temporarily stores the generated image. It then sends the generated image to the user's terminal. The terminal displays the received image to the user, providing visual feedback as if the user were actually trying on the clothes.

[0783] Step 6:

[0784] The user reviews the generated image. The device then re-analyzes the user's emotional state and sends the data to the server. The server customizes product suggestions based on the latest emotional state. For example, if the user's emotional state is "satisfied," it will suggest other related products (e.g., matching pants or accessories).

[0785] Step 7:

[0786] The server sends customized product suggestions to the user's terminal. The terminal displays details of the suggested products and images of the items being worn to the user. Based on this information, the user can make a purchase decision.

[0787] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0788] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0789] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0790] [Third Embodiment]

[0791] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0792] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0793] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0794] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0795] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0796] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0797] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0798] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0799] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0800] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0801] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0802] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0803] This invention provides a system that allows users to check the actual fit and size of clothing when purchasing it on an online shopping site (EC site). Specific embodiments of the system are described in detail below.

[0804] System Overview

[0805] This system consists of a user terminal, a server, and a generative AI model. The user uploads their facial photo and body shape information from the terminal, which the server receives and stores. The server retrieves detailed information about the products the user is viewing and uses the generative AI model to generate realistic images of the clothes being worn. The generated images are then sent to the user terminal, allowing the user to view online how the clothes would look on them.

[0806] Program processing details

[0807] 1. User registration and initial setup

[0808] The user uploads a photo of their face and detailed body information (e.g., height, weight, bust / waist / hip measurements) from their device. The device sends the entered photo and body information to the server. The server stores the received photo and body information in a database and generates a user profile.

[0809] 2. Acquisition and analysis of product information

[0810] The user browses the product page of clothing they are considering purchasing on an e-commerce site. The server retrieves detailed information about the product the user is viewing (material, color, size, etc.) from the database. The server analyzes the retrieved product details and prepares them for input into a generating AI model.

[0811] 3. Image generation using AI models

[0812] The server prepares a dataset for creating generated images based on the user's facial photograph, body type information, and product details. The server invokes a generative AI model and inputs the prepared dataset. Based on the input data, the generative AI model generates realistic images of the user wearing the selected clothing. The generated images are temporarily stored by the server.

[0813] 4. User Feedback

[0814] The server notifies the device that the generated wearing image is complete. The device displays the notification received from the server to the user, who then navigates to the product page to view the generated image. The device receives the worn image generated in response to the request from the server and displays it on the product details page. The user reviews the generated realistic wearing image and determines whether the product suits them.

[0815] Specific example

[0816] For example, if a female user is considering purchasing a linen summer dress, she uploads a photo of her face and body measurements (e.g., height 165cm, weight 55kg, bust 85cm, waist 65cm, hips 90cm) from her device. This information is sent to and stored on the server.

[0817] Next, when the user views the product page for a linen summer dress, the server retrieves the dress's details (linen material, yellow, size information S, M, L, XL) from the database. Based on the user's facial photo, body type information, and product details, the server calls a generative AI model to generate an image of the user wearing the dress.

[0818] The generated image is sent from the server to the user's terminal, and after the user confirms the notification, they check the generated image on the product page. This allows the user to check in advance whether the item suits them and whether the size is correct. Through this entire process, users can enjoy online shopping with peace of mind.

[0819] The following describes the processing flow.

[0820] Step 1:

[0821] Users input and upload their own facial photo and body information (height, weight, bust / waist / hip measurements, etc.) from their device.

[0822] Step 2:

[0823] The device sends the entered facial photo and body shape information to the server.

[0824] Step 3:

[0825] The server stores the received facial photograph and body shape information in a database and generates a user profile.

[0826] Step 4:

[0827] The user browses the product page of clothing they are considering purchasing on the e-commerce site.

[0828] Step 5:

[0829] The server retrieves detailed information about the product the user is viewing (such as material, color, and size) from the database.

[0830] Step 6:

[0831] The server analyzes the detailed information of the acquired products and prepares a dataset to be input into the generating AI model.

[0832] Step 7:

[0833] The server invokes a generative AI model to create generated images based on the user's facial photo, body type information, and product details.

[0834] Step 8:

[0835] The generative AI model generates realistic images of how the user's selected clothing would look when worn, based on the input dataset.

[0836] Step 9:

[0837] The server temporarily stores the generated image.

[0838] Step 10:

[0839] The server notifies the terminal that the generated image is complete.

[0840] Step 11:

[0841] The device receives a notification from the server and displays a message to the user indicating that the generated image is available for viewing.

[0842] Step 12:

[0843] The user checks the notification, navigates to the product page, and views the generated image.

[0844] Step 13:

[0845] The terminal requests an image from the server and displays the received image on the product details page.

[0846] Step 14:

[0847] Users can check whether the product suits them based on the generated realistic images of them wearing it, and then make a purchase decision.

[0848] (Example 1)

[0849] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0850] In online shopping, users cannot actually try on products, making it difficult to confirm whether purchased clothing fits them. This leads to problems such as incorrect sizing, the product not looking as expected, or unsuitable materials, resulting in decreased customer satisfaction and increased returns.

[0851] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0852] In this invention, the server includes means for receiving a user's facial photograph and body shape information from a user terminal; means for storing the user's facial photograph and body shape information in a database; means for obtaining detailed information related to the product the user is viewing from the database; means for calling a generation AI model that creates a generated image based on the user's facial photograph, body shape information, and product details; means for temporarily storing the image generated by the generation AI model; means for transmitting the generated image to the user terminal; and means for notifying the user terminal that the generated image is complete. This enables users to have a realistic try-on experience when shopping online and to check whether the product fits them before purchasing.

[0853] A "user terminal" is a device used by a user to access a system and input information, and includes smartphones, tablets, and personal computers.

[0854] A "server" is a computer system that processes and stores information received from a user terminal, calls a generation AI model, and sends the generated image to the user terminal.

[0855] A "face photograph" is image data of a user's face captured by a camera, and includes the user's facial features.

[0856] "Body shape information" refers to data that represents the user's physical characteristics, including height, weight, bust size, waist size, and hip size.

[0857] A "database" is a data storage system that stores facial photographs, body type information, and product details, and retrieves and updates them as needed.

[0858] "Product details" refers to information about the material, color, size, etc., of the product the user is viewing.

[0859] A "generative AI model" is an artificial intelligence model that generates realistic images of a user wearing a product they have selected, based on the user's facial photograph, body shape information, and product details.

[0860] An "image" is a visual representation of a user-selected product being worn, created by a generative AI model.

[0861] A "notification" is a message sent to inform the user that the generated image is complete, and includes push notifications, in-app notifications, and other similar messages.

[0862] "Temporary storage" refers to the process where the server temporarily stores the generated image, which is then used to send it to the user's terminal later.

[0863] "Decoding" refers to the process by which a user's terminal converts an image received from a server into a format that can be displayed.

[0864] This invention provides a system that allows users to check the actual fit and size of clothing when purchasing it on an online shopping site. Specific embodiments of the system are described in detail below.

[0865] System Configuration

[0866] This system consists of user terminals, a server, and a generative AI model. User terminals are implemented using devices such as smartphones, tablets, and personal computers. The server is a computer system that centrally manages information received from user terminals and operates the generative AI model. The generative AI model uses deep learning models such as StyleGAN2, which utilizes TensorFlow or PyTorch.

[0867] Receiving and storing data

[0868] Users input their facial photo and detailed body information (e.g., height, weight, bust / waist / hip measurements) through a dedicated application or website, and send it from their device to the server. The server stores the received facial photo and body information in a database. MySQL or PostgreSQL is used for the database, and data is transferred securely using security protocols (SSL / TLS).

[0869] Product information acquisition and analysis

[0870] When a user views a product page for clothing they are considering purchasing on an e-commerce site, the server retrieves detailed information about that product (material, color, size, etc.) from a database or external API. The retrieved information is then analyzed and formatted for input into a generating AI model.

[0871] Image generation using a generative AI model

[0872] The server prepares a dataset for generating images based on the user's facial photograph, body type information, and product details. It then calls a generation AI model and inputs the prepared dataset. The generation AI model generates realistic images of the user wearing the selected product. These generated images are temporarily stored by the server.

[0873] User Feedback

[0874] The server notifies the user's device that the generated wearing image is complete. The device displays the notification received from the server to the user, and the user navigates to the product page to view the generated image. The device receives the worn image generated in response to the request from the server and displays it on the product details page. The user reviews the generated realistic wearing image and decides whether the product suits them.

[0875] Specific example

[0876] For example, if a female user is considering purchasing a linen summer dress, she would upload a photo of her face and body measurements (e.g., height 165cm, weight 55kg, bust 85cm, waist 65cm, hips 90cm) from her device. This information would then be sent to and stored on the server.

[0877] Next, when the user views the product page for a linen summer dress, the server retrieves the product details (linen material, yellow, size information S, M, L, XL) from the database. Based on the user's face photo, body type information, and product details, the server calls a generative AI model to generate an image of the user wearing the dress.

[0878] The generated image is sent from the server to the user's terminal, and after the user confirms the notification, they check the generated image on the product page. This allows the user to check in advance whether the item suits them and whether the size is correct. Through this entire process, users can enjoy online shopping with peace of mind.

[0879] Example of a prompt

[0880] "Please generate an image of the user wearing a summer dress made of linen. The user's face photo and body information are as follows: Height 165cm, Weight 55kg, Bust 85cm, Waist 65cm, Hips 90cm. The dress details are linen material, yellow, size M."

[0881] In this way, this system can significantly improve the user experience in online shopping.

[0882] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0883] Step 1:

[0884] Enter and submit user information

[0885] Input: User's profile picture, body information (e.g., height, weight, bust / waist / hip measurements)

[0886] Processing: Users enter their facial photo and body type information through a dedicated application or website.

[0887] Output: Facial photo and body shape information (packet format) sent from the user terminal to the server.

[0888] The user inputs their facial photo and body type information from their device. The device converts the input information into a packet format and sends it to the server using a security protocol (SSL / TLS).

[0889] Step 2:

[0890] User information storage

[0891] Input: User's face photo and body type information

[0892] Processing: The server saves the received user information to a database. Specifically, it uses an RDBMS such as MySQL or PostgreSQL.

[0893] Output: User's facial photo and body type information stored in the database

[0894] The server stores the received facial image and body type information in a database. Once the data is confirmed to have been saved, a user profile is generated.

[0895] Step 3:

[0896] Product information acquisition

[0897] Input: URL of the product page the user is viewing

[0898] Processing: The server parses the URL and retrieves product details (material, color, size, etc.) from the database or an external API.

[0899] Output: Retrieved product details

[0900] When a user opens a product page for clothing they are interested in on an e-commerce site, the server parses the URL of that product page and retrieves the necessary product details.

[0901] Step 4:

[0902] Product Analysis

[0903] Input: Retrieved product details

[0904] Processing: The server analyzes the product details and prepares them in a format for input into the generated AI model.

[0905] Output: Product data formatted for input into the generating AI model.

[0906] The server analyzes the detailed information of the retrieved products and converts it into a format suitable for the generated AI model. This includes information such as material, color, and size.

[0907] Step 5:

[0908] Preparing the dataset

[0909] Input: User's face photo, body type information, and formatted product data.

[0910] Processing: The server creates a dataset to be input into the generated AI model.

[0911] Output: Dataset for generative AI models

[0912] The server creates an input dataset based on the user's facial photo, body type information, and product details.

[0913] Step 6:

[0914] Calling an AI Model

[0915] Input: Dataset for generated AI models

[0916] Processing: The server calls a generative AI model (e.g., StyleGAN2) and inputs the dataset.

[0917] Output: Generated image

[0918] The server invokes a generative AI model using TensorFlow or PyTorch and inputs the prepared dataset. The model generates realistic images of the user wearing the selected product.

[0919] Step 7:

[0920] Saving the generated image

[0921] Input: Generated image

[0922] Processing: The server temporarily stores the generated image. It uses cloud storage such as Amazon S3 or Google Cloud Storage.

[0923] Output: Temporarily saved image

[0924] The generated image data is securely and quickly stored in cloud storage by the server.

[0925] Step 8:

[0926] User notifications

[0927] Input: Notification that the generated image has been saved.

[0928] Processing: The server notifies the user's device that the generated image is complete. This notification is sent using push notifications or in-app notifications.

[0929] Output: Notification sent to the user

[0930] The server notifies the user that the generated image is complete on their device. This notification is sent promptly using push notifications or in-app notifications.

[0931] Step 9:

[0932] Image acquisition

[0933] Input: Notifications confirmed by the user

[0934] Processing: The terminal sends a request to the server and downloads the generated image.

[0935] Output: Image downloaded to the device

[0936] After the user terminal confirms the notification, it sends a request to the server to securely download the image.

[0937] Step 10:

[0938] Displaying images

[0939] Input: Downloaded image

[0940] Processing: The terminal decodes the received image and displays it on the product details page.

[0941] Output: Image displayed on the product details page

[0942] The user's terminal decodes the image received from the server and displays it on the product details page. The user then uses this information to determine if the product is suitable for them.

[0943] (Application Example 1)

[0944] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0945] When purchasing items online, it's difficult for users to accurately determine how the product will actually look on them and whether it fits properly. This is especially true for clothing, where there's no opportunity to try it on, often leading to disappointment after purchase. To address this problem, there's a need for a system that allows users to check how the clothing will look and fit beforehand.

[0946] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0947] In this invention, the server includes means for receiving a user's facial photograph and body shape information from a user terminal; means for storing the user's facial photograph and body shape information in a database; means for obtaining detailed information related to the product the user is viewing from the database; means for calling a generation AI model that creates a generated image based on the user's facial photograph, body shape information, and product details; means for transmitting the image generated by the generation AI model to the user terminal; and means for temporarily storing the generated image in cloud storage. This makes it possible for a user to check in real time how the clothing will look on them and whether the size will fit them before purchasing the product.

[0948] A "user terminal" refers to a computer device operated by a user using online shopping, and includes devices such as smartphones, tablets, and personal computers.

[0949] A "face photograph" is image data of a user's face, used as identification information.

[0950] "Body shape information" refers to data representing the physical dimensions of the user's body, including information such as height, weight, and bust / waist / hip measurements.

[0951] A "database" is an information system that organizes and stores large amounts of data, and it is a place where user information and product information are stored.

[0952] "Product details" refers to all information about the product, including data such as material, color, size, and price.

[0953] A "generative AI model" is a model that uses artificial intelligence techniques, such as generative adversarial networks (GANs), to generate new images from specific input data.

[0954] A "generated image" is an image created by a generative AI model, simulating what a user would look like wearing a specific product.

[0955] "Cloud storage" is a remote storage service for saving and managing data over a network, and is used to store generated images and other user data.

[0956] A "notification" is the process of transmitting specific information to a user's device, including a message informing them that a new generated image is ready.

[0957] "Display" refers to the act of visually presenting information on a user's terminal, and is a means of showing the generated image to the user.

[0958] Modes for carrying out the invention

[0959] This invention provides a system that allows users to check the actual fit and size of clothing when purchasing it online. This system primarily consists of a user terminal, a server, and a generating AI model. Specific embodiments are described below.

[0960] System Overview

[0961] 1. User registration and initial setup

[0962] First, the user uploads their facial photo and body shape information from their device. The user's device receives and processes this information and sends it to the server. The server stores the received facial photo and body shape information in a database and generates a user profile.

[0963] 2. Acquisition and analysis of product information

[0964] When a user views a product page on an e-commerce site that they are considering purchasing, the server retrieves detailed information about that product (material, color, size, etc.) from its database. The server then analyzes the retrieved product information and prepares it for input into a generating AI model.

[0965] 3. Image generation using AI models

[0966] The server prepares a dataset for generating images based on the user's facial photo, body type information, and product details. Next, the server invokes a generative AI model and inputs the pre-prepared dataset. Based on this data, the generative AI model generates realistic images of the user wearing the selected clothing. The generated images are temporarily stored in cloud storage by the server.

[0967] 4. User Feedback

[0968] The server notifies the user's device when the generated wearing image is complete. The user's device displays the notification received from the server, and the user can then go to the product page to check the generated image. This allows the user to check in advance whether the item suits them and whether the size is correct, supporting their purchase decision.

[0969] Technologies and hardware / software used

[0970] Smartphone app: Application for iOS or Android

[0971] Server: Cloud servers such as AWS

[0972] Generative AI Model: A custom model based on GAN (Generative Adversarial Network).

[0973] Cloud storage: Cloud storage services such as AWS S3

[0974] Specific example

[0975] For example, consider a case where user A wants to purchase a summer dress made of linen. First, A registers with the app and uploads a photo of their face and body measurements. This information is sent to and stored on the server. Next, when A views the product page for the linen summer dress on the e-commerce site, the server retrieves the detailed information about the dress and uses a generative AI model to generate an image of A wearing the dress. The generated image is temporarily stored in cloud storage, and A is notified when the generation is complete. A receives this notification in the app and checks the generated image of them wearing the dress. This allows A to check the size and fit before purchasing, enabling them to proceed with the purchase with confidence.

[0976] Examples of prompt statements

[0977] User A is browsing the product page for a linen summer dress. The server retrieves the dress's material and size information, and a generation AI model generates a model of User A wearing the dress based on their face photo and body shape information. The app notifies User A of the generated image, and User A confirms the image.

[0978] Thus, the present invention provides a system that can improve the user's online shopping experience and enhance the accuracy of their purchase decisions.

[0979] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0980] Step 1:

[0981] User registration and initial setup

[0982] Users input and upload their facial photo and body measurements using a smartphone app. The user's device sends this information to the server. The server stores the received facial photo and body measurements in a database and generates a user profile. This profile includes detailed information such as the user's name, facial photo, height, weight, and bust / waist / hip measurements.

[0983] Input: User's profile picture, body information (height, weight, bust / waist / hip measurements, etc.)

[0984] Output: User profile is saved to the database.

[0985] Step 2:

[0986] Product information acquisition and analysis

[0987] When a user views a product page on an e-commerce site, their device sends a request for that page to the server. The server retrieves detailed product information from its database based on the product ID. This information includes the product's material, color, and size. The server analyzes this information and prepares a dataset for input into a generating AI model.

[0988] Input: Product ID

[0989] Output: Product details (material, color, size, etc.)

[0990] Step 3:

[0991] Image generation using AI models

[0992] The server inputs the prepared dataset (user's facial photo, body type information, and product details) into the generating AI model. Based on the input data, the generating AI model creates realistic images of the user wearing the selected clothing. The generated images are temporarily stored in cloud storage by the server.

[0993] Input: Dataset (user's profile picture, body type information, product details)

[0994] Output: Generated image

[0995] Step 4:

[0996] User Feedback

[0997] The server notifies the user's device that the generation of the generated wearing image is complete. The user's device receives this notification and displays it to the user. The user opens the app and checks the generated image. Based on this image, the user decides whether the product suits them and whether to purchase it.

[0998] Input: Generated image, user terminal

[0999] Output: Display of notifications and images.

[1000] Specifically, if a user selects a linen summer dress, the server retrieves the product's material and size information, and a generation AI model generates an image based on the user's facial photo and body shape information. The generated image is then notified to the user, who then makes a purchase decision based on the image. This smooth process allows users to enjoy online shopping with peace of mind.

[1001] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[1002] This invention provides a system that allows users to check the actual fit and size of clothing when purchasing it on an online shopping site (EC site), and also recognizes the user's emotional state to suggest appropriate products. Specific embodiments of the system are described in detail below.

[1003] System Overview

[1004] This system consists of a user terminal, a server, a generative AI model, and an emotion engine. The user uploads their facial photo and body shape information from the terminal, which the server receives and stores. The server retrieves detailed information about the products the user is viewing and uses the generative AI model to generate realistic images of the clothes being worn. Furthermore, the emotion engine analyzes the user's facial expressions and behavioral data, recognizes the user's emotional state, adjusts the images generated by the generative AI model based on this, and notifies the user terminal. This allows the user to receive appropriate product suggestions and view images of themselves wearing clothes online.

[1005] Program processing details

[1006] 1. User registration and initial setup

[1007] The user inputs and uploads their facial photo and detailed body information (e.g., height, weight, bust / waist / hip measurements) from their device. The device sends the entered facial photo and body information to the server. The server stores the received facial photo and body information in a database and generates a user profile.

[1008] 2. Acquisition and analysis of product information

[1009] The user browses the product page of clothing they are considering purchasing on an e-commerce site. The server retrieves detailed information about the product the user is viewing (material, color, size, etc.) from the database. The server analyzes the retrieved product details and prepares a dataset to be input into a generative AI model.

[1010] 3. Analysis using an emotion engine

[1011] The terminal collects user facial expressions and behavioral data and sends it to the server. The server invokes an emotion engine and analyzes the user's emotional state based on the data. The emotional state recognized by the emotion engine is reflected in the dataset input to the generative AI model.

[1012] 4. Image generation using AI models

[1013] The server prepares a dataset for creating generated images based on the user's facial photograph, body type information, and product details. This also includes emotional states analyzed by the emotion engine. The server invokes a generative AI model and inputs the prepared dataset. Based on the input data, the generative AI model generates realistic images of the user wearing the clothing selected by the user. The generated images are temporarily stored by the server.

[1014] 5. User Feedback

[1015] The server notifies the device that the generated wearing image is complete. The device displays the notification received from the server to the user, who then navigates to the product page to view the generated image. The device receives the worn image generated in response to the request from the server and displays it on the product details page. The user checks whether the product suits them based on the generated realistic wearing image and makes a purchase decision.

[1016] Specific example

[1017] For example, if a female user is considering purchasing a linen summer dress, she uploads a photo of her face and body measurements (e.g., height 165cm, weight 55kg, bust 85cm, waist 65cm, hips 90cm) from her device. This information is sent to and stored on the server.

[1018] Next, when a user views the product page for a linen summer dress, the server retrieves detailed information about the dress (linen material, yellow, size information S, M, L, XL) from the database. Based on the user's facial photo and body type information, as well as the product details, the server prepares a dataset to create a generated image by adding the emotional state analyzed from the user's facial expression data and behavioral data using an emotion engine.

[1019] The generated image is sent from the server to the user's terminal, and after the user confirms the notification, they check the generated image on the product page. This allows the user to check in advance whether the item suits them and whether the size is correct. Furthermore, suggestions are made according to the user's emotional state, providing a more satisfying shopping experience. Through this entire process, users can enjoy online shopping with peace of mind.

[1020] The following describes the processing flow.

[1021] Step 1:

[1022] Users input and upload their own facial photo and body information (height, weight, bust / waist / hip measurements, etc.) from their device.

[1023] Step 2:

[1024] The device sends the entered facial photo and body shape information to the server.

[1025] Step 3:

[1026] The server stores the received facial photograph and body shape information in a database and generates a user profile.

[1027] Step 4:

[1028] The user browses the product page of clothing they are considering purchasing on the e-commerce site.

[1029] Step 5:

[1030] The server retrieves detailed information (such as material, color, and size) related to the product the user is viewing from the database.

[1031] Step 6:

[1032] The device collects facial and behavioral data while the user is browsing products and sends it to the server.

[1033] Step 7:

[1034] The server invokes the emotion engine to analyze the user's emotional state based on the received facial and behavioral data. This analysis result is added to the user's profile information.

[1035] Step 8:

[1036] The server prepares a dataset for generating images based on the user's facial photograph, body type information, product details, and emotional state determined by the emotion engine.

[1037] Step 9:

[1038] The server invokes the generative AI model and inputs the prepared dataset.

[1039] Step 10:

[1040] The generative AI model generates realistic images of the user wearing the clothes selected by the user, based on the input data.

[1041] Step 11:

[1042] The server temporarily stores the generated image.

[1043] Step 12:

[1044] The server notifies the terminal that the generated image is complete.

[1045] Step 13:

[1046] The device receives a notification from the server and displays a message to the user indicating that the generated image is available for viewing.

[1047] Step 14:

[1048] The user checks the notification, navigates to the product page, and views the generated image.

[1049] Step 15:

[1050] The terminal requests an image from the server and displays the received image on the product details page.

[1051] Step 16:

[1052] Users check whether the product suits them based on the generated realistic images of them wearing it. They also consider suggestions from the server, if appropriate products are presented based on their emotional state.

[1053] Step 17:

[1054] Users decide to purchase a product if they determine that it is suitable for them.

[1055] (Example 2)

[1056] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1057] In modern online shopping, users cannot physically try on products, making it difficult to check whether an item suits them or fits before purchasing. This often leads to returns or exchanges after the product arrives due to reasons such as the item not matching expectations or being the wrong size, resulting in decreased user satisfaction. Furthermore, there is a lack of appropriate product suggestions that take into account the user's emotional state, highlighting the need for improved satisfaction.

[1058] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[1059] In this invention, the server includes means for receiving a user's facial photograph and body shape information from a user terminal; means for storing the user's facial photograph and body shape information in a database; means for obtaining detailed information related to the product the user is viewing from the database; means for collecting the user's facial expression and behavior data; means for analyzing the user's emotional state based on the facial expression and behavior data; means for calling a generation AI model that creates a generated image based on the user's facial photograph, body shape information, product details, and emotional state; means for temporarily storing the image generated by the generation AI model; and means for transmitting the generated image to the user terminal. As a result, the user can not only check in advance whether the product suits them and what size it is, but also receive appropriate product suggestions according to their emotional state.

[1060] A "user terminal" is an electronic device used by a user, which is used to input the user's facial photograph and body shape information, and to communicate with the server.

[1061] A "face photograph" is image data of the user's face, providing information for the system to perform facial recognition.

[1062] "Body type information" refers to information that indicates the user's physical characteristics, such as height, weight, and bust / waist / hip measurements.

[1063] A "server" is a computing system that receives and stores data sent from user terminals, and performs data processing and analysis.

[1064] A "database" is a system that manages and stores data such as users' facial photos, body type information, and product details.

[1065] "Product details" refers to information about the product the user is viewing, including material, color, size, etc.

[1066] "Facial expression data" refers to data that shows the user's facial expressions, and is collected using sensors such as cameras.

[1067] "Behavioral data" refers to data that shows a user's online actions, such as their mouse and keyboard operations.

[1068] "Emotional analysis means" refers to a system or method for analyzing a user's facial expression data and behavioral data to identify the user's emotional state.

[1069] A "generative AI model" is an artificial intelligence model that takes various data as input and generates realistic imagery that makes it appear as if the user is wearing the product they have selected.

[1070] An "image" is a virtual image generated by a generative AI model, showing the user wearing the product they selected.

[1071] "Temporary storage" is the process of saving generated image data for a certain period of time and making it accessible as needed.

[1072] This invention provides a system that allows users to check the actual fit and size of clothing when purchasing it on an online shopping site (EC site), and also recognizes the user's emotional state to suggest appropriate products. The following describes specific embodiments of the system in detail.

[1073] System Overview

[1074] This system consists of a user terminal, a server, a generative AI model, and an emotion engine.

[1075] 1. User terminal:

[1076] A user terminal is a device used by users to input facial photos and body shape information and send it to a server. Specific examples include personal computers, smartphones, and tablets. User terminals are equipped with webcams and front cameras, enabling the collection of facial expression data and the recording of behavioral data.

[1077] 2. Server:

[1078] The server receives data sent from users and stores it in a database. Specifically, it manages data such as facial photos, body shape information, product details, and facial expression data. It also plays a role in analyzing the user's emotional state using an emotion engine and passing the results to the generating AI model.

[1079] 3. Generative AI Models:

[1080] The generative AI model generates realistic images of the user wearing the clothing selected by the user, based on the input data (user's facial photo, body type information, product details, and emotional state). The AI ​​model used includes computer vision technology and deep learning technology.

[1081] 4. Emotional Engine:

[1082] An emotion engine is software that analyzes facial expression data and behavioral data to recognize a user's emotional state. This engine identifies a user's emotions using, for example, facial expression recognition algorithms and machine learning models.

[1083] Specific example

[1084] For example, if a female user is considering purchasing a summer dress made of linen, the system will operate through the following steps.

[1085] 1. The user terminal allows female users to input their facial photograph and body measurements (e.g., height 165cm, weight 55kg, bust 85cm, waist 65cm, hips 90cm) and send this information to the server.

[1086] 2. The server stores the received facial photo and body shape information in a database and generates a user profile. At the same time, it retrieves information that the user is viewing a product page for a linen summer dress.

[1087] 3. The device collects emotional data from the user through facial expression data while the user is viewing the product page. This includes, for example, smiles or frowns while viewing product details.

[1088] 4. The server inputs the collected facial expression data into the emotion engine and obtains results that analyze the user's emotional state in real time. Based on the results, the emotional state is reflected in the dataset that is input into the generative AI model.

[1089] 5. The generation AI model generates a realistic image of the user wearing a linen summer dress, based on the user's facial photo, body type information, product details, and the results of the emotion engine's analysis. This image is temporarily stored on the server.

[1090] 6. The server notifies the user's device that the generated wearing image is complete. The user's device displays this notification to the user. The user can check the generated image on the product page and check in advance whether it suits them.

[1091] Example of a prompt

[1092] "Please generate a realistic image of a 165cm, 55kg woman trying on a yellow linen summer dress she is considering purchasing. Please also ensure the user has a happy expression in the image."

[1093] By inputting this prompt into the AI ​​model, it becomes possible to generate realistic images of the garment being worn based on specific scenarios.

[1094] The flow of the specific processing in Example 2 will be explained using Figure 13.

[1095] Step 1:

[1096] The user accesses the e-commerce site, enters a photo of their face and body information (e.g., height, weight, bust / waist / hip measurements) on their profile page, and clicks the "Upload" button. The input data (face photo, height, weight, bust / waist / hip measurements) is collected by the device. The device sends this information to the server using an HTTP POST request. The server stores the received face photo and body information in a database and generates a user profile. As output, the user profile is saved in the database.

[1097] Step 2:

[1098] A user views a product page for clothing they are considering purchasing on an e-commerce site. When a user opens a product page, the server monitors this access event and obtains the ID of the product the user is viewing. Based on the obtained product ID, the server retrieves detailed information about the product (material, color, size, etc.) from the database. As output, the server retrieves the detailed product information and passes it to the next analysis step.

[1099] Step 3:

[1100] The device collects facial expression data in real time through the user's camera and simultaneously monitors behavioral data such as the user's mouse and keyboard operations. This facial expression and behavioral data is transmitted to a server via a secure channel. The server inputs the received facial expression and behavioral data into an emotion engine, which analyzes the emotional state in real time. The emotion engine identifies the emotional state from the user's facial expressions and actions and reflects the results in a generating AI model. As output, the user's emotional state is analyzed and passed on to the next generation step.

[1101] Step 4:

[1102] The server prepares a dataset of generated images based on the user's facial photograph, body type information, product details, and the results of the emotion engine's analysis. The server inputs this dataset into the generative AI model. The generative AI model uses the input data, along with prompts, to generate realistic images of the user wearing the selected clothing. The generated images are temporarily stored on the server. The output is the generated realistic image.

[1103] Step 5:

[1104] The server notifies the user's device that the generated wearing image is complete. This notification is sent in real-time via WebSocket or push notification. The device receives the notification from the server and displays a pop-up notification to the user. The user confirms the notification and is redirected to the product page. The device receives the worn image generated in response to the request from the server and displays it on the product details page. As output, the user can review the generated image and make a final purchase decision.

[1105] (Application Example 2)

[1106] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1107] Conventional online shopping systems have made it difficult for users to check the actual fit and size of clothing they are considering purchasing, leading to problems such as high return rates and low customer satisfaction. Furthermore, because product suggestions are made uniformly without considering the user's emotional state, it has been impossible to suggest products that are best suited to each individual user. The objective of this invention is to support users' purchasing decisions and improve the customer experience by allowing them to check realistic images of how clothes look when worn online and by providing appropriate product suggestions based on their emotional state.

[1108] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving the user's facial photograph and body shape information, means for storing this information in a database, means for obtaining detailed information related to the product the user is viewing, means for recognizing the user's facial expression, means for analyzing the user's emotional state, means for calling a generation AI model that creates a generated image based on the user's facial photograph, body shape information, product details, and the user's emotional state, and means for transmitting the generated image to the user terminal. As a result, the user can check images of clothes that suit them in real time and receive product suggestions that match their emotions.

[1109] A "user terminal" is a device that allows a user to input or receive information, and specific examples include smartphones and head-mounted displays.

[1110] A "face photo" is a digital image of the user's face.

[1111] "Body type information" refers to the user's physical data, such as height, weight, and bust / waist / hip measurements.

[1112] A "database" is a place where information is stored, managed, and made easily accessible.

[1113] "Product details" refers to information that describes the characteristics of the product the user is viewing, such as its material, color, and size.

[1114] A "generated image" is an image created by a generative AI model that shows what a user would look like wearing the product.

[1115] A "generative AI model" is an artificial intelligence model that uses machine learning or deep learning techniques to create generated images from input data.

[1116] "Emotional state" refers to the psychological and emotional state of a user, obtained by analyzing their facial expressions and behavioral data.

[1117] "Customizing product recommendations" means taking into account the user's emotional state and history to individually recommend the most suitable products.

[1118] This invention provides a system that allows users to check the actual fit and size of clothing when purchasing it on an online shopping site, and also recognizes the user's emotional state to suggest appropriate products. This system consists of a user terminal, a server, a generative AI model, and an emotion engine.

[1119] System Configuration

[1120] The server is configured using the following hardware and software:

[1121] Hardware: High-performance servers, database servers, storage

[1122] software:

[1123] Sentiment analysis engine: Amazon Rekognition, Microsoft Azure Face API

[1124] Generative AI models: DALL-E, Stable Diffusion

[1125] Databases: Amazon RDS, Firebase Firestore

[1126] User terminal

[1127] Users access the system using a smartphone or head-mounted display (HMD). The user terminal performs the following actions:

[1128] 1. Enter your facial photo and body type information:

[1129] Users take a photo of their face using their device's camera and input body information such as height, weight, and bust / waist / hip measurements.

[1130] The terminal sends this information to the server.

[1131] 2. Recognition and transmission of facial expressions:

[1132] The device's camera recognizes the user's facial expressions in real time and sends that data to the server.

[1133] server

[1134] The server performs the following actions:

[1135] 1. Information storage and analysis:

[1136] The system stores the received facial photograph and body type information in a database and generates a user profile.

[1137] Retrieve detailed information about the product the user is viewing from the database.

[1138] 2. Analysis of emotional state:

[1139] The system analyzes facial expression data to determine the user's emotional state.

[1140] 3. Creating the generated image:

[1141] Based on the user's facial photo, body type information, product details, and emotional state, a generative AI model is invoked to create generated images.

[1142] The generated images are temporarily stored on the server.

[1143] 4. Customization of notifications and product suggestions:

[1144] The generated image is sent to the user's terminal, and product suggestions are customized based on the user's emotional state.

[1145] Specific example

[1146] For example, if a user is using their smartphone to access a virtual store and is looking for a casual shirt for summer:

[1147] 1. The user takes a photo of themselves with their smartphone camera and enters their body shape information.

[1148] 2. When a user selects a casual shirt, the server retrieves the product information.

[1149] 3. While the user is browsing the product page for a shirt, the smartphone camera analyzes the user's facial expressions and determines that they are showing interest.

[1150] 4. The server displays to the user in real time an image of the product being worn, generated based on the user's facial photo, body type information, product details, and emotional state.

[1151] 5. If the user looks satisfied, the app will suggest, "Would you also like some pants or accessories to go with this shirt?"

[1152] Example of a prompt

[1153] "Enter the user's profile picture, body type information, and product details (shirt material, color, size), and based on the user's emotional state (interest level), generate a real-time image of the product being worn, which will be displayed on their smartphone."

[1154] This system allows users to see images of clothes that suit them in real time and enjoy a comfortable shopping experience while receiving product suggestions tailored to their emotions.

[1155] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[1156] Step 1:

[1157] The user enters a facial photograph and body information into a smartphone or head-mounted display (HMD). The entered information includes the user's facial photograph and detailed body data (e.g., height, weight, bust / waist / hip measurements). This data is sent to a server via the device. The server stores the received data in a database and generates a user profile.

[1158] Step 2:

[1159] A user views the product page for a piece of clothing they are considering purchasing within a virtual store. The server retrieves and analyzes detailed information about the selected product (e.g., material, color, size) from the database. This information is properly configured as it will be used in subsequent processing.

[1160] Step 3:

[1161] The device uses a camera to recognize the user's facial expressions in real time. The recognized facial expression data is analyzed using an emotion analysis engine (e.g., Amazon Rekognition, Microsoft Azure Face API). The analysis results in the user's emotional state (e.g., interest, satisfaction, dissatisfaction, etc.). This emotional state data is sent to the server.

[1162] Step 4:

[1163] The server creates a dataset based on the acquired emotional state data, combining it with the user's facial photograph, body shape information, and product details. The created dataset is then input into a generative AI model (e.g., DALL-E, Stable Diffusion). Based on this input data, the generative AI model generates realistic images of the user wearing the clothes.

[1164] Step 5:

[1165] The server temporarily stores the generated image. It then sends the generated image to the user's terminal. The terminal displays the received image to the user, providing visual feedback as if the user were actually trying on the clothes.

[1166] Step 6:

[1167] The user reviews the generated image. The device then re-analyzes the user's emotional state and sends the data to the server. The server customizes product suggestions based on the latest emotional state. For example, if the user's emotional state is "satisfied," it will suggest other related products (e.g., matching pants or accessories).

[1168] Step 7:

[1169] The server sends customized product suggestions to the user's terminal. The terminal displays details of the suggested products and images of the items being worn to the user. Based on this information, the user can make a purchase decision.

[1170] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[1171] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1172] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[1173] [Fourth Embodiment]

[1174] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[1175] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1176] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1177] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[1178] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[1179] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[1180] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[1181] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[1182] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[1183] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1184] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1185] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[1186] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1187] This invention provides a system that allows users to check the actual fit and size of clothing when purchasing it on an online shopping site (EC site). Specific embodiments of the system are described in detail below.

[1188] System Overview

[1189] This system consists of a user terminal, a server, and a generative AI model. The user uploads their facial photo and body shape information from the terminal, which the server receives and stores. The server retrieves detailed information about the products the user is viewing and uses the generative AI model to generate realistic images of the clothes being worn. The generated images are then sent to the user terminal, allowing the user to view online how the clothes would look on them.

[1190] Program processing details

[1191] 1. User registration and initial setup

[1192] The user uploads a photo of their face and detailed body information (e.g., height, weight, bust / waist / hip measurements) from their device. The device sends the entered photo and body information to the server. The server stores the received photo and body information in a database and generates a user profile.

[1193] 2. Acquisition and analysis of product information

[1194] The user browses the product page of clothing they are considering purchasing on an e-commerce site. The server retrieves detailed information about the product the user is viewing (material, color, size, etc.) from the database. The server analyzes the retrieved product details and prepares them for input into a generating AI model.

[1195] 3. Image generation using AI models

[1196] The server prepares a dataset for creating generated images based on the user's facial photograph, body type information, and product details. The server invokes a generative AI model and inputs the prepared dataset. Based on the input data, the generative AI model generates realistic images of the user wearing the selected clothing. The generated images are temporarily stored by the server.

[1197] 4. User Feedback

[1198] The server notifies the device that the generated wearing image is complete. The device displays the notification received from the server to the user, who then navigates to the product page to view the generated image. The device receives the worn image generated in response to the request from the server and displays it on the product details page. The user reviews the generated realistic wearing image and determines whether the product suits them.

[1199] Specific example

[1200] For example, if a female user is considering purchasing a linen summer dress, she uploads a photo of her face and body measurements (e.g., height 165cm, weight 55kg, bust 85cm, waist 65cm, hips 90cm) from her device. This information is sent to and stored on the server.

[1201] Next, when the user views the product page for a linen summer dress, the server retrieves the dress's details (linen material, yellow, size information S, M, L, XL) from the database. Based on the user's facial photo, body type information, and product details, the server calls a generative AI model to generate an image of the user wearing the dress.

[1202] The generated image is sent from the server to the user's terminal, and after the user confirms the notification, they check the generated image on the product page. This allows the user to check in advance whether the item suits them and whether the size is correct. Through this entire process, users can enjoy online shopping with peace of mind.

[1203] The following describes the processing flow.

[1204] Step 1:

[1205] Users input and upload their own facial photo and body information (height, weight, bust / waist / hip measurements, etc.) from their device.

[1206] Step 2:

[1207] The device sends the entered facial photo and body shape information to the server.

[1208] Step 3:

[1209] The server stores the received facial photograph and body shape information in a database and generates a user profile.

[1210] Step 4:

[1211] The user browses the product page of clothing they are considering purchasing on the e-commerce site.

[1212] Step 5:

[1213] The server retrieves detailed information about the product the user is viewing (such as material, color, and size) from the database.

[1214] Step 6:

[1215] The server analyzes the detailed information of the acquired products and prepares a dataset to be input into the generating AI model.

[1216] Step 7:

[1217] The server invokes a generative AI model to create generated images based on the user's facial photo, body type information, and product details.

[1218] Step 8:

[1219] The generative AI model generates realistic images of how the user's selected clothing would look when worn, based on the input dataset.

[1220] Step 9:

[1221] The server temporarily stores the generated image.

[1222] Step 10:

[1223] The server notifies the terminal that the generated image is complete.

[1224] Step 11:

[1225] The device receives a notification from the server and displays a message to the user indicating that the generated image is available for viewing.

[1226] Step 12:

[1227] The user checks the notification, navigates to the product page, and views the generated image.

[1228] Step 13:

[1229] The terminal requests an image from the server and displays the received image on the product details page.

[1230] Step 14:

[1231] Users can check whether the product suits them based on the generated realistic images of them wearing it, and then make a purchase decision.

[1232] (Example 1)

[1233] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1234] In online shopping, users cannot actually try on products, making it difficult to confirm whether purchased clothing fits them. This leads to problems such as incorrect sizing, the product not looking as expected, or unsuitable materials, resulting in decreased customer satisfaction and increased returns.

[1235] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[1236] In this invention, the server includes means for receiving a user's facial photograph and body shape information from a user terminal; means for storing the user's facial photograph and body shape information in a database; means for obtaining detailed information related to the product the user is viewing from the database; means for calling a generation AI model that creates a generated image based on the user's facial photograph, body shape information, and product details; means for temporarily storing the image generated by the generation AI model; means for transmitting the generated image to the user terminal; and means for notifying the user terminal that the generated image is complete. This enables users to have a realistic try-on experience when shopping online and to check whether the product fits them before purchasing.

[1237] A "user terminal" is a device used by a user to access a system and input information, and includes smartphones, tablets, and personal computers.

[1238] A "server" is a computer system that processes and stores information received from a user terminal, calls a generation AI model, and sends the generated image to the user terminal.

[1239] A "face photograph" is image data of a user's face captured by a camera, and includes the user's facial features.

[1240] "Body shape information" refers to data that represents the user's physical characteristics, including height, weight, bust size, waist size, and hip size.

[1241] A "database" is a data storage system that stores facial photographs, body type information, and product details, and retrieves and updates them as needed.

[1242] "Product details" refers to information about the material, color, size, etc., of the product the user is viewing.

[1243] A "generative AI model" is an artificial intelligence model that generates realistic images of a user wearing a product they have selected, based on the user's facial photograph, body shape information, and product details.

[1244] An "image" is a visual representation of a user-selected product being worn, created by a generative AI model.

[1245] A "notification" is a message sent to inform the user that the generated image is complete, and includes push notifications, in-app notifications, and other similar messages.

[1246] "Temporary storage" refers to the process where the server temporarily stores the generated image, which is then used to send it to the user's terminal later.

[1247] "Decoding" refers to the process by which a user's terminal converts an image received from a server into a format that can be displayed.

[1248] This invention provides a system that allows users to check the actual fit and size of clothing when purchasing it on an online shopping site. Specific embodiments of the system are described in detail below.

[1249] System Configuration

[1250] This system consists of user terminals, a server, and a generative AI model. User terminals are implemented using devices such as smartphones, tablets, and personal computers. The server is a computer system that centrally manages information received from user terminals and operates the generative AI model. The generative AI model uses deep learning models such as StyleGAN2, which utilizes TensorFlow or PyTorch.

[1251] Receiving and storing data

[1252] Users input their facial photo and detailed body information (e.g., height, weight, bust / waist / hip measurements) through a dedicated application or website, and send it from their device to the server. The server stores the received facial photo and body information in a database. MySQL or PostgreSQL is used for the database, and data is transferred securely using security protocols (SSL / TLS).

[1253] Product information acquisition and analysis

[1254] When a user views a product page for clothing they are considering purchasing on an e-commerce site, the server retrieves detailed information about that product (material, color, size, etc.) from a database or external API. The retrieved information is then analyzed and formatted for input into a generating AI model.

[1255] Image generation using a generative AI model

[1256] The server prepares a dataset for generating images based on the user's facial photograph, body type information, and product details. It then calls a generation AI model and inputs the prepared dataset. The generation AI model generates realistic images of the user wearing the selected product. These generated images are temporarily stored by the server.

[1257] User Feedback

[1258] The server notifies the user's device that the generated wearing image is complete. The device displays the notification received from the server to the user, and the user navigates to the product page to view the generated image. The device receives the worn image generated in response to the request from the server and displays it on the product details page. The user reviews the generated realistic wearing image and decides whether the product suits them.

[1259] Specific example

[1260] For example, if a female user is considering purchasing a linen summer dress, she would upload a photo of her face and body measurements (e.g., height 165cm, weight 55kg, bust 85cm, waist 65cm, hips 90cm) from her device. This information would then be sent to and stored on the server.

[1261] Next, when the user views the product page for a linen summer dress, the server retrieves the product details (linen material, yellow, size information S, M, L, XL) from the database. Based on the user's face photo, body type information, and product details, the server calls a generative AI model to generate an image of the user wearing the dress.

[1262] The generated image is sent from the server to the user's terminal, and after the user confirms the notification, they check the generated image on the product page. This allows the user to check in advance whether the item suits them and whether the size is correct. Through this entire process, users can enjoy online shopping with peace of mind.

[1263] Example of a prompt

[1264] "Please generate an image of the user wearing a summer dress made of linen. The user's face photo and body information are as follows: Height 165cm, Weight 55kg, Bust 85cm, Waist 65cm, Hips 90cm. The dress details are linen material, yellow, size M."

[1265] In this way, this system can significantly improve the user experience in online shopping.

[1266] The flow of the specific processing in Example 1 will be explained using Figure 11.

[1267] Step 1:

[1268] Enter and submit user information

[1269] Input: User's profile picture, body information (e.g., height, weight, bust / waist / hip measurements)

[1270] Processing: Users enter their facial photo and body type information through a dedicated application or website.

[1271] Output: Facial photo and body shape information (packet format) sent from the user terminal to the server.

[1272] The user inputs their facial photo and body type information from their device. The device converts the input information into a packet format and sends it to the server using a security protocol (SSL / TLS).

[1273] Step 2:

[1274] User information storage

[1275] Input: User's face photo and body type information

[1276] Processing: The server saves the received user information to a database. Specifically, it uses an RDBMS such as MySQL or PostgreSQL.

[1277] Output: User's facial photo and body type information stored in the database

[1278] The server stores the received facial image and body type information in a database. Once the data is confirmed to have been saved, a user profile is generated.

[1279] Step 3:

[1280] Product information acquisition

[1281] Input: URL of the product page the user is viewing

[1282] Processing: The server parses the URL and retrieves product details (material, color, size, etc.) from the database or an external API.

[1283] Output: Retrieved product details

[1284] When a user opens a product page for clothing they are interested in on an e-commerce site, the server parses the URL of that product page and retrieves the necessary product details.

[1285] Step 4:

[1286] Product Analysis

[1287] Input: Retrieved product details

[1288] Processing: The server analyzes the product details and prepares them in a format for input into the generated AI model.

[1289] Output: Product data formatted for input into the generating AI model.

[1290] The server analyzes the detailed information of the retrieved products and converts it into a format suitable for the generated AI model. This includes information such as material, color, and size.

[1291] Step 5:

[1292] Preparing the dataset

[1293] Input: User's face photo, body type information, and formatted product data.

[1294] Processing: The server creates a dataset to be input into the generated AI model.

[1295] Output: Dataset for generative AI models

[1296] The server creates an input dataset based on the user's facial photo, body type information, and product details.

[1297] Step 6:

[1298] Calling an AI Model

[1299] Input: Dataset for generated AI models

[1300] Processing: The server calls a generative AI model (e.g., StyleGAN2) and inputs the dataset.

[1301] Output: Generated image

[1302] The server invokes a generative AI model using TensorFlow or PyTorch and inputs the prepared dataset. The model generates realistic images of the user wearing the selected product.

[1303] Step 7:

[1304] Saving the generated image

[1305] Input: Generated image

[1306] Processing: The server temporarily stores the generated image. It uses cloud storage such as Amazon S3 or Google Cloud Storage.

[1307] Output: Temporarily saved image

[1308] The generated image data is securely and quickly stored in cloud storage by the server.

[1309] Step 8:

[1310] User notifications

[1311] Input: Notification that the generated image has been saved.

[1312] Processing: The server notifies the user's device that the generated image is complete. This notification is sent using push notifications or in-app notifications.

[1313] Output: Notification sent to the user

[1314] The server notifies the user that the generated image is complete on their device. This notification is sent promptly using push notifications or in-app notifications.

[1315] Step 9:

[1316] Image acquisition

[1317] Input: Notifications confirmed by the user

[1318] Processing: The terminal sends a request to the server and downloads the generated image.

[1319] Output: Image downloaded to the device

[1320] After the user terminal confirms the notification, it sends a request to the server to securely download the image.

[1321] Step 10:

[1322] Displaying images

[1323] Input: Downloaded image

[1324] Processing: The terminal decodes the received image and displays it on the product details page.

[1325] Output: Image displayed on the product details page

[1326] The user's terminal decodes the image received from the server and displays it on the product details page. The user then uses this information to determine if the product is suitable for them.

[1327] (Application Example 1)

[1328] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1329] When purchasing items online, it's difficult for users to accurately determine how the product will actually look on them and whether it fits properly. This is especially true for clothing, where there's no opportunity to try it on, often leading to disappointment after purchase. To address this problem, there's a need for a system that allows users to check how the clothing will look and fit beforehand.

[1330] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[1331] In this invention, the server includes means for receiving a user's facial photograph and body shape information from a user terminal; means for storing the user's facial photograph and body shape information in a database; means for obtaining detailed information related to the product the user is viewing from the database; means for calling a generation AI model that creates a generated image based on the user's facial photograph, body shape information, and product details; means for transmitting the image generated by the generation AI model to the user terminal; and means for temporarily storing the generated image in cloud storage. This makes it possible for a user to check in real time how the clothing will look on them and whether the size will fit them before purchasing the product.

[1332] A "user terminal" refers to a computer device operated by a user using online shopping, and includes devices such as smartphones, tablets, and personal computers.

[1333] A "face photograph" is image data of a user's face, used as identification information.

[1334] "Body shape information" refers to data representing the physical dimensions of the user's body, including information such as height, weight, and bust / waist / hip measurements.

[1335] A "database" is an information system that organizes and stores large amounts of data, and it is a place where user information and product information are stored.

[1336] "Product details" refers to all information about the product, including data such as material, color, size, and price.

[1337] A "generative AI model" is a model that uses artificial intelligence techniques, such as generative adversarial networks (GANs), to generate new images from specific input data.

[1338] A "generated image" is an image created by a generative AI model, simulating what a user would look like wearing a specific product.

[1339] "Cloud storage" is a remote storage service for saving and managing data over a network, and is used to store generated images and other user data.

[1340] A "notification" is the process of transmitting specific information to a user's device, including a message informing them that a new generated image is ready.

[1341] "Display" refers to the act of visually presenting information on a user's terminal, and is a means of showing the generated image to the user.

[1342] Modes for carrying out the invention

[1343] This invention provides a system that allows users to check the actual fit and size of clothing when purchasing it online. This system primarily consists of a user terminal, a server, and a generating AI model. Specific embodiments are described below.

[1344] System Overview

[1345] 1. User registration and initial setup

[1346] First, the user uploads their facial photo and body shape information from their device. The user's device receives and processes this information and sends it to the server. The server stores the received facial photo and body shape information in a database and generates a user profile.

[1347] 2. Acquisition and analysis of product information

[1348] When a user views a product page on an e-commerce site that they are considering purchasing, the server retrieves detailed information about that product (material, color, size, etc.) from its database. The server then analyzes the retrieved product information and prepares it for input into a generating AI model.

[1349] 3. Image generation using AI models

[1350] The server prepares a dataset for generating images based on the user's facial photo, body type information, and product details. Next, the server invokes a generative AI model and inputs the pre-prepared dataset. Based on this data, the generative AI model generates realistic images of the user wearing the selected clothing. The generated images are temporarily stored in cloud storage by the server.

[1351] 4. User Feedback

[1352] The server notifies the user's device when the generated wearing image is complete. The user's device displays the notification received from the server, and the user can then go to the product page to check the generated image. This allows the user to check in advance whether the item suits them and whether the size is correct, supporting their purchase decision.

[1353] Technologies and hardware / software used

[1354] Smartphone app: Application for iOS or Android

[1355] Server: Cloud servers such as AWS

[1356] Generative AI Model: A custom model based on GAN (Generative Adversarial Network).

[1357] Cloud storage: Cloud storage services such as AWS S3

[1358] Specific example

[1359] For example, consider a case where user A wants to purchase a summer dress made of linen. First, A registers with the app and uploads a photo of their face and body measurements. This information is sent to and stored on the server. Next, when A views the product page for the linen summer dress on the e-commerce site, the server retrieves the detailed information about the dress and uses a generative AI model to generate an image of A wearing the dress. The generated image is temporarily stored in cloud storage, and A is notified when the generation is complete. A receives this notification in the app and checks the generated image of them wearing the dress. This allows A to check the size and fit before purchasing, enabling them to proceed with the purchase with confidence.

[1360] Examples of prompt statements

[1361] User A is browsing the product page for a linen summer dress. The server retrieves the dress's material and size information, and a generation AI model generates a model of User A wearing the dress based on their face photo and body shape information. The app notifies User A of the generated image, and User A confirms the image.

[1362] Thus, the present invention provides a system that can improve the user's online shopping experience and enhance the accuracy of their purchase decisions.

[1363] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[1364] Step 1:

[1365] User registration and initial setup

[1366] Users input and upload their facial photo and body measurements using a smartphone app. The user's device sends this information to the server. The server stores the received facial photo and body measurements in a database and generates a user profile. This profile includes detailed information such as the user's name, facial photo, height, weight, and bust / waist / hip measurements.

[1367] Input: User's profile picture, body information (height, weight, bust / waist / hip measurements, etc.)

[1368] Output: User profile is saved to the database.

[1369] Step 2:

[1370] Product information acquisition and analysis

[1371] When a user views a product page on an e-commerce site, their device sends a request for that page to the server. The server retrieves detailed product information from its database based on the product ID. This information includes the product's material, color, and size. The server analyzes this information and prepares a dataset for input into a generating AI model.

[1372] Input: Product ID

[1373] Output: Product details (material, color, size, etc.)

[1374] Step 3:

[1375] Image generation using AI models

[1376] The server inputs the prepared dataset (user's facial photo, body type information, and product details) into the generating AI model. Based on the input data, the generating AI model creates realistic images of the user wearing the selected clothing. The generated images are temporarily stored in cloud storage by the server.

[1377] Input: Dataset (user's profile picture, body type information, product details)

[1378] Output: Generated image

[1379] Step 4:

[1380] User Feedback

[1381] The server notifies the user's device that the generation of the generated wearing image is complete. The user's device receives this notification and displays it to the user. The user opens the app and checks the generated image. Based on this image, the user decides whether the product suits them and whether to purchase it.

[1382] Input: Generated image, user terminal

[1383] Output: Display of notifications and images.

[1384] Specifically, if a user selects a linen summer dress, the server retrieves the product's material and size information, and a generation AI model generates an image based on the user's facial photo and body shape information. The generated image is then notified to the user, who then makes a purchase decision based on the image. This smooth process allows users to enjoy online shopping with peace of mind.

[1385] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[1386] This invention provides a system that allows users to check the actual fit and size of clothing when purchasing it on an online shopping site (EC site), and also recognizes the user's emotional state to suggest appropriate products. Specific embodiments of the system are described in detail below.

[1387] System Overview

[1388] This system consists of a user terminal, a server, a generative AI model, and an emotion engine. The user uploads their facial photo and body shape information from the terminal, which the server receives and stores. The server retrieves detailed information about the products the user is viewing and uses the generative AI model to generate realistic images of the clothes being worn. Furthermore, the emotion engine analyzes the user's facial expressions and behavioral data, recognizes the user's emotional state, adjusts the images generated by the generative AI model based on this, and notifies the user terminal. This allows the user to receive appropriate product suggestions and view images of themselves wearing clothes online.

[1389] Program processing details

[1390] 1. User registration and initial setup

[1391] The user inputs and uploads their facial photo and detailed body information (e.g., height, weight, bust / waist / hip measurements) from their device. The device sends the entered facial photo and body information to the server. The server stores the received facial photo and body information in a database and generates a user profile.

[1392] 2. Acquisition and analysis of product information

[1393] The user browses the product page of clothing they are considering purchasing on an e-commerce site. The server retrieves detailed information about the product the user is viewing (material, color, size, etc.) from the database. The server analyzes the retrieved product details and prepares a dataset to be input into a generative AI model.

[1394] 3. Analysis using an emotion engine

[1395] The terminal collects user facial expressions and behavioral data and sends it to the server. The server invokes an emotion engine and analyzes the user's emotional state based on the data. The emotional state recognized by the emotion engine is reflected in the dataset input to the generative AI model.

[1396] 4. Image generation using AI models

[1397] The server prepares a dataset for creating generated images based on the user's facial photograph, body type information, and product details. This also includes emotional states analyzed by the emotion engine. The server invokes a generative AI model and inputs the prepared dataset. Based on the input data, the generative AI model generates realistic images of the user wearing the clothing selected by the user. The generated images are temporarily stored by the server.

[1398] 5. User Feedback

[1399] The server notifies the device that the generated wearing image is complete. The device displays the notification received from the server to the user, who then navigates to the product page to view the generated image. The device receives the worn image generated in response to the request from the server and displays it on the product details page. The user checks whether the product suits them based on the generated realistic wearing image and makes a purchase decision.

[1400] Specific example

[1401] For example, if a female user is considering purchasing a linen summer dress, she uploads a photo of her face and body measurements (e.g., height 165cm, weight 55kg, bust 85cm, waist 65cm, hips 90cm) from her device. This information is sent to and stored on the server.

[1402] Next, when a user views the product page for a linen summer dress, the server retrieves detailed information about the dress (linen material, yellow, size information S, M, L, XL) from the database. Based on the user's facial photo and body type information, as well as the product details, the server prepares a dataset to create a generated image by adding the emotional state analyzed from the user's facial expression data and behavioral data using an emotion engine.

[1403] The generated image is sent from the server to the user's terminal, and after the user confirms the notification, they check the generated image on the product page. This allows the user to check in advance whether the item suits them and whether the size is correct. Furthermore, suggestions are made according to the user's emotional state, providing a more satisfying shopping experience. Through this entire process, users can enjoy online shopping with peace of mind.

[1404] The following describes the processing flow.

[1405] Step 1:

[1406] Users input and upload their own facial photo and body information (height, weight, bust / waist / hip measurements, etc.) from their device.

[1407] Step 2:

[1408] The device sends the entered facial photo and body shape information to the server.

[1409] Step 3:

[1410] The server stores the received facial photograph and body shape information in a database and generates a user profile.

[1411] Step 4:

[1412] The user browses the product page of clothing they are considering purchasing on the e-commerce site.

[1413] Step 5:

[1414] The server retrieves detailed information (such as material, color, and size) related to the product the user is viewing from the database.

[1415] Step 6:

[1416] The device collects facial and behavioral data while the user is browsing products and sends it to the server.

[1417] Step 7:

[1418] The server invokes the emotion engine to analyze the user's emotional state based on the received facial and behavioral data. This analysis result is added to the user's profile information.

[1419] Step 8:

[1420] The server prepares a dataset for generating images based on the user's facial photograph, body type information, product details, and emotional state determined by the emotion engine.

[1421] Step 9:

[1422] The server invokes the generative AI model and inputs the prepared dataset.

[1423] Step 10:

[1424] The generative AI model generates realistic images of the user wearing the clothes selected by the user, based on the input data.

[1425] Step 11:

[1426] The server temporarily stores the generated image.

[1427] Step 12:

[1428] The server notifies the terminal that the generated image is complete.

[1429] Step 13:

[1430] The device receives a notification from the server and displays a message to the user indicating that the generated image is available for viewing.

[1431] Step 14:

[1432] The user checks the notification, navigates to the product page, and views the generated image.

[1433] Step 15:

[1434] The terminal requests an image from the server and displays the received image on the product details page.

[1435] Step 16:

[1436] Users check whether the product suits them based on the generated realistic images of them wearing it. They also consider suggestions from the server, if appropriate products are presented based on their emotional state.

[1437] Step 17:

[1438] Users decide to purchase a product if they determine that it is suitable for them.

[1439] (Example 2)

[1440] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1441] In modern online shopping, users cannot physically try on products, making it difficult to check whether an item suits them or fits before purchasing. This often leads to returns or exchanges after the product arrives due to reasons such as the item not matching expectations or being the wrong size, resulting in decreased user satisfaction. Furthermore, there is a lack of appropriate product suggestions that take into account the user's emotional state, highlighting the need for improved satisfaction.

[1442] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[1443] In this invention, the server includes means for receiving a user's facial photograph and body shape information from a user terminal; means for storing the user's facial photograph and body shape information in a database; means for obtaining detailed information related to the product the user is viewing from the database; means for collecting the user's facial expression and behavior data; means for analyzing the user's emotional state based on the facial expression and behavior data; means for calling a generation AI model that creates a generated image based on the user's facial photograph, body shape information, product details, and emotional state; means for temporarily storing the image generated by the generation AI model; and means for transmitting the generated image to the user terminal. As a result, the user can not only check in advance whether the product suits them and what size it is, but also receive appropriate product suggestions according to their emotional state.

[1444] A "user terminal" is an electronic device used by a user, which is used to input the user's facial photograph and body shape information, and to communicate with the server.

[1445] A "face photograph" is image data of the user's face, providing information for the system to perform facial recognition.

[1446] "Body type information" refers to information that indicates the user's physical characteristics, such as height, weight, and bust / waist / hip measurements.

[1447] A "server" is a computing system that receives and stores data sent from user terminals, and performs data processing and analysis.

[1448] A "database" is a system that manages and stores data such as users' facial photos, body type information, and product details.

[1449] "Product details" refers to information about the product the user is viewing, including material, color, size, etc.

[1450] "Facial expression data" refers to data that shows the user's facial expressions, and is collected using sensors such as cameras.

[1451] "Behavioral data" refers to data that shows a user's online actions, such as their mouse and keyboard operations.

[1452] "Emotional analysis means" refers to a system or method for analyzing a user's facial expression data and behavioral data to identify the user's emotional state.

[1453] A "generative AI model" is an artificial intelligence model that takes various data as input and generates realistic imagery that makes it appear as if the user is wearing the product they have selected.

[1454] An "image" is a virtual image generated by a generative AI model, showing the user wearing the product they selected.

[1455] "Temporary storage" is the process of saving generated image data for a certain period of time and making it accessible as needed.

[1456] This invention provides a system that allows users to check the actual fit and size of clothing when purchasing it on an online shopping site (EC site), and also recognizes the user's emotional state to suggest appropriate products. The following describes specific embodiments of the system in detail.

[1457] System Overview

[1458] This system consists of a user terminal, a server, a generative AI model, and an emotion engine.

[1459] 1. User terminal:

[1460] A user terminal is a device used by users to input facial photos and body shape information and send it to a server. Specific examples include personal computers, smartphones, and tablets. User terminals are equipped with webcams and front cameras, enabling the collection of facial expression data and the recording of behavioral data.

[1461] 2. Server:

[1462] The server receives data sent from users and stores it in a database. Specifically, it manages data such as facial photos, body shape information, product details, and facial expression data. It also plays a role in analyzing the user's emotional state using an emotion engine and passing the results to the generating AI model.

[1463] 3. Generative AI Models:

[1464] The generative AI model generates realistic images of the user wearing the clothing selected by the user, based on the input data (user's facial photo, body type information, product details, and emotional state). The AI ​​model used includes computer vision technology and deep learning technology.

[1465] 4. Emotional Engine:

[1466] An emotion engine is software that analyzes facial expression data and behavioral data to recognize a user's emotional state. This engine identifies a user's emotions using, for example, facial expression recognition algorithms and machine learning models.

[1467] Specific example

[1468] For example, if a female user is considering purchasing a summer dress made of linen, the system will operate through the following steps.

[1469] 1. The user terminal allows female users to input their facial photograph and body measurements (e.g., height 165cm, weight 55kg, bust 85cm, waist 65cm, hips 90cm) and send this information to the server.

[1470] 2. The server stores the received facial photo and body shape information in a database and generates a user profile. At the same time, it retrieves information that the user is viewing a product page for a linen summer dress.

[1471] 3. The device collects emotional data from the user through facial expression data while the user is viewing the product page. This includes, for example, smiles or frowns while viewing product details.

[1472] 4. The server inputs the collected facial expression data into the emotion engine and obtains results that analyze the user's emotional state in real time. Based on the results, the emotional state is reflected in the dataset that is input into the generative AI model.

[1473] 5. The generation AI model generates a realistic image of the user wearing a linen summer dress, based on the user's facial photo, body type information, product details, and the results of the emotion engine's analysis. This image is temporarily stored on the server.

[1474] 6. The server notifies the user's device that the generated wearing image is complete. The user's device displays this notification to the user. The user can check the generated image on the product page and check in advance whether it suits them.

[1475] Example of a prompt

[1476] "Please generate a realistic image of a 165cm, 55kg woman trying on a yellow linen summer dress she is considering purchasing. Please also ensure the user has a happy expression in the image."

[1477] By inputting this prompt into the AI ​​model, it becomes possible to generate realistic images of the garment being worn based on specific scenarios.

[1478] The flow of the specific processing in Example 2 will be explained using Figure 13.

[1479] Step 1:

[1480] The user accesses the e-commerce site, enters a photo of their face and body information (e.g., height, weight, bust / waist / hip measurements) on their profile page, and clicks the "Upload" button. The input data (face photo, height, weight, bust / waist / hip measurements) is collected by the device. The device sends this information to the server using an HTTP POST request. The server stores the received face photo and body information in a database and generates a user profile. As output, the user profile is saved in the database.

[1481] Step 2:

[1482] A user views a product page for clothing they are considering purchasing on an e-commerce site. When a user opens a product page, the server monitors this access event and obtains the ID of the product the user is viewing. Based on the obtained product ID, the server retrieves detailed information about the product (material, color, size, etc.) from the database. As output, the server retrieves the detailed product information and passes it to the next analysis step.

[1483] Step 3:

[1484] The device collects facial expression data in real time through the user's camera and simultaneously monitors behavioral data such as the user's mouse and keyboard operations. This facial expression and behavioral data is transmitted to a server via a secure channel. The server inputs the received facial expression and behavioral data into an emotion engine, which analyzes the emotional state in real time. The emotion engine identifies the emotional state from the user's facial expressions and actions and reflects the results in a generating AI model. As output, the user's emotional state is analyzed and passed on to the next generation step.

[1485] Step 4:

[1486] The server prepares a dataset of generated images based on the user's facial photograph, body type information, product details, and the results of the emotion engine's analysis. The server inputs this dataset into the generative AI model. The generative AI model uses the input data, along with prompts, to generate realistic images of the user wearing the selected clothing. The generated images are temporarily stored on the server. The output is the generated realistic image.

[1487] Step 5:

[1488] The server notifies the user's device that the generated wearing image is complete. This notification is sent in real-time via WebSocket or push notification. The device receives the notification from the server and displays a pop-up notification to the user. The user confirms the notification and is redirected to the product page. The device receives the worn image generated in response to the request from the server and displays it on the product details page. As output, the user can review the generated image and make a final purchase decision.

[1489] (Application Example 2)

[1490] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1491] Conventional online shopping systems have made it difficult for users to check the actual fit and size of clothing they are considering purchasing, leading to problems such as high return rates and low customer satisfaction. Furthermore, because product suggestions are made uniformly without considering the user's emotional state, it has been impossible to suggest products that are best suited to each individual user. The objective of this invention is to support users' purchasing decisions and improve the customer experience by allowing them to check realistic images of how clothes look when worn online and by providing appropriate product suggestions based on their emotional state.

[1492] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving the user's facial photograph and body shape information, means for storing this information in a database, means for obtaining detailed information related to the product the user is viewing, means for recognizing the user's facial expression, means for analyzing the user's emotional state, means for calling a generation AI model that creates a generated image based on the user's facial photograph, body shape information, product details, and the user's emotional state, and means for transmitting the generated image to the user terminal. As a result, the user can check images of clothes that suit them in real time and receive product suggestions that match their emotions.

[1493] A "user terminal" is a device that allows a user to input or receive information, and specific examples include smartphones and head-mounted displays.

[1494] A "face photo" is a digital image of the user's face.

[1495] "Body type information" refers to the user's physical data, such as height, weight, and bust / waist / hip measurements.

[1496] A "database" is a place where information is stored, managed, and made easily accessible.

[1497] "Product details" refers to information that describes the characteristics of the product the user is viewing, such as its material, color, and size.

[1498] A "generated image" is an image created by a generative AI model that shows what a user would look like wearing the product.

[1499] A "generative AI model" is an artificial intelligence model that uses machine learning or deep learning techniques to create generated images from input data.

[1500] "Emotional state" refers to the psychological and emotional state of a user, obtained by analyzing their facial expressions and behavioral data.

[1501] "Customizing product recommendations" means taking into account the user's emotional state and history to individually recommend the most suitable products.

[1502] This invention provides a system that allows users to check the actual fit and size of clothing when purchasing it on an online shopping site, and also recognizes the user's emotional state to suggest appropriate products. This system consists of a user terminal, a server, a generative AI model, and an emotion engine.

[1503] System Configuration

[1504] The server is configured using the following hardware and software:

[1505] Hardware: High-performance servers, database servers, storage

[1506] software:

[1507] Sentiment analysis engine: Amazon Rekognition, Microsoft Azure Face API

[1508] Generative AI models: DALL-E, Stable Diffusion

[1509] Databases: Amazon RDS, Firebase Firestore

[1510] User terminal

[1511] Users access the system using a smartphone or head-mounted display (HMD). The user terminal performs the following actions:

[1512] 1. Enter your facial photo and body type information:

[1513] Users take a photo of their face using their device's camera and input body information such as height, weight, and bust / waist / hip measurements.

[1514] The terminal sends this information to the server.

[1515] 2. Recognition and transmission of facial expressions:

[1516] The device's camera recognizes the user's facial expressions in real time and sends that data to the server.

[1517] server

[1518] The server performs the following actions:

[1519] 1. Information storage and analysis:

[1520] The system stores the received facial photograph and body type information in a database and generates a user profile.

[1521] Retrieve detailed information about the product the user is viewing from the database.

[1522] 2. Analysis of emotional state:

[1523] The system analyzes facial expression data to determine the user's emotional state.

[1524] 3. Creating the generated image:

[1525] Based on the user's facial photo, body type information, product details, and emotional state, a generative AI model is invoked to create generated images.

[1526] The generated images are temporarily stored on the server.

[1527] 4. Customization of notifications and product suggestions:

[1528] The generated image is sent to the user's terminal, and product suggestions are customized based on the user's emotional state.

[1529] Specific example

[1530] For example, if a user is using their smartphone to access a virtual store and is looking for a casual shirt for summer:

[1531] 1. The user takes a photo of themselves with their smartphone camera and enters their body shape information.

[1532] 2. When a user selects a casual shirt, the server retrieves the product information.

[1533] 3. While the user is browsing the product page for a shirt, the smartphone camera analyzes the user's facial expressions and determines that they are showing interest.

[1534] 4. The server displays to the user in real time an image of the product being worn, generated based on the user's facial photo, body type information, product details, and emotional state.

[1535] 5. If the user looks satisfied, the app will suggest, "Would you also like some pants or accessories to go with this shirt?"

[1536] Example of a prompt

[1537] "Enter the user's profile picture, body type information, and product details (shirt material, color, size), and based on the user's emotional state (interest level), generate a real-time image of the product being worn, which will be displayed on their smartphone."

[1538] This system allows users to see images of clothes that suit them in real time and enjoy a comfortable shopping experience while receiving product suggestions tailored to their emotions.

[1539] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[1540] Step 1:

[1541] The user enters a facial photograph and body information into a smartphone or head-mounted display (HMD). The entered information includes the user's facial photograph and detailed body data (e.g., height, weight, bust / waist / hip measurements). This data is sent to a server via the device. The server stores the received data in a database and generates a user profile.

[1542] Step 2:

[1543] A user views the product page for a piece of clothing they are considering purchasing within a virtual store. The server retrieves and analyzes detailed information about the selected product (e.g., material, color, size) from the database. This information is properly configured as it will be used in subsequent processing.

[1544] Step 3:

[1545] The device uses a camera to recognize the user's facial expressions in real time. The recognized facial expression data is analyzed using an emotion analysis engine (e.g., Amazon Rekognition, Microsoft Azure Face API). The analysis results in the user's emotional state (e.g., interest, satisfaction, dissatisfaction, etc.). This emotional state data is sent to the server.

[1546] Step 4:

[1547] The server creates a dataset based on the acquired emotional state data, combining it with the user's facial photograph, body shape information, and product details. The created dataset is then input into a generative AI model (e.g., DALL-E, Stable Diffusion). Based on this input data, the generative AI model generates realistic images of the user wearing the clothes.

[1548] Step 5:

[1549] The server temporarily stores the generated image. It then sends the generated image to the user's terminal. The terminal displays the received image to the user, providing visual feedback as if the user were actually trying on the clothes.

[1550] Step 6:

[1551] The user reviews the generated image. The device then re-analyzes the user's emotional state and sends the data to the server. The server customizes product suggestions based on the latest emotional state. For example, if the user's emotional state is "satisfied," it will suggest other related products (e.g., matching pants or accessories).

[1552] Step 7:

[1553] The server sends customized product suggestions to the user's terminal. The terminal displays details of the suggested products and images of the items being worn to the user. Based on this information, the user can make a purchase decision.

[1554] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[1555] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1556] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[1557] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1558] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[1559] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[1560] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[1561] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[1562] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[1563] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[1564] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[1565] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[1566] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[1567] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1568] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[1569] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[1570] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[1571] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[1572] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[1573] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[1574] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[1575] The following is further disclosed regarding the embodiments described above.

[1576] (Claim 1)

[1577] A means for receiving a user's facial photograph and body shape information from the user's terminal,

[1578] Means for storing the user's facial photograph and body shape information in a database,

[1579] A means for obtaining detailed information related to the product being viewed by the user from a database,

[1580] A means for calling a generative AI model that creates a generated image based on the user's facial photograph, body shape information, and product details,

[1581] A means for transmitting the image generated by the aforementioned AI model to the user terminal,

[1582] A system that includes this.

[1583] (Claim 2)

[1584] A means for notifying the user terminal of the generated image,

[1585] Means for displaying the aforementioned notification on the user's terminal,

[1586] The system according to claim 1, further comprising:

[1587] (Claim 3)

[1588] The system according to claim 1, further comprising means for the user to review the generated image and to support the user in making a decision to purchase the product.

[1589] "Example 1"

[1590] (Claim 1)

[1591] A means for receiving a user's facial photograph and body shape information from the user's terminal,

[1592] Means for storing the user's facial photograph and body shape information in a database,

[1593] A means for obtaining detailed information related to the product being viewed by the user from a database,

[1594] A means for calling a generative AI model that creates a generated image based on the user's facial photograph, body shape information, and product details,

[1595] A means for temporarily storing the image generated by the aforementioned AI model,

[1596] Means for transmitting the generated image to the user terminal,

[1597] A means for notifying the user terminal that the generated image is complete,

[1598] A system that includes this.

[1599] (Claim 2)

[1600] A means for notifying the user terminal of the generated image,

[1601] Means for displaying the aforementioned notification on the user's terminal,

[1602] The user terminal includes means for decoding and displaying the generated image,

[1603] The system according to claim 1, further comprising:

[1604] (Claim 3)

[1605] The system according to claim 1, further comprising means for the user to review the generated image and to support the user in making a decision to purchase the product.

[1606] "Application Example 1"

[1607] (Claim 1)

[1608] A means for receiving a user's facial photograph and body shape information from the user's terminal,

[1609] Means for storing the user's facial photograph and body shape information in a database,

[1610] A means for obtaining detailed information related to the product being viewed by the user from a database,

[1611] A means for calling a generative AI model that creates a generated image based on the user's facial photograph, body shape information, and product details,

[1612] A means for transmitting the image generated by the aforementioned AI model to the user terminal,

[1613] A means for notifying the user terminal of the generated image,

[1614] Means for displaying the aforementioned notification on the user's terminal,

[1615] A means for temporarily storing the generated image in cloud storage,

[1616] A system that includes this.

[1617] (Claim 2)

[1618] A means for notifying the user terminal of the generated image,

[1619] Means for displaying the aforementioned notification on the user's terminal,

[1620] The function of displaying the generated image on the user terminal,

[1621] The system according to claim 1, further comprising:

[1622] (Claim 3)

[1623] The system according to claim 1, further comprising means for the user to review the generated image and to support the user in making a decision to purchase the product.

[1624] "Example 2 of combining an emotion engine"

[1625] (Claim 1)

[1626] A means for receiving a user's facial photograph and body shape information from the user's terminal,

[1627] Means for storing the user's facial photograph and body shape information in a database,

[1628] A means for obtaining detailed information related to the product being viewed by the user from a database,

[1629] A means for collecting the user's facial expressions and behavioral data,

[1630] An emotion analysis means for analyzing the user's emotional state based on the aforementioned facial expression and behavioral data,

[1631] A means for calling a generative AI model that creates a generated image based on the user's facial photograph, body shape information, product details, and emotional state,

[1632] A means for temporarily storing the image generated by the aforementioned AI model,

[1633] Means for transmitting the generated image to the user terminal,

[1634] A system that includes this.

[1635] (Claim 2)

[1636] A means for notifying the user terminal of the generated image,

[1637] Means for displaying the aforementioned notification on the user's terminal,

[1638] The system according to claim 1, further comprising:

[1639] (Claim 3)

[1640] The system according to claim 1, further comprising means for the user to review the generated image and to support the user in making a decision to purchase the product.

[1641] "Application example 2 when combining with an emotional engine"

[1642] (Claim 1)

[1643] A means for receiving a user's facial photograph and body shape information from the user's terminal,

[1644] Means for storing the user's facial photograph and body shape information in a database,

[1645] Means for obtaining detailed information related to the product being viewed by the user,

[1646] A means of recognizing the user's facial expressions,

[1647] A means for analyzing the emotional state of the user,

[1648] A means for calling a generative AI model that creates a generated image based on the user's facial photograph, body shape information, product details, and the user's emotional state,

[1649] A means for transmitting the image generated by the aforementioned AI model to the user terminal,

[1650] A system that includes this.

[1651] (Claim 2)

[1652] A means for notifying the user terminal of the generated image,

[1653] Means for displaying the aforementioned notification on the user's terminal,

[1654] A means for customizing product suggestions based on the user's emotional state,

[1655] The system according to claim 1, further comprising:

[1656] (Claim 3)

[1657] A means for the user to review the generated image and assist in making a purchase decision,

[1658] The system according to claim 1, including the following: [Explanation of Symbols]

[1659] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. A means for receiving a user's facial photograph and body shape information from the user's terminal, Means for storing the user's facial photograph and body shape information in a database, A means for obtaining detailed information related to the product being viewed by the user from a database, A means for calling a generative AI model that creates a generated image based on the user's facial photograph, body shape information, and product details, A means for transmitting the image generated by the aforementioned AI model to the user terminal, A system that includes this.

2. A means for notifying the user terminal of the generated image, Means for displaying the aforementioned notification on the user's terminal, The system according to claim 1, further comprising:

3. The system according to claim 1, further comprising means for the user to review the generated image and to support the user in making a decision to purchase the product.

Images