System

Abstract

To provide a system.SOLUTION: The system includes: first means for receiving an input of a color, a shape, and a keyword of a style from a user; second means for generating an image for the keyword by using the generation Al model; third means for searching a database for a product on the basis of the generated image; fourth means for presenting the product searched for to the user.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] When a specific product is not yet decided, it is difficult to find a specific product from only a vague image. This problem exists not only for physical products but also for non-physical products (such as novels and songs). [Means for solving the problem]

[0005] We provide a system that generates an image based on user input and searches for products based on that image. This system has a function to modify the image based on user instructions, and is capable of searching for not only physical products but also non-physical products. [Brief explanation of the drawings]

[0006] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 2 is a sequence diagram showing a flow of processing in the data processing system according to the first embodiment of the first form example. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1 of Embodiment 1. [Figure 13] FIG. 10 is a sequence diagram showing a processing flow of a data processing system in a second embodiment of the second form example. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 of Embodiment Example 2. [Figure 15] FIG. 10 is a sequence diagram showing the flow of processing in a data processing system according to a third embodiment of the third embodiment. [Figure 16] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 3 of Embodiment 3. [Figure 17] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in the first embodiment of the first form example when an emotion engine is combined. [Figure 18] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1 of Form Example 1 when an emotion engine is combined. [Figure 19] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in the second embodiment of the second form example when an emotion engine is combined. [Figure 20] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 of Form Example 2 when an emotion engine is combined. [Figure 21] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in the third embodiment of the third form example when an emotion engine is combined. [Figure 22] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 3 of Form Example 3 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0007] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0008] First, the terms used in the following description will be explained.

[0009] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, the processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), or a TPU (TENSOR PROCESSING UNIT (registered trademark)).

[0010] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0011] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0012] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0013] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0014] [First embodiment]

[0015] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0016] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0017] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0018] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0019] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0020] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0021] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0022] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0023] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0024] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0025] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0026] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[0027] "Example 1"

[0028] The system of the present invention receives input from a user (e.g., keywords such as color, shape, and style) and generates an image based on that input. This image generation is performed using artificial intelligence techniques such as deep learning. The generated image is presented to the user and can be revised based on user feedback.

[0029] "Example 2"

[0030] The generated and modified image is used to search the product database. This search is performed using image recognition technology to find products similar to the image. The found products are then presented to the user.

[0031] "Example 3"

[0032] The system of the present invention can be applied to not only physical products but also non-physical products. For example, if a user inputs an image such as "a heartwarming story to read in a quiet forest," the system can generate a novel cover and synopsis based on that image and search for and present similar novels.

[0033] The processing flow of each embodiment will be described below.

[0034] "Example 1"

[0035] Step 1: Receive input from the user (e.g. keywords such as color, shape, style, etc.).

[0036] Step 2: Based on the received input, an image is generated using artificial intelligence techniques such as deep learning.

[0037] Step 3: Present the generated image to the user and provide the ability to modify it based on user feedback.

[0038] "Example 2"

[0039] Step 1: Generate and correct the image.

[0040] Step 2: Search the product database based on the acquired image. This search is performed using image recognition technology.

[0041] Step 3: Present the found products to the user as search results.

[0042] "Example 3"

[0043] Step 1: Receive input from the user (e.g., "a heartwarming story to read in a quiet forest").

[0044] Step 2: Generate a cover and synopsis for the novel based on the input received.

[0045] Step 3: Based on the generated cover and synopsis, search the novel database to find similar novels.

[0046] Step 4: Present the found novels to the user.

[0047] Example 1

[0048] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0049] In conventional systems, when generating images based on user input, the generated images often did not meet the user's expectations. Furthermore, when searching for products based on generated images, the system was unable to accurately reflect the user's intentions. Furthermore, the system lacked the ability to modify images based on user feedback, making it difficult to increase user satisfaction.

[0050] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0051] In this invention, the server includes means for generating an image based on user input, means for searching for products based on the generated image, means for presenting the searched products to the user, means for analyzing the user input and generating a prompt sentence for the generative AI model, means for inputting the prompt sentence into the generative AI model and generating an image, and means for modifying the prompt sentence based on user feedback and generating the image again. This makes it possible to generate an image that accurately reflects the user's input, and by further modifying the image to reflect user feedback, it is possible to increase user satisfaction.

[0052] "User input" refers to information that a user provides to the system through a terminal, including keywords such as color, shape, and style.

[0053] An "image" is a visual image generated based on user input and created by a generative AI model.

[0054] A "generative AI model" is an artificial intelligence model that uses deep learning technology to generate image images from prompt text, and examples include Stable Diffusion (registered trademark) and DALL-E (registered trademark).

[0055] A "prompt" is a text-based instruction generated based on user input and input into a generative AI model.

[0056] "Feedback" refers to requests for corrections or opinions that the user makes about the generated image, and is reflected when the system regenerates the image.

[0057] "Products" are objects that are searched for based on the generated image, and include physical products (such as clothes and furniture) and non-physical products (such as novels and songs).

[0058] The "server" is a central processing unit that receives input from users, inputs prompts into the generative AI model, and generates and modifies images.

[0059] A "terminal" is a device that allows a user to access the system and provide input or feedback, including PCs and smartphones.

[0060] MODE FOR CARRYING OUT THE INVENTION

[0061] This invention is a system that generates an image based on user input, searches for products based on the image, and presents them to the user. It also has the function of modifying the image based on user feedback.

[0062] Hardware and software used

[0063] The system uses the following hardware and software:

[0064] Server: Acts as the central processing unit, receives input from the user, provides prompts to the generative AI model, and generates and modifies images.

[0065] Terminal: A device that allows users to access the system and provide input and feedback. This includes PCs, smartphones, etc.

[0066] Generative AI model: An artificial intelligence model that uses deep learning techniques to generate images from prompts. Examples include Stable Diffusion and DALL-E.

[0067] Data processing and calculation

[0068] The server receives input from the user and performs data processing and calculations according to the following steps:

[0069] 1. Receiving user input: The user accesses the system interface through a terminal and inputs keywords such as color, shape, style, etc.

[0070] 2. Prompt generation: The server analyzes the keywords sent by the user and generates a prompt suitable for the generative AI model. For example, from the keywords "blue sky, white clouds, landscape," it generates the prompt "Generate an image of a landscape with a blue sky and white clouds."

[0071] 3. Image generation: The server inputs the generated prompt into the generative AI model to generate an image. The generative AI model generates an image based on the prompt using deep learning technology.

[0072] 4. Sending the image: The server sends the generated image to the user's terminal and presents it to the user.

[0073] 5. Receiving Feedback: The user provides feedback on the image presented to them, for example, by inputting specific requests for corrections such as "make the clouds bigger" or "make the sky brighter."

[0074] 6. Prompt Revision: The server receives feedback from the user and regenerates the prompt. For example, in response to the user's feedback, "I want the clouds to be bigger," the server generates a revised prompt, "Generate an image of a landscape with a blue sky and larger white clouds."

[0075] 7. Generate a modified image: The server inputs the modified prompt into the generative AI model to generate a modified image. The modified image is then sent back to the user's device.

[0076] Specific examples

[0077] As a concrete example, consider the case where a user inputs "a landscape with blue sky and white clouds." The user inputs the keywords "blue sky, white clouds, landscape" through the terminal. The server receives this and inputs the following prompt sentence to the generative AI model:

[0078] Example prompt sentence:

[0079] "Generate an image of a landscape with a blue sky and white clouds."

[0080] The generative AI model generates an image based on this prompt, and the server sends the generated image to the user's device. If the user provides feedback such as "I want the clouds to be bigger," the server again inputs the following modified prompt into the generative AI model:

[0081] Example of a modified prompt:

[0082] "Generate an image of a landscape with a blue sky and larger white clouds."

[0083] In this way, the image is modified based on user feedback.

[0084] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0085] Step 1:

[0086] The user enters a keyword through the terminal.

[0087] A user accesses the system's interface using their own device (e.g., a PC or smartphone). On the interface, the user enters keywords such as "blue sky, white clouds, landscape." The entered keywords are sent to the server.

[0088] Input: Keywords entered by the user (e.g., "blue sky, white clouds, landscape")

[0089] Output: Keyword data sent to the server

[0090] Step 2:

[0091] The server receives the user's input and generates a prompt.

[0092] The server receives and analyzes the keywords sent by the user. Based on the analysis results, it generates a prompt sentence appropriate for the generative AI model. For example, the keyword "blue sky, white clouds, landscape" generates the prompt sentence "Generate an image of a landscape with a blue sky and white clouds."

[0093] Input: Keyword data sent by the user

[0094] Output: The generated prompt (e.g., "Generate an image of a landscape with a blue sky and white clouds.")

[0095] Step 3:

[0096] The server inputs a prompt sentence into the generative AI model and generates an image.

[0097] The server inputs the generated prompt into the generative AI model, which uses deep learning technology to generate an image based on the prompt. The generated image is then returned to the server.

[0098] Input: Generated prompt text

[0099] Output: Generated image

[0100] Step 4:

[0101] The server sends the generated image to the user's terminal.

[0102] The server receives the image output from the generative AI model and sends it to the user's device, where the user can view the generated image on their device screen.

[0103] Input: Generated image

[0104] Output: The image sent to the user's device

[0105] Step 5:

[0106] The user provides feedback on the image.

[0107] The user provides feedback on the image presented to them, for example, by inputting specific requests for correction, such as "make the clouds bigger" or "make the sky brighter." The feedback is then sent to the server.

[0108] Input: User feedback (e.g. "I want the clouds to be bigger")

[0109] Output: Feedback data sent to the server

[0110] Step 6:

[0111] The server receives the feedback, regenerates the prompt text, and modifies the image.

[0112] The server receives feedback from the user and regenerates the prompt. For example, in response to feedback such as "I want the clouds to be larger," it generates a revised prompt, "Generate an image of a landscape with a blue sky and larger white clouds." The server then inputs this revised prompt into the generative AI model and generates a revised image. The revised image is then sent back to the user's device.

[0113] Input: User feedback data

[0114] Output: Modified prompt and modified image

[0115] (Application example 1)

[0116] Next, a description will be given of Application Example 1 of Embodiment Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0117] On traditional online shopping sites, when users searched for a specific product, they had to manually enter keywords and categories, making it difficult to find products that accurately reflected their intent. Furthermore, users had limited options for specifically expressing their desired design and style, making it difficult to find products that satisfied them. Furthermore, the lack of a function to re-search for products based on user feedback made it difficult to improve the user experience.

[0118] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0119] In this invention, the server includes means for generating an image based on input from a user, means for searching for products based on the generated image, means for presenting the searched products to the user, means for modifying the image based on feedback from the user, and means for searching for products again based on the modified image, thereby enabling a product search that accurately reflects the user's intentions and improving the user experience.

[0120] "User input" refers to keywords such as color, shape, and style that users provide to the system.

[0121] An "image" is a visual image generated based on input from a user.

[0122] "Generating means" refers to a device or program that has the function of generating image data using artificial intelligence technology such as deep learning.

[0123] The "means for searching for products" is a device or program that has the function of searching a database for related products based on the generated image.

[0124] "Means for presenting to the user" means a device or program that has the function of displaying searched items to the user in visual or text form.

[0125] "Feedback" refers to opinions and correction instructions provided by users regarding generated images and search results.

[0126] A "modifying means" is a device or program that has the ability to regenerate or modify an image based on user feedback.

[0127] The "means for searching for products again" refers to a device or program having the function of searching the database for related products again based on the corrected image.

[0128] The following system configuration and processing procedure will be described as an embodiment of the present invention.

[0129] System Configuration

[0130] Hardware:

[0131] server

[0132] User devices (smartphones, tablets, PCs, etc.)

[0133] software:

[0134] Deep learning libraries (such as TENSORFLOW (registered trademark))

[0135] Image processing library (such as PIL)

[0136] Database Management Systems

[0137] User Interface (UI) Applications

[0138] Processing steps

[0139] 1. User Input:

[0140] Users use devices such as smartphones or PCs to input keywords such as color, shape, and style, for example, "red," "dress," and "elegant."

[0141] 2. Image generation:

[0142] The server receives input from the user and generates images using a generative AI model built using deep learning libraries (such as TensorFlow).

[0143] 3. Product Search:

[0144] The server searches the database for related products based on the generated image, using an image processing library (such as PIL).

[0145] 4. User Presentation:

[0146] The searched products are presented to the user through a UI application on the user's device, where the user can review the presented products and decide whether or not to purchase them.

[0147] 5. Receiving Feedback and Making Corrections:

[0148] If the user provides feedback on the presented product or image, the server receives that feedback and modifies the image, again using the generative AI model.

[0149] 6. Search again:

[0150] Based on the revised image, the server retrieves the product again and presents it to the user. This process is repeated until the user is satisfied.

[0151] Specific examples

[0152] For example, if a user enters the keywords "red," "dress," and "elegant," the server converts these keywords into vectors and inputs them into a generative AI model. The generative AI model then generates an image of an elegant red dress based on this. The generated image is displayed on the user's device, and related products are searched for and presented. If the user provides feedback such as "I wish the red was a little brighter," the server receives that feedback, modifies the image, and searches for and presents the products again.

[0153] Example prompt sentence:

[0154] User entered keywords: ["red", "dress", "elegant"]

[0155] Vector to be input to the generative AI model: [0.5, 0.2, 0.8, ...] Keywords converted to vectors

[0156] In this way, product searches can be performed that accurately reflect the user's intent, improving the user experience.

[0157] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0158] Step 1:

[0159] The user uses the terminal to input keywords such as color, shape, style, etc. The input keywords are sent from the terminal to the server.

[0160] Input: Keywords entered by the user (e.g. "red", "dress", "elegant")

[0161] Output: Keyword data sent to the server

[0162] Step 2:

[0163] The server converts the received keywords into vectors using a pre-trained keyword embedding model.

[0164] Input: Keyword data submitted by the user

[0165] Output: Keyword vector (e.g. [0.5, 0.2, 0.8, ...])

[0166] Step 3:

[0167] The server inputs keyword vectors into the generative AI model to generate images, using deep learning libraries (such as TensorFlow).

[0168] Input: Keyword vector

[0169] Output: Generated image

[0170] Step 4:

[0171] The server searches for products based on the generated image in a database, using an image processing library (such as PIL).

[0172] Input: Generated image

[0173] Output: Searched product list

[0174] Step 5:

[0175] The server sends the searched product list to the user's terminal and presents it to the user, who then checks the presented products.

[0176] Input: Searched product list

[0177] Output: Product list displayed on user's device

[0178] Step 6:

[0179] The user provides feedback on the presented product or image, which is then sent from the device to the server.

[0180] Input: User feedback (e.g., "I wish it was a brighter red")

[0181] Output: Feedback data sent to the server

[0182] Step 7:

[0183] The server then modifies the image based on the feedback it receives, again using a generative AI model.

[0184] Input: Feedback data submitted by the user

[0185] Output: The corrected image

[0186] Step 8:

[0187] The server performs a product search again based on the corrected image, and the search results are sent again to the user's device.

[0188] Input: The corrected image

[0189] Output: Re-searched product list

[0190] Step 9:

[0191] The user reviews the presented product again and provides further feedback if necessary, and this process is repeated until the user is satisfied.

[0192] Input: Re-searched product list

[0193] Output: The user's final choice or additional feedback

[0194] Example 2

[0195] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0196] Previous systems had problems with image recognition accuracy and the relevance of search results when searching for products based on images uploaded by users. It was also difficult to quickly and accurately present the products users were looking for. Furthermore, few systems supported searches for both physical and non-physical products, making them unable to meet the diverse needs of users.

[0197] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for generating an image based on an input from a user, means for searching for products based on the generated image, means for presenting the searched products to the user, means for receiving an image uploaded by the user, means for extracting features of the received image, and means for searching a product database based on the extracted features. This makes it possible to recognize an image uploaded by a user with high accuracy and quickly search for and present appropriate products. Furthermore, by targeting both physical and non-physical products for search, it is possible to meet the diverse needs of users.

[0198] "User" means an individual or corporation that uses the system to upload images and search for products.

[0199] "Input" refers to the operations and instructions that users give to the system, and specifically includes uploading images and specifying search criteria.

[0200] "Image" refers to an image file uploaded by a user to the system, which contains visual information that is the basis for product searches.

[0201] A "generating means" is a part of a system that has the ability to create or modify an image based on input from a user.

[0202] The "searching means" is a part of the system that has the function of searching a product database based on the characteristics of the generated image and finding similar products.

[0203] The "presentation means" is a part of the system that has the function of displaying searched product information to the user.

[0204] The "means for receiving" is a part of the system that has the function of allowing the server to receive images uploaded by users.

[0205] The "means for extracting features" is a part of the system that has the function of analyzing features such as color, shape, and pattern from a received image and generating a feature vector.

[0206] "Product database" refers to a database that stores product information that the system searches, including information about physical and non-physical products.

[0207] The "searching means" is a part of the system that has the function of querying the product database based on the extracted feature vector and finding highly similar products.

[0208] MODE FOR CARRYING OUT THE INVENTION

[0209] The present invention is a system for searching for products based on images uploaded by users and presenting search results to the users. A specific embodiment of this system will be described below.

[0210] Hardware and software used

[0211] This system consists of a server and a user's terminal. The server is a computer equipped with a high-performance processor and a large amount of memory, and the user's terminal is a device such as a personal computer or smartphone. The following software is used:

[0212] Image Recognition: TensorFlow

[0213] Image processing: OpenCV (registered trademark)

[0214] Database operations: SQLAlchemy (registered trademark)

[0215] Web server: Flask (registered trademark)

[0216] System processing flow

[0217] Users access the system's web interface from their devices and upload images of the products they want to search for. The server receives these images and uses TensorFlow to perform image recognition. Specifically, it extracts features such as color, shape, and pattern from the image and generates a feature vector.

[0218] Next, the server uses SQLAlchemy to search the product database. Based on the extracted feature vector, it compares it with the feature vectors of each product in the database to find products with high similarity. The search results are sorted in descending order of similarity.

[0219] Finally, the server presents the search results to the user, specifically displaying information such as images, names, prices, and links of similar products on the user's device.

[0220] Specific examples

[0221] For example, consider the case where a user uploads an image of red shoes. The server receives the image and uses TensorFlow to extract the feature "red shoes." Next, the server searches the product database to find similar red shoe product information. Finally, the server presents the found red shoe product information to the user.

[0222] Prompt Sentence Examples

[0223] An example of a prompt to input to a generative AI model is as follows:

[0224] "I uploaded an image of a pair of red shoes. Based on this image, I want you to search our product database and find similar products."

[0225] Using this prompt, the generative AI model can perform product searches according to the user's request.

[0226] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0227] Step 1:

[0228] The user accesses the system's web interface from their terminal and uploads an image of the product they want to search for. Specifically, the user opens a browser and accesses the system's web page. They click the "Upload Image" button on the page and select an image of red shoes from the file selection dialog. After selecting, they click the "Upload" button. The input is the image selected by the user, and the output is an image file sent to the server.

[0229] Step 2:

[0230] The server receives an image uploaded by a user. The server receives the HTTP request and saves the uploaded image file in a temporary folder on the server. After saving, it checks the image format and converts it to PNG format if necessary. The input is the image file sent by the user, and the output is the image file saved on the server.

[0231] Step 3:

[0232] The server inputs the stored image file into the TensorFlow model to extract image features. Specifically, it analyzes the image's color, shape, pattern, etc., and generates a feature vector. The input is the image file stored on the server, and the output is the extracted feature vector.

[0233] Step 4:

[0234] The server uses SQLAlchemy to search the product database based on the extracted feature vector. The server converts the feature vector into a database query and searches the product database. The search results are sorted in descending order of similarity. The input is the extracted feature vector, and the output is a list of highly similar product information.

[0235] Step 5:

[0236] The server generates search results in HTML format and sends them to the user's browser. Specifically, information such as images, names, prices, and links of similar products is displayed on the user's device. The input is a list of highly similar product information, and the output is a search result page displayed in the user's browser.

[0237] (Application example 2)

[0238] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0239] On conventional online shopping sites, it was difficult for users to efficiently find the products they were looking for. In particular, when users did not know the specific product name or wanted to search for a product based on a visual image, the search process was complicated and time-consuming. This led to issues such as a decrease in users' motivation to purchase and a decrease in the frequency of site use.

[0240] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0241] In this invention, the server includes means for generating an image based on input from a user, means for searching for products based on the generated image, means for presenting the searched products to the user, and means for searching a product database based on an image uploaded by the user and displaying similar products, thereby enabling users to efficiently search for products based on visual images and easily find similar products.

[0242] "User input" refers to information or instructions provided by a user to a system.

[0243] An "image" is a visual image generated based on user input.

[0244] A "product database" is a database that stores information about various products.

[0245] "Image recognition technology" refers to the technology in which a computer analyzes images and recognizes specific features and patterns.

[0246] "Similar products" refer to products that have similar features to the image provided by the user.

[0247] A "generative AI model" is a model that uses artificial intelligence to generate new images.

[0248] A "prompt" is an instruction that a user inputs to a generative AI model.

[0249] "Upload" means the act of a user sending data from their device to the system.

[0250] "Searching" is the act of locating information in a database based on specific criteria.

[0251] "Presenting" refers to the act of displaying search results to the user.

[0252] The system for implementing this invention is based on the premise that the user operates it using a terminal such as a smartphone. The system includes the following main components:

[0253] Hardware and Software Configuration

[0254] 1. Terminal: A device such as a smartphone or tablet that is operated by a user.

[0255] 2. Server: A server for running image recognition technology and generative AI models.

[0256] 3. Software:

[0257] Python (registered trademark): Used for image processing and running machine learning models.

[0258] OpenCV: An image processing library.

[0259] scikit-learn(registered trademark): Machine learning library.

[0260] PIL (Python Imaging Library): Used to load and convert images.

[0261] Data processing and calculation

[0262] 1. User input: The user inputs a prompt to generate an image using the terminal. For example, the user can input a prompt such as, "Please upload an image of a red shirt. Similar products will be searched and displayed."

[0263] 2. Image generation: The server uses a generative AI model to generate an image based on the user's prompt.

[0264] 3. Image upload: User uploads an image from their device, which is sent to the server.

[0265] 4. Image recognition and feature extraction: The server uses OpenCV to analyze the uploaded image and extract features.

[0266] 5. Search the product database: The server uses scikit-learn to compare the extracted features with the features of products in the product database and calculate the similarity.

[0267] 6. Show similar products: The server shows similar products to the user. For example, if you upload an image of a red shirt based on the prompt "Please upload an image of a red shirt. We will search and display similar products," similar red shirt products will be displayed.

[0268] Specific examples

[0269] When a user uploads an image of a red shirt, the server analyzes the image and searches for similar red shirts in a product database. The search results show multiple red shirts on the user's device. The user can select a product to view more information or proceed with the purchase.

[0270] In this way, users can efficiently search for products based on visual images and easily find similar products.

[0271] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0272] Step 1:

[0273] The user enters a prompt at the terminal.

[0274] Input: The prompt text that the user enters (e.g., "Upload an image of a red shirt. We'll search for and display similar products.").

[0275] Output: The prompt text is sent to the server.

[0276] Specific actions: The user opens the terminal application, enters a prompt in the text box, and presses the send button.

[0277] Step 2:

[0278] The server uses a generative AI model to generate an image based on the user's prompt.

[0279] Input: The prompt statement.

[0280] Output: The generated image.

[0281] Specific operation: The server inputs a prompt sentence into the generative AI model, and the model generates an image. The generated image is temporarily saved.

[0282] Step 3:

[0283] The user uploads an image from their device.

[0284] Input: An image file selected by the user.

[0285] Output: The image file is sent to the server.

[0286] Specific behavior: The user opens the file selection dialog on their device, selects the image they want to upload, and presses the upload button.

[0287] Step 4:

[0288] The server analyzes the uploaded image using OpenCV and extracts features.

[0289] Input: The uploaded image file.

[0290] Output: Extracted features.

[0291] Specific operation: The server reads the image using OpenCV and extracts the image features (e.g., color, shape, texture, etc.).

[0292] Step 5:

[0293] The server uses scikit-learn to compare the extracted features with the features of products in the product database and calculate the similarity.

[0294] Input: Extracted features, product database.

[0295] Output: Similarity score for each product.

[0296] Specific operation: The server compares the features of each product in the product database with the features of the uploaded image and calculates the cosine similarity.

[0297] Step 6:

[0298] The server presents highly similar products to the user.

[0299] Input: Similarity score for each product.

[0300] Output: A list of products sorted by most similar.

[0301] Specific operation: The server sorts the products based on the similarity score and sends the top similar products to the user's device.

[0302] Step 7:

[0303] The user checks the presented product and views detailed information.

[0304] Input: Similar products list.

[0305] Output: Detailed product information.

[0306] Specific actions: The user selects a product of interest from the list of similar products displayed on the device and views detailed information.

[0307] Example 3

[0308] Next, a description will be given of a third embodiment of the third embodiment. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0309] In conventional systems, when searching for products based on images entered by users, the system was often limited to physical products, making it difficult to search for non-physical products. Furthermore, if the user did not have a specific image in mind, it was difficult to find the appropriate product. Furthermore, if the image generated based on the user's input did not meet the user's expectations, it was not easy to correct it.

[0310] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[0311] In this invention, the server includes a means for generating a prompt sentence based on user input, a means for sending the generated prompt sentence to a generative AI model to generate a cover and synopsis, and a means for searching for products based on the generated cover and synopsis. This allows the server to generate an appropriate cover and synopsis using the generative AI model even if the user does not have a specific image in mind, and to search for physical and non-physical products based on the generated cover and synopsis. The server also has a function for modifying the prompt sentence based on user input, making it easier to generate an image that meets the user's expectations.

[0312] "User" means an individual or organization that uses the system to input images and search for products.

[0313] A "prompt sentence" is an instruction sentence for a generative AI model that is generated based on an image entered by the user.

[0314] A "generative AI model" is an artificial intelligence model that receives a prompt and generates a cover and synopsis based on it.

[0315] A "cover" is a visual cover image of a novel or other non-physical product generated by a generative AI model.

[0316] A "plot" is a concise description of the contents of a novel or other non-physical product, generated by a generative AI model.

[0317] "Product" means any object that a user wishes to search for and acquire, including physical and non-physical products.

[0318] "Physical goods" are tangible products such as clothing and furniture.

[0319] "Non-physical products" are digital content or services that do not have a physical form, such as novels or songs.

[0320] The "server" is a central processing unit that searches for products based on the generated cover and synopsis and presents them to the user.

[0321] A "terminal" is a device that allows a user to input an image and check the generated cover and synopsis.

[0322] A "database" is an information management system that allows a server to store and search product information.

[0323] MODE FOR CARRYING OUT THE INVENTION

[0324] This invention is a system that generates prompt text based on an image input by a user, generates a cover and synopsis using a generative AI model, and searches for and presents products based on the generated text. Specific embodiments of this system are described below.

[0325] Hardware and software used

[0326] Hardware:

[0327] Device: The device on which the user inputs their image and views the generated cover and synopsis. Examples include a PC, smartphone, or tablet.

[0328] Server: A central processing unit that searches for products based on the generated cover and synopsis and presents them to users. A high-performance server machine is recommended.

[0329] software:

[0330] Generative AI model: An artificial intelligence model that receives a prompt and generates a cover or synopsis based on it. For example, OpenAI's GPT-4 is used.

[0331] Database: An information management system that the server uses to store and retrieve product information. For example, MySQL (registered trademark) is used.

[0332] System processing flow

[0333] The user inputs an image such as "A heartwarming story to read in a quiet forest" into the device's input field. The device receives this image and converts it into a prompt suitable for the generative AI model. For example, it generates a prompt such as "Generate the cover and synopsis of a heartwarming story to read in a quiet forest."

[0334] The device sends the generated prompt to the generative AI model, which analyzes the prompt and uses an internal generation algorithm to generate a cover and synopsis. The generated cover depicts a small wooden house nestled in a lush forest, and the synopsis includes the following: "This story depicts the small joys that a woman living in a quiet forest finds in her daily life."

[0335] The device sends the cover and synopsis received from the generative AI model to the server. The server analyzes the cover and synopsis and uses an SQL query to match them with novel information in the database. Based on the results of the SQL query, the server searches for novels similar to the generated cover and synopsis. For example, if a novel titled "Little House in the Forest" exists in the database and its synopsis is "a story about a woman who lives quietly in the forest and finds small joys in her daily life," this novel will be selected as a search result.

[0336] The server returns search results in JSON format to the device. The search results include the titles, covers, and summaries of similar novels. The device analyzes the search results received from the server and displays them in a user interface. The user can review the search results and click on a novel of interest to view detailed information.

[0337] Examples of concrete examples and prompts

[0338] For example, if a user inputs the image of "a heartwarming story to read in a quiet forest," the following prompt sentences are input to the generative AI model:

[0339] Example prompt sentence:

[0340] "Generate a cover and synopsis for a heartwarming story to read in the quiet of the forest."

[0341] Based on this prompt, the generative AI model generates a cover and synopsis like this:

[0342] Example of a generated cover:

[0343] The cover depicts a small wooden house nestled in a lush forest, with people enjoying reading in front of the house, and the overall image is presented in warm colors.

[0344] An example of a generated synopsis:

[0345] "This story depicts the small joys that a woman living in a quiet forest finds in her daily life. She is comforted by the animals and natural beauty she encounters in the forest, and gradually begins to reexamine herself."

[0346] In this way, even if the user does not have a specific image, the generative AI model can generate an appropriate cover and synopsis, and physical and non-physical products can be searched for based on the generated cover and synopsis. Furthermore, the system has the function of modifying prompts based on user input, making it easier to generate an image that meets the user's expectations. The flow of the identification process in Example 3 will be described with reference to Figure 15.

[0347] Step 1:

[0348] The user inputs an image.

[0349] The user enters an image such as "a heartwarming story to read in a quiet forest" into the input field on the terminal and clicks the send button.

[0350] Input: User image text

[0351] Output: Image text sent to the terminal

[0352] Step 2:

[0353] The terminal converts the input image into a prompt.

[0354] The device takes the user's input and uses its internal text processing capabilities to generate the prompt, "Generate a cover and synopsis for a heartwarming story to read in the quiet forest."

[0355] Input: User image text

[0356] Output: Generated prompt statement

[0357] Step 3:

[0358] The device sends the prompt to the generative AI model.

[0359] The device sends the generated prompt as an API request to the generative AI model, such as OpenAI's GPT-4.

[0360] Input: Generated prompt statement

[0361] Output: API request to the generative AI model

[0362] Step 4:

[0363] A generative AI model generates the cover and synopsis.

[0364] The generative AI model analyzes the prompt and uses its internal generation algorithm to generate a cover and synopsis. The cover depicts a small wooden house nestled in a lush forest, and the synopsis includes the following: "This story depicts the small joys that a woman living in a quiet forest finds in her daily life."

[0365] Input: prompt statement

[0366] Output: Generated cover and synopsis

[0367] Step 5:

[0368] The terminal transmits the generated cover and synopsis to the server.

[0369] The device sends the cover and synopsis received from the generative AI model to the server using an HTTP request.

[0370] Input: Generated cover and synopsis

[0371] Output: HTTP request to the server

[0372] Step 6:

[0373] The server compares the novel information with that in the database.

[0374] The server analyzes the cover and synopsis and uses SQL queries to match them with information about the novel in a database, such as MySQL.

[0375] Input: Generated cover and synopsis

[0376] Output: The result of the SQL query

[0377] Step 7:

[0378] The server searches for similar novels.

[0379] The server then searches for novels with similar covers and synopses based on the results of the SQL query. For example, if a novel titled "Little House in the Woods" exists in the database and its synopsis reads, "A story about a woman who lives quietly in the woods and finds small joys in her daily life," then this novel will be selected as a search result.

[0380] Input: SQL query result

[0381] Output: A list of similar novels

[0382] Step 8:

[0383] The server returns the search results to the device.

[0384] The server returns search results in JSON format to the device, including the titles, covers, and summaries of similar novels.

[0385] Input: List of similar novels

[0386] Output: Search results in JSON format to the terminal

[0387] Step 9:

[0388] The terminal presents the search results to the user.

[0389] The terminal analyzes the search results received from the server and displays them on the user interface. The user can review the search results and click on the novel of interest to view detailed information.

[0390] Input: JSON format search results

[0391] Output: Search results displayed in the user interface

[0392] (Application example 3)

[0393] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0394] Previous systems had limited functionality for searching and presenting related products based on images entered by users, and were particularly inadequate for non-physical products. Even when users had a specific image in mind, it was difficult to find the right product based on that image. Furthermore, the system lacked the functionality to search for similar non-physical products, such as novels, based on the generated image, making it unable to meet user needs.

[0395] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means.

[0396] In this invention, the server includes means for generating an image based on an input from a user, means for searching for products based on the generated image, means for presenting the searched products to the user, means for generating a novel cover and synopsis based on the input image using a generative AI model, means for searching for similar novels based on the generated cover and synopsis, and means for presenting the searched novels to the user. This makes it possible to effectively search for and present physical and non-physical products based on the image input by the user.

[0397] "User input" is information such as text or images provided by system users.

[0398] An "image" is a visual representation generated based on user input.

[0399] A "generative AI model" is an algorithm that uses artificial intelligence techniques to generate content based on user input.

[0400] A "novel cover" is an image that visually represents the contents of a novel.

[0401] A synopsis is a concise summary of the contents of a novel.

[0402] "Product" refers to physical items and non-physical content.

[0403] "Searching" is the act of finding relevant information from a database based on specific criteria.

[0404] "Presenting" is the act of displaying the retrieved information to the user.

[0405] "Similar novels" are novels with similar content and themes based on the generated covers and synopses.

[0406] A system for implementing this invention has the functions of generating an image based on user input, searching for products based on the generated image, and presenting the searched products to the user. It also has the functions of generating a novel cover and synopsis based on the input image using a generative AI model, searching for similar novels based on the generated cover and synopsis, and presenting the searched novels to the user.

[0407] Hardware and software used

[0408] Hardware: Smartphone

[0409] Software: Python, OpenAI API, HTTP request library (requests)

[0410] Data processing and calculation

[0411] 1. User Input: The user inputs a specific image in text form through a smartphone application, for example, a prompt such as "A heartwarming story to read in a quiet forest."

[0412] 2. Use of generative AI model: The server uses the OpenAI API to generate a novel cover and synopsis based on the prompt text entered by the user. The generative AI model analyzes the input text and generates appropriate content based on it.

[0413] 3. Search: Based on the generated cover and synopsis, the server searches for similar novels using an external book search API. The search results include the titles and author information of related novels.

[0414] 4. Displaying Results: The search results are presented to the user, who can then purchase or rent the novels.

[0415] Specific examples

[0416] When a user types "a heartwarming story to read in a quiet forest," the generative AI model generates a novel cover and synopsis based on the prompt. Based on the generated cover and synopsis, the server searches for similar novels and presents the search results to the user. The user can then review the novels and purchase or rent them.

[0417] Prompt Sentence Examples

[0418] "A heartwarming story to read in the quiet forest"

[0419] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[0420] Step 1:

[0421] The user launches an application on their smartphone and inputs a specific image in text format.

[0422] Input: A prompt that the user enters (e.g., "A heartwarming story to read in the quiet woods")

[0423] Output: User input data

[0424] Specific behavior: The user enters a prompt sentence into the application's text input field and presses the submit button.

[0425] Step 2:

[0426] The server receives the user's input data and uses a generative AI model to generate the novel's cover and synopsis.

[0427] Input: User-entered data

[0428] Output: The cover and synopsis of the generated novel

[0429] Specific operation: The server calls the OpenAI API, sends a prompt, and uses the generative AI model to generate a novel cover and synopsis.

[0430] Step 3:

[0431] Based on the cover and synopsis generated by the server, an external book search API is used to search for similar novels.

[0432] Input: Generated novel cover and synopsis

[0433] Output: List of similar novels (title, author information, etc.)

[0434] Specific operation: The server sends the generated cover and synopsis to the book search API to obtain data on similar novels.

[0435] Step 4:

[0436] The server presents the search results to the user.

[0437] Input: List of similar novels

[0438] Output: Search results displayed on the user's smartphone

[0439] Specific operation: The server sends the data of similar novels it has obtained to the user's smartphone, and the application displays it.

[0440] Step 5:

[0441] The user reviews the novels presented and purchases or rents them.

[0442] Input: A novel selected by the user

[0443] Output: Purchase or rental completion notification

[0444] Specific behavior: The user selects a novel of interest from the list of novels presented and presses the purchase or rental button. The server processes the request and sends a completion notification to the user.

[0445] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0446] "Example 1"

[0447] One embodiment of the present invention is a system that combines an emotion engine that recognizes a user's emotions. This system generates images based on user input, searches for products based on the generated images, and presents the searched products to the user. Furthermore, this system recognizes the user's emotions and generates images and searches for products based on the emotions. Specifically, if the user is expressing joy, an image with bright colors is generated and products are searched for based on the image. On the other hand, if the user is expressing sadness, an image with dark colors is generated and products are searched for based on the image.

[0448] "Example 2"

[0449] Another embodiment of the present invention is a system that has a function of modifying an image based on a modification instruction from a user, and further combines it with an emotion engine that recognizes the user's emotions. This system modifies a generated image in response to the modification instruction from the user, searches for products based on the modified image, and presents the searched products to the user. Furthermore, this system recognizes the user's emotions and modifies the image and searches for products based on the emotion. Specifically, if the user is expressing a happy emotion, the system modifies the image to a brighter color tone and searches for products based on the modified image. On the other hand, if the user is expressing a sad emotion, the system modifies the image to a darker color tone and searches for products based on the modified image.

[0450] "Example 3"

[0451] Yet another embodiment of the present invention is a system that searches for products, whether physical or non-physical, and further combines it with an emotion engine that recognizes a user's emotions. The system searches for physical or non-physical products based on generated images and presents the searched products to the user. The system also recognizes the user's emotions and searches for products based on the emotions. Specifically, if the user expresses a happy emotion, the system searches for physical or non-physical products based on images with bright colors. If the user expresses a sad emotion, the system searches for physical or non-physical products based on images with dark colors.

[0452] The processing flow of each embodiment will be described below.

[0453] "Example 1"

[0454] Step 1: Receive input from the user.

[0455] Step 2: Generate an image based on the input received.

[0456] Step 3: Recognize the user's emotions with the emotion engine.

[0457] Step 4: Generate images and search for products based on the recognized emotions.

[0458] Step 5: Search for products based on the generated image.

[0459] Step 6: Present the searched products to the user.

[0460] "Example 2"

[0461] Step 1: Receive user input and correction instructions.

[0462] Step 2: Generate an image based on the received input and modify the image based on the modification instructions.

[0463] Step 3: Recognize the user's emotions with the emotion engine.

[0464] Step 4: Modify images or search for products based on the recognized emotions.

[0465] Step 5: Search for products based on the modified image.

[0466] Step 6: Present the searched products to the user.

[0467] "Example 3"

[0468] Step 1: Receive input from the user.

[0469] Step 2: Generate an image based on the input received.

[0470] Step 3: Recognize the user's emotions with the emotion engine.

[0471] Step 4: Search for products based on the recognized emotion.

[0472] Step 5: Search for physical and non-physical products based on the generated image.

[0473] Step 6: Present the searched products to the user.

[0474] Example 1

[0475] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0476] Conventional image generation systems generate images without considering the user's emotions, making it difficult to provide images that match the user's intentions and emotions. Furthermore, they lack the functionality to reflect user feedback on the generated images, which means it takes a lot of effort for the user to obtain an image that satisfies them. Furthermore, when searching for products based on the generated images, they are unable to provide search results that take the user's emotions into account.

[0477] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0478] In this invention, the server includes means for generating an image based on input from a user, means for searching for products based on the generated image, means for presenting the searched products to the user, means for recognizing the user's emotions and generating an image or searching for products based on the emotions, and means for modifying the image based on feedback from the user. This makes it possible to generate an image that reflects the user's emotions and feedback and to search for products based on the image.

[0479] "User input" refers to information or instructions provided by the user to the system, including keywords such as color, shape, and style.

[0480] "Image" means a visual image generated based on user input and created using a generative AI model.

[0481] A "generative AI model" is an algorithm or program that uses artificial intelligence technology to generate images, for example, using deep learning.

[0482] A "prompt" is a text-based instruction input to a generative AI model, and contains the information necessary to generate an image.

[0483] "Products" are physical or non-physical products or services offered to users, including clothing, furniture, novels, songs, etc.

[0484] "Emotion recognition" is the process of analyzing a user's facial expressions and voice data to determine their emotional state.

[0485] "Feedback" refers to the user's evaluation of the generated image or requests for corrections, which are received and reflected in the system.

[0486] "Revising" is the process of modifying the generated image based on user feedback, again using the generative AI model.

[0487] "Search" is the process of finding related products from a database based on the generated image.

[0488] "Presentation" is the process of displaying searched products and generated images to the user.

[0489] MODE FOR CARRYING OUT THE INVENTION

[0490] This invention is a system that generates an image based on user input, searches for products based on the image, and presents them to the user. It also has the function of recognizing the user's emotions and generating an image or searching for products based on those emotions.

[0491] Hardware and software used

[0492] This system uses the following hardware and software:

[0493] Server: A computer with high-performance computing power that runs generative AI models and manages databases.

[0494] Terminal: A device used by a user, such as a smartphone or PC. It receives input from the user and communicates with the server.

[0495] Generative AI model: An artificial intelligence model for generating images, such as OpenAI's DALL-E or MidJourney (registered trademark).

[0496] Emotion engine: Software that recognizes the user's emotions by analyzing facial expressions and voice data.

[0497] Data processing and calculation

[0498] The server receives input from the user, analyzes it, and converts it into a prompt suitable for the generative AI model. The generative AI model generates an image based on this prompt. The generated image is presented to the user and receives feedback from the user. The server then modifies the image based on this feedback.

[0499] Furthermore, the server uses an emotion engine to recognize the user's emotion and generate images and search for products based on the emotion. For example, if the user expresses joy, it generates an image with bright colors and searches for products based on the image.

[0500] Specific examples

[0501] As a concrete example, consider the case where a user enters the keyword "blue sky and white clouds." The server analyzes this keyword and converts it into a prompt suitable for the generative AI model: "Please generate an image of blue sky and white clouds." The generative AI model generates an image of blue sky and white clouds based on this prompt.

[0502] The generated image is presented to the user via their device, and the user can provide feedback on the image, such as "I'd like more clouds." The server receives this feedback and uses the generative AI model again to generate an image with more clouds.

[0503] Furthermore, if the user smiles while looking at the image, the device's camera captures the facial expression and sends it to the server. The server then uses an emotion engine to recognize that the user is expressing a happy emotion and searches for related products (e.g., brightly colored clothing and accessories) based on the brightly colored image. The searched products are then presented to the user via the device.

[0504] In this way, it becomes possible to generate images that reflect the user's emotions and feedback, and to search for products based on these images.

[0505] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0506] Program processing flow

[0507] Step 1: Receiving User Input

[0508] The user uses a terminal to input keywords such as color, shape, style, etc. to the system, and the terminal sends this input to the server.

[0509] Input: Keywords entered by the user (e.g., "blue sky and white clouds")

[0510] Output: Keywords sent to the server

[0511] Step 2: Keyword analysis

[0512] The server analyzes the received keywords, specifically using natural language processing (NLP) techniques to understand them and convert them into a format suitable for the generative AI model.

[0513] Input: User-submitted keywords

[0514] Output: A prompt suitable for the generative AI model (e.g., "Generate an image of a blue sky with white clouds")

[0515] Step 3: Generate the image

[0516] The server generates an image using a generative AI model based on the analyzed prompt sentence. The generative AI model receives the prompt sentence as input and generates an image based on it.

[0517] Input: A prompt suitable for the generative AI model

[0518] Output: Generated image

[0519] Step 4: Present the image

[0520] The server sends the generated image to the terminal, which displays the image to the user.

[0521] Input: Generated image

[0522] Output: The image presented to the user

[0523] Step 5: Receiving user feedback

[0524] The user can provide feedback on the displayed image, for example by inputting a specific request such as "I'd like more clouds." The device then sends this feedback to the server.

[0525] Input: User feedback

[0526] Output: Feedback sent to the server

[0527] Step 6: Modify the image

[0528] The server receives feedback from the user and again uses the generative AI model to modify the image, which is then sent back to the device and presented to the user.

[0529] Input: User feedback

[0530] Output: The corrected image

[0531] Step 7: Emotion Recognition

[0532] The server recognizes the user's emotions using an emotion engine. The emotion engine analyzes the user's facial expressions and voice data to determine the user's emotional state. For example, if the user expresses joy, the server generates a bright-colored image.

[0533] Input: User facial and voice data

[0534] Output: Perceived emotional state of the user

[0535] Step 8: Product search and presentation

[0536] The server searches for products based on the generated image. For example, based on an image with bright colors, it searches for related products (e.g., bright-colored clothing and accessories). The searched products are presented to the user via the terminal.

[0537] Input: Generated image and perceived user emotional state

[0538] Output: Search results (products) presented to the user

[0539] In this way, a system is realized in which the server, terminals, and users work together to generate image images and search for products.

[0540] (Application example 1)

[0541] Next, a description will be given of Application Example 1 of Embodiment Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0542] On traditional online shopping sites, when users search for a specific product, they simply input keywords to display the search results, which made it difficult to suggest optimal products based on the user's emotions and preferences. Another issue was that if users were not satisfied with the search results, they had to go through the trouble of re-entering the keywords. This could result in a poor user experience and reduced purchasing motivation.

[0543] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0544] In this invention, the server includes means for generating an image based on user input, means for searching for products based on the generated image, means for presenting the searched products to the user, means for recognizing the user's emotions, means for generating or modifying an image based on the recognized emotion, and means for modifying the image and search results based on feedback from the user. This makes it possible to suggest optimal products according to the user's emotions and preferences, which is expected to improve the user experience and increase purchasing motivation.

[0545] "User input" refers to keywords and text information such as color, shape, and style that the user provides to the system.

[0546] An "image" is a visual image generated based on input from a user.

[0547] "Product" means an object offered to a user, including physical products (e.g., clothes, furniture) and non-physical products (e.g., novels, songs).

[0548] "Means for recognizing emotions" refers to technology for analyzing and recognizing emotions from the user's facial expressions, voice, etc.

[0549] "Feedback" refers to the evaluations and opinions that users provide to the system.

[0550] A "generative AI model" is an algorithm or program that uses artificial intelligence techniques such as deep learning to generate image data.

[0551] A "prompt" is an instruction or keyword input to a generative AI model.

[0552] "Modification" refers to techniques for changing and adjusting images and search results based on user feedback and emotional recognition.

[0553] A system for implementing this invention generates an image based on user input, searches for products based on the image, and presents the search results to the user. It also has the function of recognizing the user's emotions, generating or modifying an image based on the emotions, and modifying the image and search results based on user feedback.

[0554] Hardware and software used

[0555] Hardware: Smartphones, smart glasses, servers

[0556] Software: Python, OpenCV, deep learning models (TensorFlow or PyTorch®)

[0557] System configuration

[0558] 1. Obtaining user input: The user uses a smartphone or smart glasses to input keywords such as color, shape, and style.

[0559] 2. Emotion Recognition: The server uses an emotion engine to recognize the user's emotions by analyzing the user's facial expressions using a camera.

[0560] 3. Image generation: The server uses a generative AI model to generate an image based on the user's keywords and the recognized emotions.

[0561] 4. Product search: The server searches for related products using a product search engine based on the generated image.

[0562] 5. Displaying results: The search results are displayed on the user's device, and the user can view the results.

[0563] 6. Getting feedback: Users provide feedback on search results, which is then fed back into the system.

[0564] 7. Revision: The server revises the images and search results based on user feedback.

[0565] Specific examples

[0566] User input: "blue shirt"

[0567] Emotion recognition: The user is expressing the emotion of happiness

[0568] Generated image: Light blue shirt image

[0569] Search results: "Blue shirt - Price: 3000 yen", "Light blue shirt - Price: 2500 yen"

[0570] Feedback: User types, "I want a lighter colored shirt."

[0571] Corrected search results: "Light blue shirt - Price: 2800 yen", "Sky blue shirt - Price: 3200 yen"

[0572] Prompt Sentence Examples

[0573] If a user types in "blue shirt" and indicates the emotion "happy," generate an image of a bright blue shirt and search for products based on that image.

[0574] In this way, a system can be realized that suggests optimal products based on the user's emotions and keywords.

[0575] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0576] Step 1:

[0577] Users use their smartphones or smart glasses to input keywords such as color, shape, and style.

[0578] Input: The keyword the user types in (e.g., "blue shirt")

[0579] Output: The keyword is sent to the server

[0580] Specific behavior: The user enters a keyword into the application's input field and presses the submit button.

[0581] Step 2:

[0582] The server uses an emotion engine to recognize the user's emotions.

[0583] Input: User's facial expression data (camera footage)

[0584] Output: Perceived emotion (e.g., joy)

[0585] How it works: The server analyzes the camera footage and uses a facial expression recognition algorithm to identify the user's emotions.

[0586] Step 3:

[0587] The server uses a generative AI model to generate images based on the user's keywords and perceived emotions.

[0588] Input: Keyword (e.g., "blue shirt"), perceived emotion (e.g., joy)

[0589] Output: The generated image (e.g., an image of a light blue shirt)

[0590] Specific operation: The server inputs a prompt into the generative AI model and generates an image. An example of a prompt is: "If the user inputs 'blue shirt' and expresses the emotion of joy, generate an image of a bright blue shirt."

[0591] Step 4:

[0592] The server uses a product search engine to search for related products based on the generated image.

[0593] Input: Generated image

[0594] Output: Search results (e.g. "Blue shirt - Price: 3000 yen", "Light blue shirt - Price: 2500 yen")

[0595] Specific operation: The server inputs an image into a product search engine and retrieves related product information.

[0596] Step 5:

[0597] The server displays the search results on the user's device.

[0598] Input: Search results

[0599] Output: Product list displayed on the user's device

[0600] Specific operation: The server sends the search results to the user's device, and the application displays the product list.

[0601] Step 6:

[0602] Users provide feedback on search results.

[0603] Input: User feedback (e.g., "I'd like a lighter colored shirt")

[0604] Output: Feedback is sent to the server

[0605] Specific behavior: A user enters their feedback into the feedback field of the application and presses the submit button.

[0606] Step 7:

[0607] The server modifies images and search results based on user feedback.

[0608] Input: User feedback, original image, original search results

[0609] Output: Modified image and search results

[0610] What it does: The server analyzes the feedback, re-uses the generative AI model to modify the image and generate new search results.

[0611] Example 2

[0612] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0613] Conventional systems generate and modify images without considering the user's emotions, and search for products based on those images, making it difficult to provide optimal products that match the user's emotions. Furthermore, the lack of a function to modify images based on the user's emotions makes it difficult to increase user satisfaction.

[0614] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0615] In this invention, the server includes means for generating an image based on an input from a user, means for searching for products based on the generated image, means for presenting the searched products to the user, means for recognizing the user's emotion, means for modifying the image based on the recognized emotion, and means for re-searching for products based on the modified image, thereby making it possible to provide the most suitable product according to the user's emotion.

[0616] "User input" refers to instructions or information provided by a user to a system.

[0617] "Image" means visual image data generated or modified using a generative AI model.

[0618] A "generative AI model" refers to an algorithm or program that uses artificial intelligence techniques to generate new images from text or other input data.

[0619] "Means for searching for products" refers to the technology and process for searching a product database based on an image and finding similar products.

[0620] "Means of presenting products to users" refers to the interface or method for displaying searched product information to users.

[0621] "Means for recognizing user emotions" refers to technologies and systems for analyzing and recognizing emotions from a user's facial expressions, voice, etc.

[0622] "Image modification methods" refers to techniques and processes for changing or modifying existing images based on user sentiment or instructions.

[0623] "Means for re-searching for products" refers to the technology or process for searching the product database again based on the corrected image to find similar products.

[0624] MODE FOR CARRYING OUT THE INVENTION

[0625] This invention is a system that generates an image based on user input, searches for and presents products based on that image, and also has the function of recognizing the user's emotions, modifying the image based on those emotions, and then searching for and presenting products again.

[0626] Hardware and software used

[0627] Device: The device operated by the user (e.g., smartphone, tablet, computer)

[0628] Server: A server that processes data and manages the product database

[0629] Generative AI model: An artificial intelligence model for generating and modifying images (e.g., DALL-E, Stable Diffusion)

[0630] Image recognition technology: Technology for analyzing images and searching product databases (e.g., Google® Cloud Vision API, Amazon® Rekognition)

[0631] Emotion engine: Technology for recognizing user emotions (e.g., Microsoft® Azure® Emotion API)

[0632] Specific operation of the system

[0633] 1. User Input

[0634] The user uses the terminal to instruct the creation or modification of an image.

[0635] For example, a user types into a terminal, "Generate an image of a light-colored shirt."

[0636] 2. Image generation

[0637] The device calls up a generative AI model and generates an image based on the user's instructions.

[0638] The generated image is displayed on the terminal.

[0639] 3. Send image and search for products

[0640] The terminal transmits the generated image to the server.

[0641] The server uses image recognition technology to analyze the image and search the product database.

[0642] The searched product information is sent from the server to the terminal and presented to the user.

[0643] 4. User Emotion Recognition

[0644] The device uses an emotion engine to recognize the user's emotions.

[0645] For example, if a user expresses the emotion of happiness after seeing a search result, the emotion engine will analyze that emotion.

[0646] 5. Emotion-based image correction

[0647] The device uses a generative AI model to modify the image based on the recognized emotion.

[0648] As a specific example, if the user is expressing an emotion of joy, the image is corrected to have a brighter color tone.

[0649] 6. Re-search and presentation after correction

[0650] The corrected image is sent to the server again, and the product database is searched again.

[0651] The server sends the re-searched product information to the terminal and presents it to the user again.

[0652] Specific examples

[0653] Example prompt sentence:

[0654] "Generate an image of a light-colored shirt"

[0655] "Generate an image of a dark shirt"

[0656] "Search for products based on this image"

[0657] This system makes it possible to provide optimal products according to the user's emotions and instructions. To increase user satisfaction, emotion recognition and image correction functions are combined to create a more personalized product search experience.

[0658] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0659] Step 1:

[0660] The user instructs the terminal to generate or modify an image.

[0661] Input: User instructions (e.g., "Generate an image of a light-colored shirt.")

[0662] Output: User's instructions

[0663] Specific operation: The user inputs a prompt sentence into the terminal interface.

[0664] Step 2:

[0665] The device calls the generative AI model and generates an image based on the user's instructions.

[0666] Input: User's instructions

[0667] Output: Generated image

[0668] Specific operation: The device sends a prompt to the generative AI model and receives the generated image.

[0669] Step 3:

[0670] The terminal transmits the generated image to the server.

[0671] Input: Generated image

[0672] Output: Image sent to the server

[0673] Specific operation: The terminal sends the image to the server as an HTTP request.

[0674] Step 4:

[0675] The server uses image recognition technology to analyze the image and search the product database.

[0676] Input: Image sent to the server

[0677] Output: Searched product information

[0678] Specific operation: The server calls the image recognition API and queries the product database based on the analysis results.

[0679] Step 5:

[0680] The server transmits the searched product information to the terminal.

[0681] Input: Searched product information

[0682] Output: Product information sent to the terminal

[0683] Specific operation: The server sends product information in JSON format to the terminal.

[0684] Step 6:

[0685] The terminal displays the received product information to the user.

[0686] Input: Product information sent to the terminal

[0687] Output: Product information displayed to the user

[0688] Specific operation: The terminal analyzes the product information and displays it to the user.

[0689] Step 7:

[0690] The terminal uses an emotion engine to recognize the user's emotions.

[0691] Input: User's facial expressions and voice

[0692] Output: Recognized emotion information

[0693] Specific operation: The device captures the user's facial expressions and voice, sends them to the emotion engine, and receives the analysis results.

[0694] Step 8:

[0695] The device uses a generative AI model to modify the image based on the recognized emotion.

[0696] Input: Recognized emotion information, original image

[0697] Output: Corrected image

[0698] Specific operation: Based on the analysis results of the emotion engine, the device sends a prompt to the generative AI model and receives a modified image.

[0699] Step 9:

[0700] The terminal transmits the corrected image to the server again, and searches the product database again.

[0701] Input: The modified image

[0702] Output: Re-searched product information

[0703] Specific operation: The device sends the corrected image to the server as an HTTP request, and the server sends the re-search results to the device in JSON format.

[0704] Step 10:

[0705] The terminal again displays the search results to the user.

[0706] Input: Re-searched product information

[0707] Output: Search results displayed to the user

[0708] Specific operation: The terminal analyzes the search results and displays them to the user.

[0709] (Application example 2)

[0710] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0711] On conventional online shopping sites, users often rely on text input and category selection when searching for products, making it difficult to operate intuitively. Furthermore, product suggestions do not take user emotions into consideration, leaving a need for an improved user experience. Furthermore, systems that search for products based on user-provided images lack advanced search functionality that combines image correction and emotion recognition.

[0712] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0713] In this invention, the server includes means for generating an image based on an input from a user, means for searching for products based on the generated image, means for presenting the searched products to the user, means for recognizing the user's emotion, means for modifying the image based on the recognized emotion, and means for re-searching for products based on the modified image, thereby enabling intuitive and advanced product searches that take user emotions into consideration.

[0714] "User input" refers to information or instructions provided by a user to a system.

[0715] "Image" means a user-provided or system-generated image.

[0716] "Products" means items available for purchase by Users, including physical and non-physical products.

[0717] "Search" refers to the act of finding information that matches specific criteria within a database.

[0718] "Presenting" refers to the act of displaying search results or information to the user.

[0719] "Emotion recognition" is a technology that determines a user's emotions from their facial expressions, voice, etc.

[0720] "Modification" refers to the act of changing an image based on the user's instructions or feelings.

[0721] "Re-search" refers to the act of searching for a product again based on the corrected image.

[0722] The system for implementing this invention is designed to be operated by a user using a device such as a smartphone. The system generates an image based on user input, searches for products based on that image, and presents the search results to the user. It also has the function of recognizing the user's emotions, modifying the image based on those emotions, and then searching for products again based on the modified image.

[0723] Program processing explanation

[0724] Hardware and Software

[0725] Hardware: Smartphones, servers

[0726] Software: Image recognition models (e.g., models using Keras (registered trademark)), emotion recognition models (e.g., EmotionRecognizer), product database search engine

[0727] Data processing and calculation

[0728] 1. User input: The user uploads images taken or selected with their smartphone to the system.

[0729] 2. Image generation: The server receives the image provided by the user and analyzes it using an image recognition model.

[0730] 3. Emotion Recognition: The server analyzes the user's emotions using an emotion recognition model. For example, it can recognize emotions such as "happiness" or "sadness" from the user's facial expressions and voice.

[0731] 4. Image Modification: Based on the recognized emotion, the server modifies the image. For example, if the user is showing the emotion "happy," it brightens the image.

[0732] 5. Product search: Based on the modified image, a product database search engine is used to search for similar products.

[0733] 6. Presenting search results: Present the searched products to the user.

[0734] Specific examples

[0735] Consider a scenario in which a user uploads a photo of clothing taken with a smartphone and searches for similar clothing on an online shopping site based on that photo. If the user expresses the emotion "joy," the system will brighten the image and search for products based on the corrected image.

[0736] Prompt Sentence Examples

[0737] Users upload photos of their clothes, and the system uses the photos to find similar products and modify the images based on the user's emotions. For example, if the user expresses joy, the system brightens the image and searches for products based on that image.

[0738] In this way, an application for online shopping sites can be created that combines user emotions and image recognition technology.

[0739] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0740] Step 1:

[0741] Users upload images taken with their smartphones.

[0742] Input: An image file taken by the user.

[0743] Output: Image data sent to the server.

[0744] Specific operation: The user uses the camera function of their smartphone to take a photo of a product (e.g., clothing) and uploads the image to the server through the application.

[0745] Step 2:

[0746] The server analyzes the received image and extracts features using an image recognition model.

[0747] Input: Uploaded image data.

[0748] Output: Image feature vector.

[0749] How it works: The server uses an image recognition model using Keras to extract feature vectors from uploaded images, which are then used for subsequent search processing.

[0750] Step 3:

[0751] The server analyzes the user's emotions using an emotion recognition model.

[0752] Input: User facial and voice data (optional).

[0753] Output: User's emotion label (e.g., happy, sad).

[0754] Specific operation: The server uses EmotionRecognizer to recognize emotions from the user's facial expressions and voice data. For example, if the user shows the emotion "joy," that emotion label is output.

[0755] Step 4:

[0756] The server modifies the image based on the recognized emotion.

[0757] Input: Image feature vector, user emotion label.

[0758] Output: The corrected image data.

[0759] What it does: The server adjusts the brightness and color of the image based on the recognized emotion. For example, if the user expresses the emotion "happiness," it brightens the image.

[0760] Step 5:

[0761] The server searches the product database based on the corrected image.

[0762] Input: The corrected image data.

[0763] Output: A list of similar products.

[0764] Specific operations: The server executes a product database search engine using the corrected image data to search for similar products, and generates a list of similar products as search results.

[0765] Step 6:

[0766] The server presents the searched products to the user.

[0767] Input: A list of similar products.

[0768] Output: Product information displayed on the user's smartphone.

[0769] Specific operation: The server sends the list of similar products obtained as search results to the user's smartphone and displays it to the user through the application. The user can then check the presented product information and consider purchasing it.

[0770] Example 3

[0771] Next, a description will be given of a third embodiment of the third embodiment. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0772] Conventional systems were unable to consider user sentiment when searching for and presenting products based on user input, making it difficult to fully meet user needs. Furthermore, there were few systems that could search for and present both physical and non-physical products in a unified manner.

[0773] The identification process by the identification processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means. In this invention, the server includes means for generating an image based on an input from a user, means for searching for products based on the generated image, means for presenting the searched products to the user, means for recognizing the user's emotion, and means for adjusting the search results based on the recognized emotion. This enables product search and presentation taking into account the user's emotion, and enables unified search and presentation of both physical and non-physical products.

[0774] "User input" refers to information such as text and images that a user provides to the system.

[0775] The "means for generating an image" is a function for creating a visual image based on input from a user.

[0776] The "means for searching for products" is a function for searching for related products from a database based on the generated image.

[0777] "Means for presenting products to users" refers to a function for displaying searched products on the user's device.

[0778] "Means for recognizing user emotions" refers to a function that analyzes emotions from the user's facial expressions and input content and obtains the results.

[0779] "Means for adjusting search results based on recognized emotions" refers to a function that optimizes search results based on the user's emotional information and prioritizes the presentation of products that meet the user's needs.

[0780] "Physical goods" are tangible products such as clothing and furniture.

[0781] "Non-physical goods" are digital content and services such as novels and songs.

[0782] The system of the present invention generates an image based on user input, and searches for and presents products based on that image. It also has the ability to recognize user emotions and adjust search results based on those emotions. This system is applicable to both physical and non-physical products.

[0783] Hardware and software used

[0784] Hardware: Servers, user devices (PCs, smartphones, etc.)

[0785] Software: Generative AI models (e.g., GPT-4), emotion recognition engines, database management systems (e.g., MySQL)

[0786] Specific operation of the system

[0787] User image input

[0788] The user uses a terminal to input an image into the system, such as "a heartwarming story to read in a quiet forest." The terminal then sends this input to the server.

[0789] Image analysis and generation

[0790] The server calls a generative AI model (e.g., GPT-4) to analyze the received image. The server inputs a prompt to the generative AI model: "Generate a cover and synopsis of a heartwarming story to read in a quiet forest." The generative AI model generates a cover and synopsis of the novel based on the prompt. Specifically, the generative AI model outputs a cover image of "a scene of sunlight filtering through the trees in a lush forest" and a synopsis of "heartwarming interactions with animals encountered in the forest." The server receives the generated cover image and synopsis and proceeds to the next processing step.

[0791] Search for similar products

[0792] The server connects to a database management system (e.g., MySQL) and searches for similar novels based on the generated cover and synopsis. Specifically, the server searches the database for keywords such as "heartwarming story" and "forest scenery" and lists relevant novels. The server temporarily stores the search results and proceeds to the next processing step.

[0793] Emotion recognition and search result tailoring

[0794] While a user is using a terminal to access the system, the terminal monitors the user's facial expressions and input content and sends them to the emotion recognition engine. The emotion recognition engine analyzes the user's emotions and sends the results to the server. Specifically, if the user shows the emotion of joy, the emotion recognition engine determines this as "joy" and notifies the server. The server adjusts the search results based on the received emotion information. Specifically, the server prioritizes listing novels related to brightly colored images based on the emotion of "joy."

[0795] Presenting search results

[0796] The server generates the final search results and sends them to the user's device. The user can then view the results through the device. Specifically, the user browses the covers and summaries of the listed novels on the device screen and selects the novels that interest them.

[0797] Examples and prompts

[0798] Specific examples

[0799] User Input: "A heartwarming story to read in the quiet of the forest."

[0800] Prompt for generative AI model: "Generate a cover and synopsis for a heartwarming story to read in a quiet forest."

[0801] Produced output:

[0802] Cover: A lush forest with sunlight filtering through the trees

[0803] Story: A heartwarming encounter with animals in the forest

[0804] Specific examples of emotion recognition

[0805] User Emotion: Delight

[0806] Adjust search results: Based on the bright color images, novels with a bright atmosphere are searched and presented preferentially.

[0807] In this way, the system searches for and presents the most suitable product based on the user's image and feelings. The flow of the specification process in the third embodiment will be described with reference to FIG.

[0808] Step 1:

[0809] The user inputs an image into the system using a terminal, such as "a heartwarming story to read in a quiet forest." The terminal sends this input to the server, which receives the input data in text format.

[0810] Step 2:

[0811] The server calls a generative AI model (e.g., GPT-4) to analyze the received image. The server inputs a prompt to the generative AI model: "Generate a cover and synopsis of a heartwarming story to read in a quiet forest." The generative AI model generates a cover and synopsis of the novel based on the prompt. Specifically, the generative AI model outputs a cover image of "a scene of sunlight filtering through the trees in a lush forest" and a synopsis of "heartwarming interactions with animals encountered in the forest." The server receives the generated cover image and synopsis and proceeds to the next processing step.

[0812] Step 3:

[0813] The server connects to a database management system (e.g., MySQL) and searches for similar novels based on the generated cover and synopsis. The input data are the generated cover and synopsis, and the server uses these as keywords to search the database. Specifically, the server searches the database for keywords such as "heartwarming story" and "forest scenery" and lists relevant novels. The server temporarily stores the search results and proceeds to the next processing step.

[0814] Step 4:

[0815] While a user is using a terminal to access the system, the terminal monitors the user's facial expressions and input content and sends them to the emotion recognition engine. The input data is the user's facial expressions and input content, and the terminal sends this to the emotion recognition engine in real time. The emotion recognition engine analyzes the user's emotions and sends the results to the server. Specifically, if the user is showing the emotion of joy, the emotion recognition engine will determine this as "joy" and notify the server.

[0816] Step 5:

[0817] The server adjusts the search results based on the received emotional information. The input data is emotional information from the emotion recognition engine, and the server optimizes the search results based on this. Specifically, the server prioritizes listing novels related to brightly colored images based on the emotion of "joy."

[0818] Step 6:

[0819] The server generates the final search results and sends them to the user's device. The input data is the adjusted search results, which the server sends to the user's device. The user can check the presented search results through the device. Specifically, the user views the covers and summaries of the listed novels on the device screen and selects the novels that interest them.

[0820] (Application example 3)

[0821] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0822] In conventional mail-order systems, it was difficult for users to search for specific products based on their emotions or images. Furthermore, product suggestions did not take the user's emotions into account, which could lead to a decrease in user satisfaction. Furthermore, there was a lack of a system that could search and present physical and non-physical products in a unified manner.

[0823] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means.

[0824] In this invention, the server includes means for generating an image based on an input from a user, means for searching for products based on the generated image, means for presenting the searched products to the user, means for recognizing the user's emotion, and means for searching for products based on the recognized emotion. This enables product search and suggestions based on the user's emotion and image, and makes it possible to search for and present physical and non-physical products in an integrated manner.

[0825] "User input" refers to information such as text and images that a user provides to the system.

[0826] An "image" is a visual representation generated based on user input.

[0827] "Products" include physical goods and non-physical services and content.

[0828] "Emotion" refers to the user's psychological state or mood.

[0829] "Means for recognizing emotions" refers to technology that analyzes and identifies emotions from user input and behavior.

[0830] A "generative AI model" is an algorithm that uses artificial intelligence to generate images or text based on user input.

[0831] A "prompt" is an instruction given to a generative AI model to guide the content that the model generates.

[0832] The system for implementing this invention generates an image based on user input, searches for products based on the image, and presents the searched products to the user. It also has the function of recognizing the user's emotions and searching for products based on those emotions.

[0833] The system includes the following main components:

[0834] 1. User Input Module: Provides an interface for users to input text and images using devices such as smartphones and tablets.

[0835] 2. Image generation module: Generates an image based on user input. It uses a generative AI model to analyze the user input and generate an appropriate image.

[0836] 3. Emotion Recognition Module: Recognizes emotions from user input and behavior. Analyzes the user's psychological state using the emotion engine.

[0837] 4. Product Search Module: Search for products based on the generated images and the recognized emotions. Use a search engine to search for physical and non-physical products in the database.

[0838] 5. Product presentation module: Presents searched products to users, visually displaying search results through the user interface.

[0839] Hardware and software used

[0840] Hardware: smartphones, tablets, servers

[0841] Software: Generative AI models (e.g., OpenAI APIs), emotion engines, search engines, user interfaces

[0842] Specific examples of processing

[0843] A user uses a smartphone to input "candles with relaxing scents." The emotion recognition module detects the emotion of "relaxing" from the user's input. The image generation module uses a generative AI model to generate images of candles with relaxing scents. The product search module searches the database for the most suitable products based on the generated images and the recognized emotions. The product presentation module presents the search results to the user.

[0844] Prompt Sentence Examples

[0845] If a user types "a candle with a relaxing scent" and the emotion engine detects the emotion "relaxed," it will input the following prompt sentence into the generative AI model:

[0846] If the user is expressing the emotion of relaxation, search for the best products based on relaxing scented candles.

[0847] Based on this prompt, the generative AI model searches for the most suitable product and presents it to the user, for example, "lavender-scented candles."

[0848] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[0849] Step 1:

[0850] The user inputs product images using a smartphone.

[0851] Input: User types the text "Relaxing scented candle."

[0852] Output: User input data is sent to the server.

[0853] Specific actions: A user opens an application on their smartphone, types "relaxing scented candles" into the text box, and presses the send button.

[0854] Step 2:

[0855] The server receives the user's input and analyzes the emotion with an emotion recognition module.

[0856] Input: User-supplied data ("A candle with a relaxing scent")

[0857] Output: Recognized emotion data ("Relaxed")

[0858] Specific operation: The server uses the emotion engine to detect the emotion "relaxed" from the user's input text.

[0859] Step 3:

[0860] The server uses a generative AI model to generate an image based on the user's input.

[0861] Input: User-input data ("A candle with a relaxing scent"), Recognized emotion data ("Relaxed")

[0862] Output: Generated image

[0863] Specific operation: The server inputs a prompt into the generative AI model and generates an image. An example of a prompt is: "If the user is expressing the emotion of relaxation, please search for the best products based on candles with relaxing scents."

[0864] Step 4:

[0865] The server searches for products based on the generated images and the recognized emotions.

[0866] Input: Generated image, recognized emotion data ("Relax")

[0867] Output: Searched product data

[0868] What happens: The server uses a search engine to search a database for products related to relaxing scented candles.

[0869] Step 5:

[0870] The server presents the search results to the user.

[0871] Input: Searched product data

[0872] Output: The product list presented to the user

[0873] Specific operation: The server displays the search results on the smartphone screen through the user interface, allowing the user to see a list of candles with relaxing scents.

[0874] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0875] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0876] Another example of generative AI is Gemini (registered trademark) (Internet search engine). <url: https: gemini.google.com ?hl="ja">) are mentioned.

[0877] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0878] [Second embodiment]

[0879] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0880] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0881] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0882] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0883] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0884] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0885] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0886] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0887] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0888] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0889] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0890] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[0891] "Example 1"

[0892] The system of the present invention receives input from a user (e.g., keywords such as color, shape, and style) and generates an image based on that input. This image generation is performed using artificial intelligence techniques such as deep learning. The generated image is presented to the user and can be revised based on user feedback.

[0893] "Example 2"

[0894] The generated and modified image is used to search the product database. This search is performed using image recognition technology to find products similar to the image. The found products are then presented to the user.

[0895] "Example 3"

[0896] The system of the present invention can be applied to not only physical products but also non-physical products. For example, if a user inputs an image such as "a heartwarming story to read in a quiet forest," the system can generate a novel cover and synopsis based on that image and search for and present similar novels.

[0897] The processing flow of each embodiment will be described below.

[0898] "Example 1"

[0899] Step 1: Receive input from the user (e.g. keywords such as color, shape, style, etc.).

[0900] Step 2: Based on the received input, an image is generated using artificial intelligence techniques such as deep learning.

[0901] Step 3: Present the generated image to the user and provide the ability to modify it based on user feedback.

[0902] "Example 2"

[0903] Step 1: Generate and correct the image.

[0904] Step 2: Search the product database based on the acquired image. This search is performed using image recognition technology.

[0905] Step 3: Present the found products to the user as search results.

[0906] "Example 3"

[0907] Step 1: Receive input from the user (e.g., "a heartwarming story to read in a quiet forest").

[0908] Step 2: Generate a cover and synopsis for the novel based on the input received.

[0909] Step 3: Based on the generated cover and synopsis, search the novel database to find similar novels.

[0910] Step 4: Present the found novels to the user.

[0911] Example 1

[0912] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0913] In conventional systems, when generating images based on user input, the generated images often did not meet the user's expectations. Furthermore, when searching for products based on generated images, the system was unable to accurately reflect the user's intentions. Furthermore, the system lacked the ability to modify images based on user feedback, making it difficult to increase user satisfaction.

[0914] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0915] In this invention, the server includes means for generating an image based on user input, means for searching for products based on the generated image, means for presenting the searched products to the user, means for analyzing the user input and generating a prompt sentence for the generative AI model, means for inputting the prompt sentence into the generative AI model and generating an image, and means for modifying the prompt sentence based on user feedback and generating the image again. This makes it possible to generate an image that accurately reflects the user's input, and by further modifying the image to reflect user feedback, it is possible to increase user satisfaction.

[0916] "User input" refers to information that a user provides to the system through a terminal, including keywords such as color, shape, and style.

[0917] An "image" is a visual image generated based on user input and created by a generative AI model.

[0918] A "generative AI model" is an artificial intelligence model that uses deep learning technology to generate image images from prompt text, and examples include Stable Diffusion and DALL-E.

[0919] A "prompt" is a text-based instruction generated based on user input and input into a generative AI model.

[0920] "Feedback" refers to requests for corrections or opinions that the user makes about the generated image, and is reflected when the system regenerates the image.

[0921] "Products" are objects that are searched for based on the generated image, and include physical products (such as clothes and furniture) and non-physical products (such as novels and songs).

[0922] The "server" is a central processing unit that receives input from users, inputs prompts into the generative AI model, and generates and modifies images.

[0923] A "terminal" is a device that allows a user to access the system and provide input or feedback, including PCs and smartphones.

[0924] MODE FOR CARRYING OUT THE INVENTION

[0925] This invention is a system that generates an image based on user input, searches for products based on the image, and presents them to the user. It also has the function of modifying the image based on user feedback.

[0926] Hardware and software used

[0927] The system uses the following hardware and software:

[0928] Server: Acts as the central processing unit, receives input from the user, provides prompts to the generative AI model, and generates and modifies images.

[0929] Terminal: A device that allows users to access the system and provide input and feedback. This includes PCs, smartphones, etc.

[0930] Generative AI model: An artificial intelligence model that uses deep learning techniques to generate images from prompts. Examples include Stable Diffusion and DALL-E.

[0931] Data processing and calculation

[0932] The server receives input from the user and performs data processing and calculations according to the following steps:

[0933] 1. Receiving user input: The user accesses the system interface through a terminal and inputs keywords such as color, shape, style, etc.

[0934] 2. Prompt generation: The server analyzes the keywords sent by the user and generates a prompt suitable for the generative AI model. For example, from the keywords "blue sky, white clouds, landscape," it generates the prompt "Generate an image of a landscape with a blue sky and white clouds."

[0935] 3. Image generation: The server inputs the generated prompt into the generative AI model to generate an image. The generative AI model generates an image based on the prompt using deep learning technology.

[0936] 4. Sending the image: The server sends the generated image to the user's terminal and presents it to the user.

[0937] 5. Receiving Feedback: The user provides feedback on the image presented to them, for example, by inputting specific requests for corrections such as "make the clouds bigger" or "make the sky brighter."

[0938] 6. Prompt Revision: The server receives feedback from the user and regenerates the prompt. For example, in response to the user's feedback, "I want the clouds to be bigger," the server generates a revised prompt, "Generate an image of a landscape with a blue sky and larger white clouds."

[0939] 7. Generate a modified image: The server inputs the modified prompt into the generative AI model to generate a modified image. The modified image is then sent back to the user's device.

[0940] Specific examples

[0941] As a concrete example, consider the case where a user inputs "a landscape with blue sky and white clouds." The user inputs the keywords "blue sky, white clouds, landscape" through the terminal. The server receives this and inputs the following prompt sentence to the generative AI model:

[0942] Example prompt sentence:

[0943] "Generate an image of a landscape with a blue sky and white clouds."

[0944] The generative AI model generates an image based on this prompt, and the server sends the generated image to the user's device. If the user provides feedback such as "I want the clouds to be bigger," the server again inputs the following modified prompt into the generative AI model:

[0945] Example of a modified prompt:

[0946] "Generate an image of a landscape with a blue sky and larger white clouds."

[0947] In this way, the image is modified based on user feedback.

[0948] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0949] Step 1:

[0950] The user enters a keyword through the terminal.

[0951] A user accesses the system's interface using their own device (e.g., a PC or smartphone). On the interface, the user enters keywords such as "blue sky, white clouds, landscape." The entered keywords are sent to the server.

[0952] Input: Keywords entered by the user (e.g., "blue sky, white clouds, landscape")

[0953] Output: Keyword data sent to the server

[0954] Step 2:

[0955] The server receives the user's input and generates a prompt.

[0956] The server receives and analyzes the keywords sent by the user. Based on the analysis results, it generates a prompt sentence appropriate for the generative AI model. For example, the keyword "blue sky, white clouds, landscape" generates the prompt sentence "Generate an image of a landscape with a blue sky and white clouds."

[0957] Input: Keyword data sent by the user

[0958] Output: The generated prompt (e.g., "Generate an image of a landscape with a blue sky and white clouds.")

[0959] Step 3:

[0960] The server inputs a prompt sentence into the generative AI model and generates an image.

[0961] The server inputs the generated prompt into the generative AI model, which uses deep learning technology to generate an image based on the prompt. The generated image is then returned to the server.

[0962] Input: Generated prompt text

[0963] Output: Generated image

[0964] Step 4:

[0965] The server sends the generated image to the user's terminal.

[0966] The server receives the image output from the generative AI model and sends it to the user's device, where the user can view the generated image on their device screen.

[0967] Input: Generated image

[0968] Output: The image sent to the user's device

[0969] Step 5:

[0970] The user provides feedback on the image.

[0971] The user provides feedback on the image presented to them, for example, by inputting specific requests for correction, such as "make the clouds bigger" or "make the sky brighter." The feedback is then sent to the server.

[0972] Input: User feedback (e.g. "I want the clouds to be bigger")

[0973] Output: Feedback data sent to the server

[0974] Step 6:

[0975] The server receives the feedback, regenerates the prompt text, and modifies the image.

[0976] The server receives feedback from the user and regenerates the prompt. For example, in response to feedback such as "I want the clouds to be larger," it generates a revised prompt, "Generate an image of a landscape with a blue sky and larger white clouds." The server then inputs this revised prompt into the generative AI model and generates a revised image. The revised image is then sent back to the user's device.

[0977] Input: User feedback data

[0978] Output: Modified prompt and modified image

[0979] (Application example 1)

[0980] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0981] On traditional online shopping sites, when users searched for a specific product, they had to manually enter keywords and categories, making it difficult to find products that accurately reflected their intent. Furthermore, users had limited options for specifically expressing their desired design and style, making it difficult to find products that satisfied them. Furthermore, the lack of a function to re-search for products based on user feedback made it difficult to improve the user experience.

[0982] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0983] In this invention, the server includes means for generating an image based on input from a user, means for searching for products based on the generated image, means for presenting the searched products to the user, means for modifying the image based on feedback from the user, and means for searching for products again based on the modified image, thereby enabling a product search that accurately reflects the user's intentions and improving the user experience.

[0984] "User input" refers to keywords such as color, shape, and style that users provide to the system.

[0985] An "image" is a visual image generated based on input from a user.

[0986] "Generating means" refers to a device or program that has the function of generating image data using artificial intelligence technology such as deep learning.

[0987] The "means for searching for products" is a device or program that has the function of searching a database for related products based on the generated image.

[0988] "Means for presenting to the user" means a device or program that has the function of displaying searched items to the user in visual or text form.

[0989] "Feedback" refers to opinions and correction instructions provided by users regarding generated images and search results.

[0990] A "modifying means" is a device or program that has the ability to regenerate or modify an image based on user feedback.

[0991] The "means for searching for products again" refers to a device or program having the function of searching the database for related products again based on the corrected image.

[0992] The following system configuration and processing procedure will be described as an embodiment of the present invention.

[0993] System Configuration

[0994] Hardware:

[0995] server

[0996] User devices (smartphones, tablets, PCs, etc.)

[0997] software:

[0998] Deep learning libraries (such as TensorFlow)

[0999] Image processing library (such as PIL)

[1000] Database Management Systems

[1001] User Interface (UI) Applications

[1002] Processing steps

[1003] 1. User Input:

[1004] Users use devices such as smartphones or PCs to input keywords such as color, shape, and style, for example, "red," "dress," and "elegant."

[1005] 2. Image generation:

[1006] The server receives input from the user and generates images using a generative AI model built using deep learning libraries (such as TensorFlow).

[1007] 3. Product Search:

[1008] The server searches the database for related products based on the generated image, using an image processing library (such as PIL).

[1009] 4. User Presentation:

[1010] The searched products are presented to the user through a UI application on the user's device, where the user can review the presented products and decide whether or not to purchase them.

[1011] 5. Receiving Feedback and Making Corrections:

[1012] If the user provides feedback on the presented product or image, the server receives that feedback and modifies the image, again using the generative AI model.

[1013] 6. Search again:

[1014] Based on the revised image, the server retrieves the product again and presents it to the user. This process is repeated until the user is satisfied.

[1015] Specific examples

[1016] For example, if a user enters the keywords "red," "dress," and "elegant," the server converts these keywords into vectors and inputs them into a generative AI model. The generative AI model then generates an image of an elegant red dress based on this. The generated image is displayed on the user's device, and related products are searched for and presented. If the user provides feedback such as "I wish the red was a little brighter," the server receives that feedback, modifies the image, and searches for and presents the products again.

[1017] Example prompt sentence:

[1018] User entered keywords: ["red", "dress", "elegant"]

[1019] Vector to be input to the generative AI model: [0.5, 0.2, 0.8, ...] Keywords converted to vectors

[1020] In this way, product searches can be performed that accurately reflect the user's intent, improving the user experience.

[1021] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1022] Step 1:

[1023] The user uses the terminal to input keywords such as color, shape, style, etc. The input keywords are sent from the terminal to the server.

[1024] Input: Keywords entered by the user (e.g. "red", "dress", "elegant")

[1025] Output: Keyword data sent to the server

[1026] Step 2:

[1027] The server converts the received keywords into vectors using a pre-trained keyword embedding model.

[1028] Input: Keyword data submitted by the user

[1029] Output: Keyword vector (e.g. [0.5, 0.2, 0.8, ...])

[1030] Step 3:

[1031] The server inputs keyword vectors into the generative AI model to generate images, using deep learning libraries (such as TensorFlow).

[1032] Input: Keyword vector

[1033] Output: Generated image

[1034] Step 4:

[1035] The server searches for products based on the generated image in a database, using an image processing library (such as PIL).

[1036] Input: Generated image

[1037] Output: Searched product list

[1038] Step 5:

[1039] The server sends the searched product list to the user's terminal and presents it to the user, who then checks the presented products.

[1040] Input: Searched product list

[1041] Output: Product list displayed on user's device

[1042] Step 6:

[1043] The user provides feedback on the presented product or image, which is then sent from the device to the server.

[1044] Input: User feedback (e.g., "I wish it was a brighter red")

[1045] Output: Feedback data sent to the server

[1046] Step 7:

[1047] The server then modifies the image based on the feedback it receives, again using a generative AI model.

[1048] Input: Feedback data submitted by the user

[1049] Output: The corrected image

[1050] Step 8:

[1051] The server performs a product search again based on the corrected image, and the search results are sent again to the user's device.

[1052] Input: The corrected image

[1053] Output: Re-searched product list

[1054] Step 9:

[1055] The user reviews the presented product again and provides further feedback if necessary, and this process is repeated until the user is satisfied.

[1056] Input: Re-searched product list

[1057] Output: The user's final choice or additional feedback

[1058] Example 2

[1059] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1060] Previous systems had problems with image recognition accuracy and the relevance of search results when searching for products based on images uploaded by users. It was also difficult to quickly and accurately present the products users were looking for. Furthermore, few systems supported searches for both physical and non-physical products, making them unable to meet the diverse needs of users.

[1061] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for generating an image based on an input from a user, means for searching for products based on the generated image, means for presenting the searched products to the user, means for receiving an image uploaded by the user, means for extracting features of the received image, and means for searching a product database based on the extracted features. This makes it possible to recognize an image uploaded by a user with high accuracy and quickly search for and present appropriate products. Furthermore, by targeting both physical and non-physical products for search, it is possible to meet the diverse needs of users.

[1062] "User" means an individual or corporation that uses the system to upload images and search for products.

[1063] "Input" refers to the operations and instructions that users give to the system, and specifically includes uploading images and specifying search criteria.

[1064] "Image" refers to an image file uploaded by a user to the system, which contains visual information that is the basis for product searches.

[1065] A "generating means" is a part of a system that has the ability to create or modify an image based on input from a user.

[1066] The "searching means" is a part of the system that has the function of searching a product database based on the characteristics of the generated image and finding similar products.

[1067] The "presentation means" is a part of the system that has the function of displaying searched product information to the user.

[1068] The "means for receiving" is a part of the system that has the function of allowing the server to receive images uploaded by users.

[1069] The "means for extracting features" is a part of the system that has the function of analyzing features such as color, shape, and pattern from a received image and generating a feature vector.

[1070] "Product database" refers to a database that stores product information that the system searches, including information about physical and non-physical products.

[1071] The "searching means" is a part of the system that has the function of querying the product database based on the extracted feature vector and finding highly similar products.

[1072] MODE FOR CARRYING OUT THE INVENTION

[1073] The present invention is a system for searching for products based on images uploaded by users and presenting search results to the users. A specific embodiment of this system will be described below.

[1074] Hardware and software used

[1075] This system consists of a server and a user's terminal. The server is a computer equipped with a high-performance processor and a large amount of memory, and the user's terminal is a device such as a personal computer or smartphone. The following software is used:

[1076] Image Recognition: TensorFlow

[1077] Image processing: OpenCV

[1078] Database operations: SQLAlchemy

[1079] Web server: Flask

[1080] System processing flow

[1081] Users access the system's web interface from their devices and upload images of the products they want to search for. The server receives these images and uses TensorFlow to perform image recognition. Specifically, it extracts features such as color, shape, and pattern from the image and generates a feature vector.

[1082] Next, the server uses SQLAlchemy to search the product database. Based on the extracted feature vector, it compares it with the feature vectors of each product in the database to find products with high similarity. The search results are sorted in descending order of similarity.

[1083] Finally, the server presents the search results to the user, specifically displaying information such as images, names, prices, and links of similar products on the user's device.

[1084] Specific examples

[1085] For example, consider the case where a user uploads an image of red shoes. The server receives the image and uses TensorFlow to extract the feature "red shoes." Next, the server searches the product database to find similar red shoe product information. Finally, the server presents the found red shoe product information to the user.

[1086] Prompt Sentence Examples

[1087] An example of a prompt to input to a generative AI model is as follows:

[1088] "I uploaded an image of a pair of red shoes. Based on this image, I want you to search our product database and find similar products."

[1089] Using this prompt, the generative AI model can perform product searches according to the user's request.

[1090] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1091] Step 1:

[1092] The user accesses the system's web interface from their terminal and uploads an image of the product they want to search for. Specifically, the user opens a browser and accesses the system's web page. They click the "Upload Image" button on the page and select an image of red shoes from the file selection dialog. After selecting, they click the "Upload" button. The input is the image selected by the user, and the output is an image file sent to the server.

[1093] Step 2:

[1094] The server receives an image uploaded by a user. The server receives the HTTP request and saves the uploaded image file in a temporary folder on the server. After saving, it checks the image format and converts it to PNG format if necessary. The input is the image file sent by the user, and the output is the image file saved on the server.

[1095] Step 3:

[1096] The server inputs the stored image file into the TensorFlow model to extract image features. Specifically, it analyzes the image's color, shape, pattern, etc., and generates a feature vector. The input is the image file stored on the server, and the output is the extracted feature vector.

[1097] Step 4:

[1098] The server uses SQLAlchemy to search the product database based on the extracted feature vector. The server converts the feature vector into a database query and searches the product database. The search results are sorted in descending order of similarity. The input is the extracted feature vector, and the output is a list of highly similar product information.

[1099] Step 5:

[1100] The server generates search results in HTML format and sends them to the user's browser. Specifically, information such as images, names, prices, and links of similar products is displayed on the user's device. The input is a list of highly similar product information, and the output is a search result page displayed in the user's browser.

[1101] (Application example 2)

[1102] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1103] On conventional online shopping sites, it was difficult for users to efficiently find the products they were looking for. In particular, when users did not know the specific product name or wanted to search for a product based on a visual image, the search process was complicated and time-consuming. This led to issues such as a decrease in users' motivation to purchase and a decrease in the frequency of site use.

[1104] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1105] In this invention, the server includes means for generating an image based on input from a user, means for searching for products based on the generated image, means for presenting the searched products to the user, and means for searching a product database based on an image uploaded by the user and displaying similar products, thereby enabling users to efficiently search for products based on visual images and easily find similar products.

[1106] "User input" refers to information or instructions provided by a user to a system.

[1107] An "image" is a visual image generated based on user input.

[1108] A "product database" is a database that stores information about various products.

[1109] "Image recognition technology" refers to the technology in which a computer analyzes images and recognizes specific features and patterns.

[1110] "Similar products" refer to products that have similar features to the image provided by the user.

[1111] A "generative AI model" is a model that uses artificial intelligence to generate new images.

[1112] A "prompt" is an instruction that a user inputs to a generative AI model.

[1113] "Upload" means the act of a user sending data from their device to the system.

[1114] "Searching" is the act of locating information in a database based on specific criteria.

[1115] "Presenting" refers to the act of displaying search results to the user.

[1116] The system for implementing this invention is based on the premise that the user operates it using a terminal such as a smartphone. The system includes the following main components:

[1117] Hardware and Software Configuration

[1118] 1. Terminal: A device such as a smartphone or tablet that is operated by a user.

[1119] 2. Server: A server for running image recognition technology and generative AI models.

[1120] 3. Software:

[1121] Python: Used for image processing and running machine learning models.

[1122] OpenCV: An image processing library.

[1123] scikit-learn: A machine learning library.

[1124] PIL (Python Imaging Library): Used to load and convert images.

[1125] Data processing and calculation

[1126] 1. User input: The user inputs a prompt to generate an image using the terminal. For example, the user can input a prompt such as, "Please upload an image of a red shirt. Similar products will be searched and displayed."

[1127] 2. Image generation: The server uses a generative AI model to generate an image based on the user's prompt.

[1128] 3. Image upload: User uploads an image from their device, which is sent to the server.

[1129] 4. Image recognition and feature extraction: The server uses OpenCV to analyze the uploaded image and extract features.

[1130] 5. Search the product database: The server uses scikit-learn to compare the extracted features with the features of products in the product database and calculate the similarity.

[1131] 6. Show similar products: The server shows similar products to the user. For example, if you upload an image of a red shirt based on the prompt "Please upload an image of a red shirt. We will search and display similar products," similar red shirt products will be displayed.

[1132] Specific examples

[1133] When a user uploads an image of a red shirt, the server analyzes the image and searches for similar red shirts in a product database. The search results show multiple red shirts on the user's device. The user can select a product to view more information or proceed with the purchase.

[1134] In this way, users can efficiently search for products based on visual images and easily find similar products.

[1135] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1136] Step 1:

[1137] The user enters a prompt at the terminal.

[1138] Input: The prompt text that the user enters (e.g., "Upload an image of a red shirt. We'll search for and display similar products.").

[1139] Output: The prompt text is sent to the server.

[1140] Specific actions: The user opens the terminal application, enters a prompt in the text box, and presses the send button.

[1141] Step 2:

[1142] The server uses a generative AI model to generate an image based on the user's prompt.

[1143] Input: The prompt statement.

[1144] Output: The generated image.

[1145] Specific operation: The server inputs a prompt sentence into the generative AI model, and the model generates an image. The generated image is temporarily saved.

[1146] Step 3:

[1147] The user uploads an image from their device.

[1148] Input: An image file selected by the user.

[1149] Output: The image file is sent to the server.

[1150] Specific behavior: The user opens the file selection dialog on their device, selects the image they want to upload, and presses the upload button.

[1151] Step 4:

[1152] The server analyzes the uploaded image using OpenCV and extracts features.

[1153] Input: The uploaded image file.

[1154] Output: Extracted features.

[1155] Specific operation: The server reads the image using OpenCV and extracts the image features (e.g., color, shape, texture, etc.).

[1156] Step 5:

[1157] The server uses scikit-learn to compare the extracted features with the features of products in the product database and calculate the similarity.

[1158] Input: Extracted features, product database.

[1159] Output: Similarity score for each product.

[1160] Specific operation: The server compares the features of each product in the product database with the features of the uploaded image and calculates the cosine similarity.

[1161] Step 6:

[1162] The server presents highly similar products to the user.

[1163] Input: Similarity score for each product.

[1164] Output: A list of products sorted by most similar.

[1165] Specific operation: The server sorts the products based on the similarity score and sends the top similar products to the user's device.

[1166] Step 7:

[1167] The user checks the presented product and views detailed information.

[1168] Input: Similar products list.

[1169] Output: Detailed product information.

[1170] Specific actions: The user selects a product of interest from the list of similar products displayed on the device and views detailed information.

[1171] Example 3

[1172] Next, a description will be given of Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1173] In conventional systems, when searching for products based on images entered by users, the system was often limited to physical products, making it difficult to search for non-physical products. Furthermore, if the user did not have a specific image in mind, it was difficult to find the appropriate product. Furthermore, if the image generated based on the user's input did not meet the user's expectations, it was not easy to correct it.

[1174] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[1175] In this invention, the server includes a means for generating a prompt sentence based on user input, a means for sending the generated prompt sentence to a generative AI model to generate a cover and synopsis, and a means for searching for products based on the generated cover and synopsis. This allows the server to generate an appropriate cover and synopsis using the generative AI model even if the user does not have a specific image in mind, and to search for physical and non-physical products based on the generated cover and synopsis. The server also has a function for modifying the prompt sentence based on user input, making it easier to generate an image that meets the user's expectations.

[1176] "User" means an individual or organization that uses the system to input images and search for products.

[1177] A "prompt sentence" is an instruction sentence for a generative AI model that is generated based on an image entered by the user.

[1178] A "generative AI model" is an artificial intelligence model that receives a prompt and generates a cover and synopsis based on it.

[1179] A "cover" is a visual cover image of a novel or other non-physical product generated by a generative AI model.

[1180] A "plot" is a concise description of the contents of a novel or other non-physical product, generated by a generative AI model.

[1181] "Product" means any object that a user wishes to search for and acquire, including physical and non-physical products.

[1182] "Physical goods" are tangible products such as clothing and furniture.

[1183] "Non-physical products" are digital content or services that do not have a physical form, such as novels or songs.

[1184] The "server" is a central processing unit that searches for products based on the generated cover and synopsis and presents them to the user.

[1185] A "terminal" is a device that allows a user to input an image and check the generated cover and synopsis.

[1186] A "database" is an information management system that allows a server to store and search product information.

[1187] MODE FOR CARRYING OUT THE INVENTION

[1188] This invention is a system that generates prompt text based on an image input by a user, generates a cover and synopsis using a generative AI model, and searches for and presents products based on the generated text. Specific embodiments of this system are described below.

[1189] Hardware and software used

[1190] Hardware:

[1191] Device: The device on which the user inputs their image and views the generated cover and synopsis. Examples include a PC, smartphone, or tablet.

[1192] Server: A central processing unit that searches for products based on the generated cover and synopsis and presents them to users. A high-performance server machine is recommended.

[1193] software:

[1194] Generative AI model: An artificial intelligence model that receives a prompt and generates a cover or synopsis based on it. For example, OpenAI's GPT-4 is used.

[1195] Database: An information management system that allows the server to store and retrieve product information. For example, MySQL is used.

[1196] System processing flow

[1197] The user inputs an image such as "A heartwarming story to read in a quiet forest" into the device's input field. The device receives this image and converts it into a prompt suitable for the generative AI model. For example, it generates a prompt such as "Generate the cover and synopsis of a heartwarming story to read in a quiet forest."

[1198] The device sends the generated prompt to the generative AI model, which analyzes the prompt and uses an internal generation algorithm to generate a cover and synopsis. The generated cover depicts a small wooden house nestled in a lush forest, and the synopsis includes the following: "This story depicts the small joys that a woman living in a quiet forest finds in her daily life."

[1199] The device sends the cover and synopsis received from the generative AI model to the server. The server analyzes the cover and synopsis and uses an SQL query to match them with novel information in the database. Based on the results of the SQL query, the server searches for novels similar to the generated cover and synopsis. For example, if a novel titled "Little House in the Forest" exists in the database and its synopsis is "a story about a woman who lives quietly in the forest and finds small joys in her daily life," this novel will be selected as a search result.

[1200] The server returns search results in JSON format to the device. The search results include the titles, covers, and summaries of similar novels. The device analyzes the search results received from the server and displays them in a user interface. The user can review the search results and click on a novel of interest to view detailed information.

[1201] Examples of concrete examples and prompts

[1202] For example, if a user inputs the image of "a heartwarming story to read in a quiet forest," the following prompt sentences are input to the generative AI model:

[1203] Example prompt sentence:

[1204] "Generate a cover and synopsis for a heartwarming story to read in the quiet of the forest."

[1205] Based on this prompt, the generative AI model generates a cover and synopsis like this:

[1206] Example of a generated cover:

[1207] The cover depicts a small wooden house nestled in a lush forest, with people enjoying reading in front of the house, and the overall image is presented in warm colors.

[1208] An example of a generated synopsis:

[1209] "This story depicts the small joys that a woman living in a quiet forest finds in her daily life. She is comforted by the animals and natural beauty she encounters in the forest, and gradually begins to reexamine herself."

[1210] In this way, even if the user does not have a specific image, the generative AI model can generate an appropriate cover and synopsis, and physical and non-physical products can be searched for based on the generated cover and synopsis. Furthermore, the system has the function of modifying prompts based on user input, making it easier to generate an image that meets the user's expectations. The flow of the identification process in Example 3 will be described with reference to Figure 15.

[1211] Step 1:

[1212] The user inputs an image.

[1213] The user enters an image such as "a heartwarming story to read in a quiet forest" into the input field on the terminal and clicks the send button.

[1214] Input: User image text

[1215] Output: Image text sent to the terminal

[1216] Step 2:

[1217] The terminal converts the input image into a prompt.

[1218] The device takes the user's input and uses its internal text processing capabilities to generate the prompt, "Generate a cover and synopsis for a heartwarming story to read in the quiet forest."

[1219] Input: User image text

[1220] Output: Generated prompt statement

[1221] Step 3:

[1222] The device sends the prompt to the generative AI model.

[1223] The device sends the generated prompt as an API request to the generative AI model, such as OpenAI's GPT-4.

[1224] Input: Generated prompt statement

[1225] Output: API request to the generative AI model

[1226] Step 4:

[1227] A generative AI model generates the cover and synopsis.

[1228] The generative AI model analyzes the prompt and uses its internal generation algorithm to generate a cover and synopsis. The cover depicts a small wooden house nestled in a lush forest, and the synopsis includes the following: "This story depicts the small joys that a woman living in a quiet forest finds in her daily life."

[1229] Input: prompt statement

[1230] Output: Generated cover and synopsis

[1231] Step 5:

[1232] The terminal transmits the generated cover and synopsis to the server.

[1233] The device sends the cover and synopsis received from the generative AI model to the server using an HTTP request.

[1234] Input: Generated cover and synopsis

[1235] Output: HTTP request to the server

[1236] Step 6:

[1237] The server compares the novel information with that in the database.

[1238] The server analyzes the cover and synopsis and uses SQL queries to match them with information about the novel in a database, such as MySQL.

[1239] Input: Generated cover and synopsis

[1240] Output: The result of the SQL query

[1241] Step 7:

[1242] The server searches for similar novels.

[1243] The server then searches for novels with similar covers and synopses based on the results of the SQL query. For example, if a novel titled "Little House in the Woods" exists in the database and its synopsis reads, "A story about a woman who lives quietly in the woods and finds small joys in her daily life," then this novel will be selected as a search result.

[1244] Input: SQL query result

[1245] Output: A list of similar novels

[1246] Step 8:

[1247] The server returns the search results to the device.

[1248] The server returns search results in JSON format to the device, including the titles, covers, and summaries of similar novels.

[1249] Input: List of similar novels

[1250] Output: Search results in JSON format to the terminal

[1251] Step 9:

[1252] The terminal presents the search results to the user.

[1253] The terminal analyzes the search results received from the server and displays them on the user interface. The user can review the search results and click on the novel of interest to view detailed information.

[1254] Input: JSON format search results

[1255] Output: Search results displayed in the user interface

[1256] (Application example 3)

[1257] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1258] Previous systems had limited functionality for searching and presenting related products based on images entered by users, and were particularly inadequate for non-physical products. Even when users had a specific image in mind, it was difficult to find the right product based on that image. Furthermore, the system lacked the functionality to search for similar non-physical products, such as novels, based on the generated image, making it unable to meet user needs.

[1259] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means.

[1260] In this invention, the server includes means for generating an image based on an input from a user, means for searching for products based on the generated image, means for presenting the searched products to the user, means for generating a novel cover and synopsis based on the input image using a generative AI model, means for searching for similar novels based on the generated cover and synopsis, and means for presenting the searched novels to the user. This makes it possible to effectively search for and present physical and non-physical products based on the image input by the user.

[1261] "User input" is information such as text or images provided by system users.

[1262] An "image" is a visual representation generated based on user input.

[1263] A "generative AI model" is an algorithm that uses artificial intelligence techniques to generate content based on user input.

[1264] A "novel cover" is an image that visually represents the contents of a novel.

[1265] A synopsis is a concise summary of the contents of a novel.

[1266] "Product" refers to physical items and non-physical content.

[1267] "Searching" is the act of finding relevant information from a database based on specific criteria.

[1268] "Presenting" is the act of displaying the retrieved information to the user.

[1269] "Similar novels" are novels with similar content and themes based on the generated covers and synopses.

[1270] A system for implementing this invention has the functions of generating an image based on user input, searching for products based on the generated image, and presenting the searched products to the user. It also has the functions of generating a novel cover and synopsis based on the input image using a generative AI model, searching for similar novels based on the generated cover and synopsis, and presenting the searched novels to the user.

[1271] Hardware and software used

[1272] Hardware: Smartphone

[1273] Software: Python, OpenAI API, HTTP request library (requests)

[1274] Data processing and calculation

[1275] 1. User Input: The user inputs a specific image in text form through a smartphone application, for example, a prompt such as "A heartwarming story to read in a quiet forest."

[1276] 2. Use of generative AI model: The server uses the OpenAI API to generate a novel cover and synopsis based on the prompt text entered by the user. The generative AI model analyzes the input text and generates appropriate content based on it.

[1277] 3. Search: Based on the generated cover and synopsis, the server searches for similar novels using an external book search API. The search results include the titles and author information of related novels.

[1278] 4. Displaying Results: The search results are presented to the user, who can then purchase or rent the novels.

[1279] Specific examples

[1280] When a user types "a heartwarming story to read in a quiet forest," the generative AI model generates a novel cover and synopsis based on the prompt. Based on the generated cover and synopsis, the server searches for similar novels and presents the search results to the user. The user can then review the novels and purchase or rent them.

[1281] Prompt Sentence Examples

[1282] "A heartwarming story to read in the quiet forest"

[1283] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[1284] Step 1:

[1285] The user launches an application on their smartphone and inputs a specific image in text format.

[1286] Input: A prompt that the user enters (e.g., "A heartwarming story to read in the quiet woods")

[1287] Output: User input data

[1288] Specific behavior: The user enters a prompt sentence into the application's text input field and presses the submit button.

[1289] Step 2:

[1290] The server receives the user's input data and uses a generative AI model to generate the novel's cover and synopsis.

[1291] Input: User-entered data

[1292] Output: The cover and synopsis of the generated novel

[1293] Specific operation: The server calls the OpenAI API, sends a prompt, and uses the generative AI model to generate a novel cover and synopsis.

[1294] Step 3:

[1295] Based on the cover and synopsis generated by the server, an external book search API is used to search for similar novels.

[1296] Input: Generated novel cover and synopsis

[1297] Output: List of similar novels (title, author information, etc.)

[1298] Specific operation: The server sends the generated cover and synopsis to the book search API to obtain data on similar novels.

[1299] Step 4:

[1300] The server presents the search results to the user.

[1301] Input: List of similar novels

[1302] Output: Search results displayed on the user's smartphone

[1303] Specific operation: The server sends the data of similar novels it has obtained to the user's smartphone, and the application displays it.

[1304] Step 5:

[1305] The user reviews the novels presented and purchases or rents them.

[1306] Input: A novel selected by the user

[1307] Output: Purchase or rental completion notification

[1308] Specific behavior: The user selects a novel of interest from the list of novels presented and presses the purchase or rental button. The server processes the request and sends a completion notification to the user.

[1309] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1310] "Example 1"

[1311] One embodiment of the present invention is a system that combines an emotion engine that recognizes a user's emotions. This system generates images based on user input, searches for products based on the generated images, and presents the searched products to the user. Furthermore, this system recognizes the user's emotions and generates images and searches for products based on the emotions. Specifically, if the user is expressing joy, an image with bright colors is generated and products are searched for based on the image. On the other hand, if the user is expressing sadness, an image with dark colors is generated and products are searched for based on the image.

[1312] "Example 2"

[1313] Another embodiment of the present invention is a system that has a function of modifying an image based on a modification instruction from a user, and further combines it with an emotion engine that recognizes the user's emotions. This system modifies a generated image in response to the modification instruction from the user, searches for products based on the modified image, and presents the searched products to the user. Furthermore, this system recognizes the user's emotions and modifies the image and searches for products based on the emotion. Specifically, if the user is expressing a happy emotion, the system modifies the image to a brighter color tone and searches for products based on the modified image. On the other hand, if the user is expressing a sad emotion, the system modifies the image to a darker color tone and searches for products based on the modified image.

[1314] "Example 3"

[1315] Yet another embodiment of the present invention is a system that searches for products, whether physical or non-physical, and further combines it with an emotion engine that recognizes a user's emotions. The system searches for physical or non-physical products based on generated images and presents the searched products to the user. The system also recognizes the user's emotions and searches for products based on the emotions. Specifically, if the user expresses a happy emotion, the system searches for physical or non-physical products based on images with bright colors. If the user expresses a sad emotion, the system searches for physical or non-physical products based on images with dark colors.

[1316] The processing flow of each embodiment will be described below.

[1317] "Example 1"

[1318] Step 1: Receive input from the user.

[1319] Step 2: Generate an image based on the input received.

[1320] Step 3: Recognize the user's emotions with the emotion engine.

[1321] Step 4: Generate images and search for products based on the recognized emotions.

[1322] Step 5: Search for products based on the generated image.

[1323] Step 6: Present the searched products to the user.

[1324] "Example 2"

[1325] Step 1: Receive user input and correction instructions.

[1326] Step 2: Generate an image based on the received input and modify the image based on the modification instructions.

[1327] Step 3: Recognize the user's emotions with the emotion engine.

[1328] Step 4: Modify images or search for products based on the recognized emotions.

[1329] Step 5: Search for products based on the modified image.

[1330] Step 6: Present the searched products to the user.

[1331] "Example 3"

[1332] Step 1: Receive input from the user.

[1333] Step 2: Generate an image based on the input received.

[1334] Step 3: Recognize the user's emotions with the emotion engine.

[1335] Step 4: Search for products based on the recognized emotion.

[1336] Step 5: Search for physical and non-physical products based on the generated image.

[1337] Step 6: Present the searched products to the user.

[1338] Example 1

[1339] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1340] Conventional image generation systems generate images without considering the user's emotions, making it difficult to provide images that match the user's intentions and emotions. Furthermore, they lack the functionality to reflect user feedback on the generated images, which means it takes a lot of effort for the user to obtain an image that satisfies them. Furthermore, when searching for products based on the generated images, they are unable to provide search results that take the user's emotions into account.

[1341] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1342] In this invention, the server includes means for generating an image based on input from a user, means for searching for products based on the generated image, means for presenting the searched products to the user, means for recognizing the user's emotions and generating an image or searching for products based on the emotions, and means for modifying the image based on feedback from the user. This makes it possible to generate an image that reflects the user's emotions and feedback and to search for products based on the image.

[1343] "User input" refers to information or instructions provided by the user to the system, including keywords such as color, shape, and style.

[1344] "Image" means a visual image generated based on user input and created using a generative AI model.

[1345] A "generative AI model" is an algorithm or program that uses artificial intelligence technology to generate images, for example, using deep learning.

[1346] A "prompt" is a text-based instruction input to a generative AI model, and contains the information necessary to generate an image.

[1347] "Products" are physical or non-physical products or services offered to users, including clothing, furniture, novels, songs, etc.

[1348] "Emotion recognition" is the process of analyzing a user's facial expressions and voice data to determine their emotional state.

[1349] "Feedback" refers to the user's evaluation of the generated image or requests for corrections, which are received and reflected in the system.

[1350] "Revising" is the process of modifying the generated image based on user feedback, again using the generative AI model.

[1351] "Search" is the process of finding related products from a database based on the generated image.

[1352] "Presentation" is the process of displaying searched products and generated images to the user.

[1353] MODE FOR CARRYING OUT THE INVENTION

[1354] This invention is a system that generates an image based on user input, searches for products based on the image, and presents them to the user. It also has the function of recognizing the user's emotions and generating an image or searching for products based on those emotions.

[1355] Hardware and software used

[1356] This system uses the following hardware and software:

[1357] Server: A computer with high-performance computing power that runs generative AI models and manages databases.

[1358] Terminal: A device used by a user, such as a smartphone or PC. It receives input from the user and communicates with the server.

[1359] Generative AI model: An artificial intelligence model for generating images, such as OpenAI's DALL-E or MidJourney.

[1360] Emotion engine: Software that recognizes the user's emotions by analyzing facial expressions and voice data.

[1361] Data processing and calculation

[1362] The server receives input from the user, analyzes it, and converts it into a prompt suitable for the generative AI model. The generative AI model generates an image based on this prompt. The generated image is presented to the user and receives feedback from the user. The server then modifies the image based on this feedback.

[1363] Furthermore, the server uses an emotion engine to recognize the user's emotion and generate images and search for products based on the emotion. For example, if the user expresses joy, it generates an image with bright colors and searches for products based on the image.

[1364] Specific examples

[1365] As a concrete example, consider the case where a user enters the keyword "blue sky and white clouds." The server analyzes this keyword and converts it into a prompt suitable for the generative AI model: "Please generate an image of blue sky and white clouds." The generative AI model generates an image of blue sky and white clouds based on this prompt.

[1366] The generated image is presented to the user via their device, and the user can provide feedback on the image, such as "I'd like more clouds." The server receives this feedback and uses the generative AI model again to generate an image with more clouds.

[1367] Furthermore, if the user smiles while looking at the image, the device's camera captures the facial expression and sends it to the server. The server then uses an emotion engine to recognize that the user is expressing a happy emotion and searches for related products (e.g., brightly colored clothing and accessories) based on the brightly colored image. The searched products are then presented to the user via the device.

[1368] In this way, it becomes possible to generate images that reflect the user's emotions and feedback, and to search for products based on these images.

[1369] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1370] Program processing flow

[1371] Step 1: Receiving User Input

[1372] The user uses a terminal to input keywords such as color, shape, style, etc. to the system, and the terminal sends this input to the server.

[1373] Input: Keywords entered by the user (e.g., "blue sky and white clouds")

[1374] Output: Keywords sent to the server

[1375] Step 2: Keyword analysis

[1376] The server analyzes the received keywords, specifically using natural language processing (NLP) techniques to understand them and convert them into a format suitable for the generative AI model.

[1377] Input: User-submitted keywords

[1378] Output: A prompt suitable for the generative AI model (e.g., "Generate an image of a blue sky with white clouds")

[1379] Step 3: Generate the image

[1380] The server generates an image using a generative AI model based on the analyzed prompt sentence. The generative AI model receives the prompt sentence as input and generates an image based on it.

[1381] Input: A prompt suitable for the generative AI model

[1382] Output: Generated image

[1383] Step 4: Present the image

[1384] The server sends the generated image to the terminal, which displays the image to the user.

[1385] Input: Generated image

[1386] Output: The image presented to the user

[1387] Step 5: Receiving user feedback

[1388] The user can provide feedback on the displayed image, for example by inputting a specific request such as "I'd like more clouds." The device then sends this feedback to the server.

[1389] Input: User feedback

[1390] Output: Feedback sent to the server

[1391] Step 6: Modify the image

[1392] The server receives feedback from the user and again uses the generative AI model to modify the image, which is then sent back to the device and presented to the user.

[1393] Input: User feedback

[1394] Output: The corrected image

[1395] Step 7: Emotion Recognition

[1396] The server recognizes the user's emotions using an emotion engine. The emotion engine analyzes the user's facial expressions and voice data to determine the user's emotional state. For example, if the user expresses joy, the server generates a bright-colored image.

[1397] Input: User facial and voice data

[1398] Output: Perceived emotional state of the user

[1399] Step 8: Product search and presentation

[1400] The server searches for products based on the generated image. For example, based on an image with bright colors, it searches for related products (e.g., bright-colored clothing and accessories). The searched products are presented to the user via the terminal.

[1401] Input: Generated image and perceived user emotional state

[1402] Output: Search results (products) presented to the user

[1403] In this way, a system is realized in which the server, terminals, and users work together to generate image images and search for products.

[1404] (Application example 1)

[1405] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1406] On traditional online shopping sites, when users search for a specific product, they simply input keywords to display the search results, which made it difficult to suggest optimal products based on the user's emotions and preferences. Another issue was that if users were not satisfied with the search results, they had to go through the trouble of re-entering the keywords. This could result in a poor user experience and reduced purchasing motivation.

[1407] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1408] In this invention, the server includes means for generating an image based on user input, means for searching for products based on the generated image, means for presenting the searched products to the user, means for recognizing the user's emotions, means for generating or modifying an image based on the recognized emotion, and means for modifying the image and search results based on feedback from the user. This makes it possible to suggest optimal products according to the user's emotions and preferences, which is expected to improve the user experience and increase purchasing motivation.

[1409] "User input" refers to keywords and text information such as color, shape, and style that the user provides to the system.

[1410] An "image" is a visual image generated based on input from a user.

[1411] "Product" means an object offered to a user, including physical products (e.g., clothes, furniture) and non-physical products (e.g., novels, songs).

[1412] "Means for recognizing emotions" refers to technology for analyzing and recognizing emotions from the user's facial expressions, voice, etc.

[1413] "Feedback" refers to the evaluations and opinions that users provide to the system.

[1414] A "generative AI model" is an algorithm or program that uses artificial intelligence techniques such as deep learning to generate image data.

[1415] A "prompt" is an instruction or keyword input to a generative AI model.

[1416] "Modification" refers to techniques for changing and adjusting images and search results based on user feedback and emotional recognition.

[1417] A system for implementing this invention generates an image based on user input, searches for products based on the image, and presents the search results to the user. It also has the function of recognizing the user's emotions, generating or modifying an image based on the emotions, and modifying the image and search results based on user feedback.

[1418] Hardware and software used

[1419] Hardware: Smartphones, smart glasses, servers

[1420] Software: Python, OpenCV, deep learning model (TensorFlow or PyTorch)

[1421] System configuration

[1422] 1. Obtaining user input: The user uses a smartphone or smart glasses to input keywords such as color, shape, and style.

[1423] 2. Emotion Recognition: The server uses an emotion engine to recognize the user's emotions by analyzing the user's facial expressions using a camera.

[1424] 3. Image generation: The server uses a generative AI model to generate an image based on the user's keywords and the recognized emotions.

[1425] 4. Product search: The server searches for related products using a product search engine based on the generated image.

[1426] 5. Displaying results: The search results are displayed on the user's device, and the user can view the results.

[1427] 6. Getting feedback: Users provide feedback on search results, which is then fed back into the system.

[1428] 7. Revision: The server revises the images and search results based on user feedback.

[1429] Specific examples

[1430] User input: "blue shirt"

[1431] Emotion recognition: The user is expressing the emotion of happiness

[1432] Generated image: Light blue shirt image

[1433] Search results: "Blue shirt - Price: 3000 yen", "Light blue shirt - Price: 2500 yen"

[1434] Feedback: User types, "I want a lighter colored shirt."

[1435] Corrected search results: "Light blue shirt - Price: 2800 yen", "Sky blue shirt - Price: 3200 yen"

[1436] Prompt Sentence Examples

[1437] If a user types in "blue shirt" and indicates the emotion "happy," generate an image of a bright blue shirt and search for products based on that image.

[1438] In this way, a system can be realized that suggests optimal products based on the user's emotions and keywords.

[1439] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1440] Step 1:

[1441] Users use their smartphones or smart glasses to input keywords such as color, shape, and style.

[1442] Input: The keyword the user types in (e.g., "blue shirt")

[1443] Output: The keyword is sent to the server

[1444] Specific behavior: The user enters a keyword into the application's input field and presses the submit button.

[1445] Step 2:

[1446] The server uses an emotion engine to recognize the user's emotions.

[1447] Input: User's facial expression data (camera footage)

[1448] Output: Perceived emotion (e.g., joy)

[1449] How it works: The server analyzes the camera footage and uses a facial expression recognition algorithm to identify the user's emotions.

[1450] Step 3:

[1451] The server uses a generative AI model to generate images based on the user's keywords and perceived emotions.

[1452] Input: Keyword (e.g., "blue shirt"), perceived emotion (e.g., joy)

[1453] Output: The generated image (e.g., an image of a light blue shirt)

[1454] Specific operation: The server inputs a prompt into the generative AI model and generates an image. An example of a prompt is: "If the user inputs 'blue shirt' and shows the emotion of joy, generate an image of a bright blue shirt."

[1455] Step 4:

[1456] The server uses a product search engine to search for related products based on the generated image.

[1457] Input: Generated image

[1458] Output: Search results (e.g. "Blue shirt - Price: 3000 yen", "Light blue shirt - Price: 2500 yen")

[1459] Specific operation: The server inputs an image into a product search engine and retrieves related product information.

[1460] Step 5:

[1461] The server displays the search results on the user's device.

[1462] Input: Search results

[1463] Output: Product list displayed on the user's device

[1464] Specific operation: The server sends the search results to the user's device, and the application displays the product list.

[1465] Step 6:

[1466] Users provide feedback on search results.

[1467] Input: User feedback (e.g., "I'd like a lighter colored shirt")

[1468] Output: Feedback is sent to the server

[1469] Specific behavior: A user enters their feedback into the feedback field of the application and presses the submit button.

[1470] Step 7:

[1471] The server modifies images and search results based on user feedback.

[1472] Input: User feedback, original image, original search results

[1473] Output: Modified image and search results

[1474] What it does: The server analyzes the feedback, re-uses the generative AI model to modify the image and generate new search results.

[1475] Example 2

[1476] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1477] Conventional systems generate and modify images without considering the user's emotions, and search for products based on those images, making it difficult to provide optimal products that match the user's emotions. Furthermore, the lack of a function to modify images based on the user's emotions makes it difficult to increase user satisfaction.

[1478] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1479] In this invention, the server includes means for generating an image based on an input from a user, means for searching for products based on the generated image, means for presenting the searched products to the user, means for recognizing the user's emotion, means for modifying the image based on the recognized emotion, and means for re-searching for products based on the modified image, thereby making it possible to provide the most suitable product according to the user's emotion.

[1480] "User input" refers to instructions or information provided by a user to a system.

[1481] "Image" means visual image data generated or modified using a generative AI model.

[1482] A "generative AI model" refers to an algorithm or program that uses artificial intelligence techniques to generate new images from text or other input data.

[1483] "Means for searching for products" refers to the technology and process for searching a product database based on an image and finding similar products.

[1484] "Means of presenting products to users" refers to the interface or method for displaying searched product information to users.

[1485] "Means for recognizing user emotions" refers to technologies and systems for analyzing and recognizing emotions from a user's facial expressions, voice, etc.

[1486] "Image modification methods" refers to techniques and processes for changing or modifying existing images based on user sentiment or instructions.

[1487] "Means for re-searching for products" refers to the technology or process for searching the product database again based on the corrected image to find similar products.

[1488] MODE FOR CARRYING OUT THE INVENTION

[1489] This invention is a system that generates an image based on user input, searches for and presents products based on that image, and also has the function of recognizing the user's emotions, modifying the image based on those emotions, and then searching for and presenting products again.

[1490] Hardware and software used

[1491] Device: The device operated by the user (e.g., smartphone, tablet, computer)

[1492] Server: A server that processes data and manages the product database

[1493] Generative AI model: An artificial intelligence model for generating and modifying images (e.g., DALL-E, Stable Diffusion)

[1494] Image recognition technology: Technology for analyzing images and searching product databases (e.g., Google Cloud Vision API, Amazon Rekognition)

[1495] Emotion engine: Technology for recognizing user emotions (e.g., Microsoft Azure Emotion API)

[1496] Specific operation of the system

[1497] 1. User Input

[1498] The user uses the terminal to instruct the creation or modification of an image.

[1499] For example, a user types into a terminal, "Generate an image of a light-colored shirt."

[1500] 2. Image generation

[1501] The device calls up a generative AI model and generates an image based on the user's instructions.

[1502] The generated image is displayed on the terminal.

[1503] 3. Send image and search for products

[1504] The terminal transmits the generated image to the server.

[1505] The server uses image recognition technology to analyze the image and search the product database.

[1506] The searched product information is sent from the server to the terminal and presented to the user.

[1507] 4. User Emotion Recognition

[1508] The device uses an emotion engine to recognize the user's emotions.

[1509] For example, if a user expresses the emotion of happiness after seeing a search result, the emotion engine will analyze that emotion.

[1510] 5. Emotion-based image correction

[1511] The device uses a generative AI model to modify the image based on the recognized emotion.

[1512] As a specific example, if the user is expressing an emotion of joy, the image is corrected to have a brighter color tone.

[1513] 6. Re-search and presentation after correction

[1514] The corrected image is sent to the server again, and the product database is searched again.

[1515] The server sends the re-searched product information to the terminal and presents it to the user again.

[1516] Specific examples

[1517] Example prompt sentence:

[1518] "Generate an image of a light-colored shirt"

[1519] "Generate an image of a dark shirt"

[1520] "Search for products based on this image"

[1521] This system makes it possible to provide optimal products according to the user's emotions and instructions. To increase user satisfaction, emotion recognition and image correction functions are combined to create a more personalized product search experience.

[1522] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1523] Step 1:

[1524] The user instructs the terminal to generate or modify an image.

[1525] Input: User instructions (e.g., "Generate an image of a light-colored shirt.")

[1526] Output: User's instructions

[1527] Specific operation: The user inputs a prompt sentence into the terminal interface.

[1528] Step 2:

[1529] The device calls the generative AI model and generates an image based on the user's instructions.

[1530] Input: User's instructions

[1531] Output: Generated image

[1532] Specific operation: The device sends a prompt to the generative AI model and receives the generated image.

[1533] Step 3:

[1534] The terminal transmits the generated image to the server.

[1535] Input: Generated image

[1536] Output: Image sent to the server

[1537] Specific operation: The terminal sends the image to the server as an HTTP request.

[1538] Step 4:

[1539] The server uses image recognition technology to analyze the image and search the product database.

[1540] Input: Image sent to the server

[1541] Output: Searched product information

[1542] Specific operation: The server calls the image recognition API and queries the product database based on the analysis results.

[1543] Step 5:

[1544] The server transmits the searched product information to the terminal.

[1545] Input: Searched product information

[1546] Output: Product information sent to the terminal

[1547] Specific operation: The server sends product information in JSON format to the terminal.

[1548] Step 6:

[1549] The terminal displays the received product information to the user.

[1550] Input: Product information sent to the terminal

[1551] Output: Product information displayed to the user

[1552] Specific operation: The terminal analyzes the product information and displays it to the user.

[1553] Step 7:

[1554] The terminal uses an emotion engine to recognize the user's emotions.

[1555] Input: User's facial expressions and voice

[1556] Output: Recognized emotion information

[1557] Specific operation: The device captures the user's facial expressions and voice, sends them to the emotion engine, and receives the analysis results.

[1558] Step 8:

[1559] The device uses a generative AI model to modify the image based on the recognized emotion.

[1560] Input: Recognized emotion information, original image

[1561] Output: Corrected image

[1562] Specific operation: Based on the analysis results of the emotion engine, the device sends a prompt to the generative AI model and receives a modified image.

[1563] Step 9:

[1564] The terminal transmits the corrected image to the server again, and searches the product database again.

[1565] Input: The modified image

[1566] Output: Re-searched product information

[1567] Specific operation: The device sends the corrected image to the server as an HTTP request, and the server sends the re-search results to the device in JSON format.

[1568] Step 10:

[1569] The terminal again displays the search results to the user.

[1570] Input: Re-searched product information

[1571] Output: Search results displayed to the user

[1572] Specific operation: The terminal analyzes the search results and displays them to the user.

[1573] (Application example 2)

[1574] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1575] On conventional online shopping sites, users often rely on text input and category selection when searching for products, making it difficult to operate intuitively. Furthermore, product suggestions do not take user emotions into consideration, leaving a need for an improved user experience. Furthermore, systems that search for products based on user-provided images lack advanced search functionality that combines image correction and emotion recognition.

[1576] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1577] In this invention, the server includes means for generating an image based on an input from a user, means for searching for products based on the generated image, means for presenting the searched products to the user, means for recognizing the user's emotion, means for modifying the image based on the recognized emotion, and means for re-searching for products based on the modified image, thereby enabling intuitive and advanced product searches that take user emotions into consideration.

[1578] "User input" refers to information or instructions provided by a user to a system.

[1579] "Image" means a user-provided or system-generated image.

[1580] "Products" means items available for purchase by Users, including physical and non-physical products.

[1581] "Search" refers to the act of finding information that matches specific criteria within a database.

[1582] "Presenting" refers to the act of displaying search results or information to the user.

[1583] "Emotion recognition" is a technology that determines a user's emotions from their facial expressions, voice, etc.

[1584] "Modification" refers to the act of changing an image based on the user's instructions or feelings.

[1585] "Re-search" refers to the act of searching for a product again based on the corrected image.

[1586] The system for implementing this invention is designed to be operated by a user using a device such as a smartphone. The system generates an image based on user input, searches for products based on that image, and presents the search results to the user. It also has the function of recognizing the user's emotions, modifying the image based on those emotions, and then searching for products again based on the modified image.

[1587] Program processing explanation

[1588] Hardware and Software

[1589] Hardware: Smartphones, servers

[1590] Software: Image recognition models (e.g., models using Keras), emotion recognition models (e.g., EmotionRecognizer), product database search engine

[1591] Data processing and calculation

[1592] 1. User input: The user uploads images taken or selected with their smartphone to the system.

[1593] 2. Image generation: The server receives the image provided by the user and analyzes it using an image recognition model.

[1594] 3. Emotion Recognition: The server analyzes the user's emotions using an emotion recognition model. For example, it can recognize emotions such as "happiness" or "sadness" from the user's facial expressions and voice.

[1595] 4. Image Modification: Based on the recognized emotion, the server modifies the image. For example, if the user is showing the emotion "happy," it brightens the image.

[1596] 5. Product search: Based on the modified image, a product database search engine is used to search for similar products.

[1597] 6. Presenting search results: Present the searched products to the user.

[1598] Specific examples

[1599] Consider a scenario in which a user uploads a photo of clothing taken with a smartphone and searches for similar clothing on an online shopping site based on that photo. If the user expresses the emotion "joy," the system will brighten the image and search for products based on the corrected image.

[1600] Prompt Sentence Examples

[1601] Users upload photos of their clothes, and the system uses the photos to find similar products and modify the images based on the user's emotions. For example, if the user expresses joy, the system brightens the image and searches for products based on that image.

[1602] In this way, an application for online shopping sites can be created that combines user emotions and image recognition technology.

[1603] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1604] Step 1:

[1605] Users upload images taken with their smartphones.

[1606] Input: An image file taken by the user.

[1607] Output: Image data sent to the server.

[1608] Specific operation: The user uses the camera function of their smartphone to take a photo of a product (e.g., clothing) and uploads the image to the server through the application.

[1609] Step 2:

[1610] The server analyzes the received image and extracts features using an image recognition model.

[1611] Input: Uploaded image data.

[1612] Output: Image feature vector.

[1613] How it works: The server uses an image recognition model using Keras to extract feature vectors from uploaded images, which are then used for subsequent search processing.

[1614] Step 3:

[1615] The server analyzes the user's emotions using an emotion recognition model.

[1616] Input: User facial and voice data (optional).

[1617] Output: User's emotion label (e.g., happy, sad).

[1618] Specific operation: The server uses EmotionRecognizer to recognize emotions from the user's facial expressions and voice data. For example, if the user shows the emotion "joy," that emotion label is output.

[1619] Step 4:

[1620] The server modifies the image based on the recognized emotion.

[1621] Input: Image feature vector, user emotion label.

[1622] Output: The corrected image data.

[1623] What it does: The server adjusts the brightness and color of the image based on the recognized emotion. For example, if the user expresses the emotion "happiness," it brightens the image.

[1624] Step 5:

[1625] The server searches the product database based on the corrected image.

[1626] Input: The corrected image data.

[1627] Output: A list of similar products.

[1628] Specific operations: The server executes a product database search engine using the corrected image data to search for similar products, and generates a list of similar products as search results.

[1629] Step 6:

[1630] The server presents the searched products to the user.

[1631] Input: A list of similar products.

[1632] Output: Product information displayed on the user's smartphone.

[1633] Specific operation: The server sends the list of similar products obtained as search results to the user's smartphone and displays it to the user through the application. The user can then check the presented product information and consider purchasing it.

[1634] Example 3

[1635] Next, a description will be given of Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1636] Conventional systems were unable to consider user sentiment when searching for and presenting products based on user input, making it difficult to fully meet user needs. Furthermore, there were few systems that could search for and present both physical and non-physical products in a unified manner.

[1637] The identification process by the identification processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means. In this invention, the server includes means for generating an image based on an input from a user, means for searching for products based on the generated image, means for presenting the searched products to the user, means for recognizing the user's emotion, and means for adjusting the search results based on the recognized emotion. This enables product search and presentation taking into account the user's emotion, and enables unified search and presentation of both physical and non-physical products.

[1638] "User input" refers to information such as text and images that a user provides to the system.

[1639] The "means for generating an image" is a function for creating a visual image based on input from a user.

[1640] The "means for searching for products" is a function for searching for related products from a database based on the generated image.

[1641] "Means for presenting products to users" refers to a function for displaying searched products on the user's device.

[1642] "Means for recognizing user emotions" refers to a function that analyzes emotions from the user's facial expressions and input content and obtains the results.

[1643] "Means for adjusting search results based on recognized emotions" refers to a function that optimizes search results based on the user's emotional information and prioritizes the presentation of products that meet the user's needs.

[1644] "Physical goods" are tangible products such as clothing and furniture.

[1645] "Non-physical goods" are digital content and services such as novels and songs.

[1646] The system of the present invention generates an image based on user input, and searches for and presents products based on that image. It also has the ability to recognize user emotions and adjust search results based on those emotions. This system is applicable to both physical and non-physical products.

[1647] Hardware and software used

[1648] Hardware: Servers, user devices (PCs, smartphones, etc.)

[1649] Software: Generative AI models (e.g., GPT-4), emotion recognition engines, database management systems (e.g., MySQL)

[1650] Specific operation of the system

[1651] User image input

[1652] The user uses a terminal to input an image into the system, such as "a heartwarming story to read in a quiet forest." The terminal then sends this input to the server.

[1653] Image analysis and generation

[1654] The server calls a generative AI model (e.g., GPT-4) to analyze the received image. The server inputs a prompt to the generative AI model: "Generate a cover and synopsis of a heartwarming story to read in a quiet forest." The generative AI model generates a cover and synopsis of the novel based on the prompt. Specifically, the generative AI model outputs a cover image of "a scene of sunlight filtering through the trees in a lush forest" and a synopsis of "heartwarming interactions with animals encountered in the forest." The server receives the generated cover image and synopsis and proceeds to the next processing step.

[1655] Search for similar products

[1656] The server connects to a database management system (e.g., MySQL) and searches for similar novels based on the generated cover and synopsis. Specifically, the server searches the database for keywords such as "heartwarming story" and "forest scenery" and lists relevant novels. The server temporarily stores the search results and proceeds to the next processing step.

[1657] Emotion recognition and search result tailoring

[1658] While a user is using a terminal to access the system, the terminal monitors the user's facial expressions and input content and sends them to the emotion recognition engine. The emotion recognition engine analyzes the user's emotions and sends the results to the server. Specifically, if the user shows the emotion of joy, the emotion recognition engine determines this as "joy" and notifies the server. The server adjusts the search results based on the received emotion information. Specifically, the server prioritizes listing novels related to brightly colored images based on the emotion of "joy."

[1659] Presenting search results

[1660] The server generates the final search results and sends them to the user's device. The user can then view the results through the device. Specifically, the user browses the covers and summaries of the listed novels on the device screen and selects the novels that interest them.

[1661] Examples and prompts

[1662] Specific examples

[1663] User Input: "A heartwarming story to read in the quiet of the forest."

[1664] Prompt for generative AI model: "Generate a cover and synopsis for a heartwarming story to read in a quiet forest."

[1665] Produced output:

[1666] Cover: A lush forest with sunlight filtering through the trees

[1667] Story: A heartwarming encounter with animals in the forest

[1668] Specific examples of emotion recognition

[1669] User Emotion: Delight

[1670] Adjust search results: Based on the bright color images, novels with a bright atmosphere are searched and presented preferentially.

[1671] In this way, the system searches for and presents the most suitable product based on the user's image and feelings. The flow of the specification process in the third embodiment will be described with reference to FIG.

[1672] Step 1:

[1673] The user uses a terminal to input an image into the system, such as "a heartwarming story to read in a quiet forest." The terminal sends this input to the server, which receives the input data in text format.

[1674] Step 2:

[1675] The server calls a generative AI model (e.g., GPT-4) to analyze the received image. The server inputs a prompt to the generative AI model: "Generate a cover and synopsis of a heartwarming story to read in a quiet forest." The generative AI model generates a cover and synopsis of the novel based on the prompt. Specifically, the generative AI model outputs a cover image of "a scene of sunlight filtering through the trees in a lush forest" and a synopsis of "heartwarming interactions with animals encountered in the forest." The server receives the generated cover image and synopsis and proceeds to the next processing step.

[1676] Step 3:

[1677] The server connects to a database management system (e.g., MySQL) and searches for similar novels based on the generated cover and synopsis. The input data are the generated cover and synopsis, and the server uses these as keywords to search the database. Specifically, the server searches the database for keywords such as "heartwarming story" and "forest scenery" and lists relevant novels. The server temporarily stores the search results and proceeds to the next processing step.

[1678] Step 4:

[1679] While a user is using a terminal to access the system, the terminal monitors the user's facial expressions and input content and sends them to the emotion recognition engine. The input data is the user's facial expressions and input content, and the terminal sends this to the emotion recognition engine in real time. The emotion recognition engine analyzes the user's emotions and sends the results to the server. Specifically, if the user is showing the emotion of joy, the emotion recognition engine will determine this as "joy" and notify the server.

[1680] Step 5:

[1681] The server adjusts the search results based on the received emotional information. The input data is emotional information from the emotion recognition engine, and the server optimizes the search results based on this. Specifically, the server prioritizes listing novels related to brightly colored images based on the emotion of "joy."

[1682] Step 6:

[1683] The server generates the final search results and sends them to the user's device. The input data is the adjusted search results, which the server sends to the user's device. The user can check the presented search results through the device. Specifically, the user views the covers and summaries of the listed novels on the device screen and selects the novels that interest them.

[1684] (Application example 3)

[1685] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1686] In conventional mail-order systems, it was difficult for users to search for specific products based on their emotions or images. Furthermore, product suggestions did not take the user's emotions into account, which could lead to a decrease in user satisfaction. Furthermore, there was a lack of a system that could search and present physical and non-physical products in a unified manner.

[1687] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means.

[1688] In this invention, the server includes means for generating an image based on an input from a user, means for searching for products based on the generated image, means for presenting the searched products to the user, means for recognizing the user's emotion, and means for searching for products based on the recognized emotion. This enables product search and suggestions based on the user's emotion and image, and makes it possible to search for and present physical and non-physical products in an integrated manner.

[1689] "User input" refers to information such as text and images that a user provides to the system.

[1690] An "image" is a visual representation generated based on user input.

[1691] "Products" include physical goods and non-physical services and content.

[1692] "Emotion" refers to the user's psychological state or mood.

[1693] "Means for recognizing emotions" refers to technology that analyzes and identifies emotions from user input and behavior.

[1694] A "generative AI model" is an algorithm that uses artificial intelligence to generate images or text based on user input.

[1695] A "prompt" is an instruction given to a generative AI model to guide the content that the model generates.

[1696] The system for implementing this invention generates an image based on user input, searches for products based on the image, and presents the searched products to the user. It also has the function of recognizing the user's emotions and searching for products based on those emotions.

[1697] The system includes the following main components:

[1698] 1. User Input Module: Provides an interface for users to input text and images using devices such as smartphones and tablets.

[1699] 2. Image generation module: Generates an image based on user input. It uses a generative AI model to analyze the user input and generate an appropriate image.

[1700] 3. Emotion Recognition Module: Recognizes emotions from user input and behavior. Analyzes the user's psychological state using the emotion engine.

[1701] 4. Product Search Module: Search for products based on the generated images and the recognized emotions. Use a search engine to search for physical and non-physical products in the database.

[1702] 5. Product presentation module: Presents searched products to users, visually displaying search results through the user interface.

[1703] Hardware and software used

[1704] Hardware: smartphones, tablets, servers

[1705] Software: Generative AI models (e.g., OpenAI APIs), emotion engines, search engines, user interfaces

[1706] Specific examples of processing

[1707] A user uses a smartphone to input "candles with relaxing scents." The emotion recognition module detects the emotion of "relaxing" from the user's input. The image generation module uses a generative AI model to generate images of candles with relaxing scents. The product search module searches the database for the most suitable products based on the generated images and the recognized emotions. The product presentation module presents the search results to the user.

[1708] Prompt Sentence Examples

[1709] If a user types "a candle with a relaxing scent" and the emotion engine detects the emotion "relaxed," it will input the following prompt sentence into the generative AI model:

[1710] If the user is expressing the emotion of relaxation, search for the best products based on relaxing scented candles.

[1711] Based on this prompt, the generative AI model searches for the most suitable product and presents it to the user, for example, "lavender-scented candles."

[1712] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[1713] Step 1:

[1714] The user inputs product images using a smartphone.

[1715] Input: User types the text "Relaxing scented candle."

[1716] Output: User input data is sent to the server.

[1717] Specific actions: A user opens an application on their smartphone, types "relaxing scented candles" into the text box, and presses the send button.

[1718] Step 2:

[1719] The server receives the user's input and analyzes the emotion with an emotion recognition module.

[1720] Input: User-supplied data ("A candle with a relaxing scent")

[1721] Output: Recognized emotion data ("Relaxed")

[1722] Specific operation: The server uses the emotion engine to detect the emotion "relaxed" from the user's input text.

[1723] Step 3:

[1724] The server uses a generative AI model to generate an image based on the user's input.

[1725] Input: User-input data ("A candle with a relaxing scent"), Recognized emotion data ("Relaxed")

[1726] Output: Generated image

[1727] Specific operation: The server inputs a prompt into the generative AI model and generates an image. An example of a prompt is: "If the user is expressing the emotion of relaxation, please search for the best products based on candles with relaxing scents."

[1728] Step 4:

[1729] The server searches for products based on the generated images and the recognized emotions.

[1730] Input: Generated image, recognized emotion data ("Relax")

[1731] Output: Searched product data

[1732] What happens: The server uses a search engine to search a database for products related to relaxing scented candles.

[1733] Step 5:

[1734] The server presents the search results to the user.

[1735] Input: Searched product data

[1736] Output: The product list presented to the user

[1737] Specific operation: The server displays the search results on the smartphone screen through the user interface, allowing the user to see a list of candles with relaxing scents.

[1738] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1739] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1740] Another example of generative AI is Gemini (internet search engine). <url: https: gemini.google.com ?hl="ja">) are mentioned.

[1741] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1742] [Third embodiment]

[1743] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1744] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[1745] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1746] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1747] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1748] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1749] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1750] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1751] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1752] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1753] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1754] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[1755] "Example 1"

[1756] The system of the present invention receives input from a user (e.g., keywords such as color, shape, and style) and generates an image based on that input. This image generation is performed using artificial intelligence techniques such as deep learning. The generated image is presented to the user and can be revised based on user feedback.

[1757] "Example 2"

[1758] The generated and modified image is used to search the product database. This search is performed using image recognition technology to find products similar to the image. The found products are then presented to the user.

[1759] "Example 3"

[1760] The system of the present invention can be applied to not only physical products but also non-physical products. For example, if a user inputs an image such as "a heartwarming story to read in a quiet forest," the system can generate a novel cover and synopsis based on that image and search for and present similar novels.

[1761] The processing flow of each embodiment will be described below.

[1762] "Example 1"

[1763] Step 1: Receive input from the user (e.g. keywords such as color, shape, style, etc.).

[1764] Step 2: Based on the received input, an image is generated using artificial intelligence techniques such as deep learning.

[1765] Step 3: Present the generated image to the user and provide the ability to modify it based on user feedback.

[1766] "Example 2"

[1767] Step 1: Generate and correct the image.

[1768] Step 2: Search the product database based on the acquired image. This search is performed using image recognition technology.

[1769] Step 3: Present the found products to the user as search results.

[1770] "Example 3"

[1771] Step 1: Receive input from the user (e.g., "a heartwarming story to read in a quiet forest").

[1772] Step 2: Generate a cover and synopsis for the novel based on the input received.

[1773] Step 3: Based on the generated cover and synopsis, search the novel database to find similar novels.

[1774] Step 4: Present the found novels to the user.

[1775] Example 1

[1776] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1777] In conventional systems, when generating images based on user input, the generated images often did not meet the user's expectations. Furthermore, when searching for products based on generated images, the system was unable to accurately reflect the user's intentions. Furthermore, the system lacked the ability to modify images based on user feedback, making it difficult to increase user satisfaction.

[1778] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1779] In this invention, the server includes means for generating an image based on user input, means for searching for products based on the generated image, means for presenting the searched products to the user, means for analyzing the user input and generating a prompt sentence for the generative AI model, means for inputting the prompt sentence into the generative AI model and generating an image, and means for modifying the prompt sentence based on user feedback and generating the image again. This makes it possible to generate an image that accurately reflects the user's input, and by further modifying the image to reflect user feedback, it is possible to increase user satisfaction.

[1780] "User input" refers to information that a user provides to the system through a terminal, including keywords such as color, shape, and style.

[1781] An "image" is a visual image generated based on user input and created by a generative AI model.

[1782] A "generative AI model" is an artificial intelligence model that uses deep learning technology to generate image images from prompt text, and examples include Stable Diffusion and DALL-E.

[1783] A "prompt" is a text-based instruction generated based on user input and input into a generative AI model.

[1784] "Feedback" refers to requests for corrections or opinions that the user makes about the generated image, and is reflected when the system regenerates the image.

[1785] "Products" are objects that are searched for based on the generated image, and include physical products (such as clothes and furniture) and non-physical products (such as novels and songs).

[1786] The "server" is a central processing unit that receives input from users, inputs prompts into the generative AI model, and generates and modifies images.

[1787] A "terminal" is a device that allows a user to access the system and provide input or feedback, including PCs and smartphones.

[1788] MODE FOR CARRYING OUT THE INVENTION

[1789] This invention is a system that generates an image based on user input, searches for products based on the image, and presents them to the user. It also has the function of modifying the image based on user feedback.

[1790] Hardware and software used

[1791] The system uses the following hardware and software:

[1792] Server: Acts as the central processing unit, receives input from the user, provides prompts to the generative AI model, and generates and modifies images.

[1793] Terminal: A device that allows users to access the system and provide input and feedback. This includes PCs, smartphones, etc.

[1794] Generative AI model: An artificial intelligence model that uses deep learning techniques to generate images from prompts. Examples include Stable Diffusion and DALL-E.

[1795] Data processing and calculation

[1796] The server receives input from the user and performs data processing and calculations according to the following steps:

[1797] 1. Receiving user input: The user accesses the system interface through a terminal and inputs keywords such as color, shape, style, etc.

[1798] 2. Prompt generation: The server analyzes the keywords sent by the user and generates a prompt suitable for the generative AI model. For example, from the keywords "blue sky, white clouds, landscape," it generates the prompt "Generate an image of a landscape with a blue sky and white clouds."

[1799] 3. Image generation: The server inputs the generated prompt into the generative AI model to generate an image. The generative AI model generates an image based on the prompt using deep learning technology.

[1800] 4. Sending the image: The server sends the generated image to the user's terminal and presents it to the user.

[1801] 5. Receiving Feedback: The user provides feedback on the image presented to them, for example, by inputting specific requests for corrections such as "make the clouds bigger" or "make the sky brighter."

[1802] 6. Prompt Revision: The server receives feedback from the user and regenerates the prompt. For example, in response to the user's feedback, "I want the clouds to be bigger," the server generates a revised prompt, "Generate an image of a landscape with a blue sky and larger white clouds."

[1803] 7. Generate a modified image: The server inputs the modified prompt into the generative AI model to generate a modified image. The modified image is then sent back to the user's device.

[1804] Specific examples

[1805] As a concrete example, consider the case where a user inputs "a landscape with blue sky and white clouds." The user inputs the keywords "blue sky, white clouds, landscape" through the terminal. The server receives this and inputs the following prompt sentence to the generative AI model:

[1806] Example prompt sentence:

[1807] "Generate an image of a landscape with a blue sky and white clouds."

[1808] The generative AI model generates an image based on this prompt, and the server sends the generated image to the user's device. If the user provides feedback such as "I want the clouds to be bigger," the server again inputs the following modified prompt into the generative AI model:

[1809] Example of a modified prompt:

[1810] "Generate an image of a landscape with a blue sky and larger white clouds."

[1811] In this way, the image is modified based on user feedback.

[1812] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1813] Step 1:

[1814] The user enters a keyword through the terminal.

[1815] A user accesses the system's interface using their own device (e.g., a PC or smartphone). On the interface, the user enters keywords such as "blue sky, white clouds, landscape." The entered keywords are sent to the server.

[1816] Input: Keywords entered by the user (e.g., "blue sky, white clouds, landscape")

[1817] Output: Keyword data sent to the server

[1818] Step 2:

[1819] The server receives the user's input and generates a prompt.

[1820] The server receives and analyzes the keywords sent by the user. Based on the analysis results, it generates a prompt sentence appropriate for the generative AI model. For example, the keyword "blue sky, white clouds, landscape" generates the prompt sentence "Generate an image of a landscape with a blue sky and white clouds."

[1821] Input: Keyword data sent by the user

[1822] Output: The generated prompt (e.g., "Generate an image of a landscape with a blue sky and white clouds.")

[1823] Step 3:

[1824] The server inputs a prompt sentence into the generative AI model and generates an image.

[1825] The server inputs the generated prompt into the generative AI model, which uses deep learning technology to generate an image based on the prompt. The generated image is then returned to the server.

[1826] Input: Generated prompt text

[1827] Output: Generated image

[1828] Step 4:

[1829] The server sends the generated image to the user's terminal.

[1830] The server receives the image output from the generative AI model and sends it to the user's device, where the user can view the generated image on their device screen.

[1831] Input: Generated image

[1832] Output: The image sent to the user's device

[1833] Step 5:

[1834] The user provides feedback on the image.

[1835] The user provides feedback on the image presented to them, for example, by inputting specific requests for correction, such as "make the clouds bigger" or "make the sky brighter." The feedback is then sent to the server.

[1836] Input: User feedback (e.g. "I want the clouds to be bigger")

[1837] Output: Feedback data sent to the server

[1838] Step 6:

[1839] The server receives the feedback, regenerates the prompt text, and modifies the image.

[1840] The server receives feedback from the user and regenerates the prompt. For example, in response to feedback such as "I want the clouds to be larger," it generates a revised prompt, "Generate an image of a landscape with a blue sky and larger white clouds." The server then inputs this revised prompt into the generative AI model and generates a revised image. The revised image is then sent back to the user's device.

[1841] Input: User feedback data

[1842] Output: Modified prompt and modified image

[1843] (Application example 1)

[1844] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1845] On traditional online shopping sites, when users searched for a specific product, they had to manually enter keywords and categories, making it difficult to find products that accurately reflected their intent. Furthermore, users had limited options for specifically expressing their desired design and style, making it difficult to find products that satisfied them. Furthermore, the lack of a function to re-search for products based on user feedback made it difficult to improve the user experience.

[1846] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1847] In this invention, the server includes means for generating an image based on input from a user, means for searching for products based on the generated image, means for presenting the searched products to the user, means for modifying the image based on feedback from the user, and means for searching for products again based on the modified image, thereby enabling a product search that accurately reflects the user's intentions and improving the user experience.

[1848] "User input" refers to keywords such as color, shape, and style that users provide to the system.

[1849] An "image" is a visual image generated based on input from a user.

[1850] "Generating means" refers to a device or program that has the function of generating image data using artificial intelligence technology such as deep learning.

[1851] The "means for searching for products" is a device or program that has the function of searching a database for related products based on the generated image.

[1852] "Means for presenting to the user" means a device or program that has the function of displaying searched items to the user in visual or text form.

[1853] "Feedback" refers to opinions and correction instructions provided by users regarding generated images and search results.

[1854] A "modifying means" is a device or program that has the ability to regenerate or modify an image based on user feedback.

[1855] The "means for searching for products again" refers to a device or program having the function of searching the database for related products again based on the corrected image.

[1856] The following system configuration and processing procedure will be described as an embodiment of the present invention.

[1857] System Configuration

[1858] Hardware:

[1859] server

[1860] User devices (smartphones, tablets, PCs, etc.)

[1861] software:

[1862] Deep learning libraries (such as TensorFlow)

[1863] Image processing library (such as PIL)

[1864] Database Management Systems

[1865] User Interface (UI) Applications

[1866] Processing steps

[1867] 1. User Input:

[1868] Users use devices such as smartphones or PCs to input keywords such as color, shape, and style, for example, "red," "dress," and "elegant."

[1869] 2. Image generation:

[1870] The server receives input from the user and generates images using a generative AI model built using deep learning libraries (such as TensorFlow).

[1871] 3. Product Search:

[1872] The server searches the database for related products based on the generated image, using an image processing library (such as PIL).

[1873] 4. User Presentation:

[1874] The searched products are presented to the user through a UI application on the user's device, where the user can review the presented products and decide whether or not to purchase them.

[1875] 5. Receiving Feedback and Making Corrections:

[1876] If the user provides feedback on the presented product or image, the server receives that feedback and modifies the image, again using the generative AI model.

[1877] 6. Search again:

[1878] Based on the revised image, the server retrieves the product again and presents it to the user. This process is repeated until the user is satisfied.

[1879] Specific examples

[1880] For example, if a user enters the keywords "red," "dress," and "elegant," the server converts these keywords into vectors and inputs them into a generative AI model. The generative AI model then generates an image of an elegant red dress based on this. The generated image is displayed on the user's device, and related products are searched for and presented. If the user provides feedback such as "I wish the red was a little brighter," the server receives that feedback, modifies the image, and searches for and presents the products again.

[1881] Example prompt sentence:

[1882] User entered keywords: ["red", "dress", "elegant"]

[1883] Vector to be input to the generative AI model: [0.5, 0.2, 0.8, ...] Keywords converted to vectors

[1884] In this way, product searches can be performed that accurately reflect the user's intent, improving the user experience.

[1885] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1886] Step 1:

[1887] The user uses the terminal to input keywords such as color, shape, style, etc. The input keywords are sent from the terminal to the server.

[1888] Input: Keywords entered by the user (e.g. "red", "dress", "elegant")

[1889] Output: Keyword data sent to the server

[1890] Step 2:

[1891] The server converts the received keywords into vectors using a pre-trained keyword embedding model.

[1892] Input: Keyword data submitted by the user

[1893] Output: Keyword vector (e.g. [0.5, 0.2, 0.8, ...])

[1894] Step 3:

[1895] The server inputs keyword vectors into the generative AI model to generate images, using deep learning libraries (such as TensorFlow).

[1896] Input: Keyword vector

[1897] Output: Generated image

[1898] Step 4:

[1899] The server searches for products based on the generated image in a database, using an image processing library (such as PIL).

[1900] Input: Generated image

[1901] Output: Searched product list

[1902] Step 5:

[1903] The server sends the searched product list to the user's terminal and presents it to the user, who then checks the presented products.

[1904] Input: Searched product list

[1905] Output: Product list displayed on user's device

[1906] Step 6:

[1907] The user provides feedback on the presented product or image, which is then sent from the device to the server.

[1908] Input: User feedback (e.g., "I wish it was a brighter red")

[1909] Output: Feedback data sent to the server

[1910] Step 7:

[1911] The server then modifies the image based on the feedback it receives, again using a generative AI model.

[1912] Input: Feedback data submitted by the user

[1913] Output: The corrected image

[1914] Step 8:

[1915] The server performs a product search again based on the corrected image, and the search results are sent again to the user's device.

[1916] Input: The corrected image

[1917] Output: Re-searched product list

[1918] Step 9:

[1919] The user reviews the presented product again and provides further feedback if necessary, and this process is repeated until the user is satisfied.

[1920] Input: Re-searched product list

[1921] Output: The user's final choice or additional feedback

[1922] Example 2

[1923] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1924] Previous systems had problems with image recognition accuracy and the relevance of search results when searching for products based on images uploaded by users. It was also difficult to quickly and accurately present the products users were looking for. Furthermore, few systems supported searches for both physical and non-physical products, making them unable to meet the diverse needs of users.

[1925] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for generating an image based on an input from a user, means for searching for products based on the generated image, means for presenting the searched products to the user, means for receiving an image uploaded by the user, means for extracting features of the received image, and means for searching a product database based on the extracted features. This makes it possible to recognize an image uploaded by a user with high accuracy and quickly search for and present appropriate products. Furthermore, by targeting both physical and non-physical products for search, it is possible to meet the diverse needs of users.

[1926] "User" means an individual or corporation that uses the system to upload images and search for products.

[1927] "Input" refers to the operations and instructions that users give to the system, and specifically includes uploading images and specifying search criteria.

[1928] "Image" refers to an image file uploaded by a user to the system, which contains visual information that is the basis for product searches.

[1929] A "generating means" is a part of a system that has the ability to create or modify an image based on input from a user.

[1930] The "searching means" is a part of the system that has the function of searching a product database based on the characteristics of the generated image and finding similar products.

[1931] The "presentation means" is a part of the system that has the function of displaying searched product information to the user.

[1932] The "means for receiving" is a part of the system that has the function of allowing the server to receive images uploaded by users.

[1933] The "means for extracting features" is a part of the system that has the function of analyzing features such as color, shape, and pattern from a received image and generating a feature vector.

[1934] "Product database" refers to a database that stores product information that the system searches, including information about physical and non-physical products.

[1935] The "searching means" is a part of the system that has the function of querying the product database based on the extracted feature vector and finding highly similar products.

[1936] MODE FOR CARRYING OUT THE INVENTION

[1937] The present invention is a system for searching for products based on images uploaded by users and presenting search results to the users. A specific embodiment of this system will be described below.

[1938] Hardware and software used

[1939] This system consists of a server and a user's terminal. The server is a computer equipped with a high-performance processor and a large amount of memory, and the user's terminal is a device such as a personal computer or smartphone. The following software is used:

[1940] Image Recognition: TensorFlow

[1941] Image processing: OpenCV

[1942] Database operations: SQLAlchemy

[1943] Web server: Flask

[1944] System processing flow

[1945] Users access the system's web interface from their devices and upload images of the products they want to search for. The server receives these images and uses TensorFlow to perform image recognition. Specifically, it extracts features such as color, shape, and pattern from the image and generates a feature vector.

[1946] Next, the server uses SQLAlchemy to search the product database. Based on the extracted feature vector, it compares it with the feature vectors of each product in the database to find products with high similarity. The search results are sorted in descending order of similarity.

[1947] Finally, the server presents the search results to the user, specifically displaying information such as images, names, prices, and links of similar products on the user's device.

[1948] Specific examples

[1949] For example, consider the case where a user uploads an image of red shoes. The server receives the image and uses TensorFlow to extract the feature "red shoes." Next, the server searches the product database to find similar red shoe product information. Finally, the server presents the found red shoe product information to the user.

[1950] Prompt Sentence Examples

[1951] An example of a prompt to input to a generative AI model is as follows:

[1952] "I uploaded an image of a pair of red shoes. Based on this image, I want you to search our product database and find similar products."

[1953] Using this prompt, the generative AI model can perform product searches according to the user's request.

[1954] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1955] Step 1:

[1956] The user accesses the system's web interface from their terminal and uploads an image of the product they want to search for. Specifically, the user opens a browser and accesses the system's web page. They click the "Upload Image" button on the page and select an image of red shoes from the file selection dialog. After selecting, they click the "Upload" button. The input is the image selected by the user, and the output is an image file sent to the server.

[1957] Step 2:

[1958] The server receives an image uploaded by a user. The server receives the HTTP request and saves the uploaded image file in a temporary folder on the server. After saving, it checks the image format and converts it to PNG format if necessary. The input is the image file sent by the user, and the output is the image file saved on the server.

[1959] Step 3:

[1960] The server inputs the stored image file into the TensorFlow model to extract image features. Specifically, it analyzes the image's color, shape, pattern, etc., and generates a feature vector. The input is the image file stored on the server, and the output is the extracted feature vector.

[1961] Step 4:

[1962] The server uses SQLAlchemy to search the product database based on the extracted feature vector. The server converts the feature vector into a database query and searches the product database. The search results are sorted in descending order of similarity. The input is the extracted feature vector, and the output is a list of highly similar product information.

[1963] Step 5:

[1964] The server generates search results in HTML format and sends them to the user's browser. Specifically, information such as images, names, prices, and links of similar products is displayed on the user's device. The input is a list of highly similar product information, and the output is a search result page displayed in the user's browser.

[1965] (Application example 2)

[1966] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1967] On conventional online shopping sites, it was difficult for users to efficiently find the products they were looking for. In particular, when users did not know the specific product name or wanted to search for a product based on a visual image, the search process was complicated and time-consuming. This led to issues such as a decrease in users' motivation to purchase and a decrease in the frequency of site use.

[1968] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1969] In this invention, the server includes means for generating an image based on input from a user, means for searching for products based on the generated image, means for presenting the searched products to the user, and means for searching a product database based on an image uploaded by the user and displaying similar products, thereby enabling users to efficiently search for products based on visual images and easily find similar products.

[1970] "User input" refers to information or instructions provided by a user to a system.

[1971] An "image" is a visual image generated based on user input.

[1972] A "product database" is a database that stores information about various products.

[1973] "Image recognition technology" refers to the technology in which a computer analyzes images and recognizes specific features and patterns.

[1974] "Similar products" refer to products that have similar features to the image provided by the user.

[1975] A "generative AI model" is a model that uses artificial intelligence to generate new images.

[1976] A "prompt" is an instruction that a user inputs to a generative AI model.

[1977] "Upload" means the act of a user sending data from their device to the system.

[1978] "Searching" is the act of locating information in a database based on specific criteria.

[1979] "Presenting" refers to the act of displaying search results to the user.

[1980] The system for implementing this invention is based on the premise that the user operates it using a terminal such as a smartphone. The system includes the following main components:

[1981] Hardware and Software Configuration

[1982] 1. Terminal: A device such as a smartphone or tablet that is operated by a user.

[1983] 2. Server: A server for running image recognition technology and generative AI models.

[1984] 3. Software:

[1985] Python: Used for image processing and running machine learning models.

[1986] OpenCV: An image processing library.

[1987] scikit-learn: A machine learning library.

[1988] PIL (Python Imaging Library): Used to load and convert images.

[1989] Data processing and calculation

[1990] 1. User input: The user inputs a prompt to generate an image using the terminal. For example, the user can input a prompt such as, "Please upload an image of a red shirt. Similar products will be searched and displayed."

[1991] 2. Image generation: The server uses a generative AI model to generate an image based on the user's prompt.

[1992] 3. Image upload: User uploads an image from their device, which is sent to the server.

[1993] 4. Image recognition and feature extraction: The server uses OpenCV to analyze the uploaded image and extract features.

[1994] 5. Search the product database: The server uses scikit-learn to compare the extracted features with the features of products in the product database and calculate the similarity.

[1995] 6. Show similar products: The server shows similar products to the user. For example, if you upload an image of a red shirt based on the prompt "Please upload an image of a red shirt. We will search and display similar products," similar red shirt products will be displayed.

[1996] Specific examples

[1997] When a user uploads an image of a red shirt, the server analyzes the image and searches for similar red shirts in a product database. The search results show multiple red shirts on the user's device. The user can select a product to view more information or proceed with the purchase.

[1998] In this way, users can efficiently search for products based on visual images and easily find similar products.

[1999] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[2000] Step 1:

[2001] The user enters a prompt at the terminal.

[2002] Input: The prompt text that the user enters (e.g., "Upload an image of a red shirt. We'll search for and display similar products.").

[2003] Output: The prompt text is sent to the server.

[2004] Specific actions: The user opens the terminal application, enters a prompt in the text box, and presses the send button.

[2005] Step 2:

[2006] The server uses a generative AI model to generate an image based on the user's prompt.

[2007] Input: The prompt statement.

[2008] Output: The generated image.

[2009] Specific operation: The server inputs a prompt sentence into the generative AI model, and the model generates an image. The generated image is temporarily saved.

[2010] Step 3:

[2011] The user uploads an image from their device.

[2012] Input: An image file selected by the user.

[2013] Output: The image file is sent to the server.

[2014] Specific behavior: The user opens the file selection dialog on their device, selects the image they want to upload, and presses the upload button.

[2015] Step 4:

[2016] The server analyzes the uploaded image using OpenCV and extracts features.

[2017] Input: The uploaded image file.

[2018] Output: Extracted features.

[2019] Specific operation: The server reads the image using OpenCV and extracts the image features (e.g., color, shape, texture, etc.).

[2020] Step 5:

[2021] The server uses scikit-learn to compare the extracted features with the features of products in the product database and calculate the similarity.

[2022] Input: Extracted features, product database.

[2023] Output: Similarity score for each product.

[2024] Specific operation: The server compares the features of each product in the product database with the features of the uploaded image and calculates the cosine similarity.

[2025] Step 6:

[2026] The server presents highly similar products to the user.

[2027] Input: Similarity score for each product.

[2028] Output: A list of products sorted by most similar.

[2029] Specific operation: The server sorts the products based on the similarity score and sends the top similar products to the user's device.

[2030] Step 7:

[2031] The user checks the presented product and views detailed information.

[2032] Input: Similar products list.

[2033] Output: Detailed product information.

[2034] Specific actions: The user selects a product of interest from the list of similar products displayed on the device and views detailed information.

[2035] Example 3

[2036] Next, a third embodiment of the third embodiment will be described. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[2037] In conventional systems, when searching for products based on images entered by users, the system was often limited to physical products, making it difficult to search for non-physical products. Furthermore, if the user did not have a specific image in mind, it was difficult to find the appropriate product. Furthermore, if the image generated based on the user's input did not meet the user's expectations, it was not easy to correct it.

[2038] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[2039] In this invention, the server includes a means for generating a prompt sentence based on user input, a means for sending the generated prompt sentence to a generative AI model to generate a cover and synopsis, and a means for searching for products based on the generated cover and synopsis. This allows the server to generate an appropriate cover and synopsis using the generative AI model even if the user does not have a specific image in mind, and to search for physical and non-physical products based on the generated cover and synopsis. The server also has a function for modifying the prompt sentence based on user input, making it easier to generate an image that meets the user's expectations.

[2040] "User" means an individual or organization that uses the system to input images and search for products.

[2041] A "prompt sentence" is an instruction sentence for a generative AI model that is generated based on an image entered by the user.

[2042] A "generative AI model" is an artificial intelligence model that receives a prompt and generates a cover and synopsis based on it.

[2043] A "cover" is a visual cover image of a novel or other non-physical product generated by a generative AI model.

[2044] A "plot" is a concise description of the contents of a novel or other non-physical product, generated by a generative AI model.

[2045] "Product" means any object that a user wishes to search for and acquire, including physical and non-physical products.

[2046] "Physical goods" are tangible products such as clothing and furniture.

[2047] "Non-physical products" are digital content or services that do not have a physical form, such as novels or songs.

[2048] The "server" is a central processing unit that searches for products based on the generated cover and synopsis and presents them to the user.

[2049] A "terminal" is a device that allows a user to input an image and check the generated cover and synopsis.

[2050] A "database" is an information management system that allows a server to store and search product information.

[2051] MODE FOR CARRYING OUT THE INVENTION

[2052] This invention is a system that generates prompt text based on an image input by a user, generates a cover and synopsis using a generative AI model, and searches for and presents products based on the generated text. Specific embodiments of this system are described below.

[2053] Hardware and software used

[2054] Hardware:

[2055] Device: The device on which the user inputs their image and views the generated cover and synopsis. Examples include a PC, smartphone, or tablet.

[2056] Server: A central processing unit that searches for products based on the generated cover and synopsis and presents them to users. A high-performance server machine is recommended.

[2057] software:

[2058] Generative AI model: An artificial intelligence model that receives a prompt and generates a cover or synopsis based on it. For example, OpenAI's GPT-4 is used.

[2059] Database: An information management system that allows the server to store and retrieve product information. For example, MySQL is used.

[2060] System processing flow

[2061] The user inputs an image such as "A heartwarming story to read in a quiet forest" into the device's input field. The device receives this image and converts it into a prompt suitable for the generative AI model. For example, it generates a prompt such as "Generate the cover and synopsis of a heartwarming story to read in a quiet forest."

[2062] The device sends the generated prompt to the generative AI model, which analyzes the prompt and uses an internal generation algorithm to generate a cover and synopsis. The generated cover depicts a small wooden house nestled in a lush forest, and the synopsis includes the following: "This story depicts the small joys that a woman living in a quiet forest finds in her daily life."

[2063] The device sends the cover and synopsis received from the generative AI model to the server. The server analyzes the cover and synopsis and uses an SQL query to match them with novel information in the database. Based on the results of the SQL query, the server searches for novels similar to the generated cover and synopsis. For example, if a novel titled "Little House in the Forest" exists in the database and its synopsis is "a story about a woman who lives quietly in the forest and finds small joys in her daily life," this novel will be selected as a search result.

[2064] The server returns search results in JSON format to the device. The search results include the titles, covers, and summaries of similar novels. The device analyzes the search results received from the server and displays them in a user interface. The user can review the search results and click on a novel of interest to view detailed information.

[2065] Examples of concrete examples and prompts

[2066] For example, if a user inputs the image of "a heartwarming story to read in a quiet forest," the following prompt sentences are input to the generative AI model:

[2067] Example prompt sentence:

[2068] "Generate a cover and synopsis for a heartwarming story to read in the quiet of the forest."

[2069] Based on this prompt, the generative AI model generates a cover and synopsis like this:

[2070] Example of a generated cover:

[2071] The cover depicts a small wooden house nestled in a lush forest, with people enjoying reading in front of the house, and the overall image is presented in warm colors.

[2072] An example of a generated synopsis:

[2073] "This story depicts the small joys that a woman living in a quiet forest finds in her daily life. She is comforted by the animals and natural beauty she encounters in the forest, and gradually begins to reexamine herself."

[2074] In this way, even if the user does not have a specific image, the generative AI model can generate an appropriate cover and synopsis, and physical and non-physical products can be searched for based on the generated cover and synopsis. Furthermore, the system has the function of modifying prompts based on user input, making it easier to generate an image that meets the user's expectations. The flow of the identification process in Example 3 will be described with reference to Figure 15.

[2075] Step 1:

[2076] The user inputs an image.

[2077] The user enters an image such as "a heartwarming story to read in a quiet forest" into the input field on the terminal and clicks the send button.

[2078] Input: User image text

[2079] Output: Image text sent to the terminal

[2080] Step 2:

[2081] The terminal converts the input image into a prompt.

[2082] The device takes the user's input and uses its internal text processing capabilities to generate the prompt, "Generate a cover and synopsis for a heartwarming story to read in the quiet forest."

[2083] Input: User image text

[2084] Output: Generated prompt statement

[2085] Step 3:

[2086] The device sends the prompt to the generative AI model.

[2087] The device sends the generated prompt as an API request to the generative AI model, such as OpenAI's GPT-4.

[2088] Input: Generated prompt statement

[2089] Output: API request to the generative AI model

[2090] Step 4:

[2091] A generative AI model generates the cover and synopsis.

[2092] The generative AI model analyzes the prompt and uses its internal generation algorithm to generate a cover and synopsis. The cover depicts a small wooden house nestled in a lush forest, and the synopsis includes the following: "This story depicts the small joys that a woman living in a quiet forest finds in her daily life."

[2093] Input: prompt statement

[2094] Output: Generated cover and synopsis

[2095] Step 5:

[2096] The terminal transmits the generated cover and synopsis to the server.

[2097] The device sends the cover and synopsis received from the generative AI model to the server using an HTTP request.

[2098] Input: Generated cover and synopsis

[2099] Output: HTTP request to the server

[2100] Step 6:

[2101] The server compares the novel information with that in the database.

[2102] The server analyzes the cover and synopsis and uses SQL queries to match them with information about the novel in a database, such as MySQL.

[2103] Input: Generated cover and synopsis

[2104] Output: The result of the SQL query

[2105] Step 7:

[2106] The server searches for similar novels.

[2107] The server then searches for novels with similar covers and synopses based on the results of the SQL query. For example, if a novel titled "Little House in the Woods" exists in the database and its synopsis reads, "A story about a woman who lives quietly in the woods and finds small joys in her daily life," then this novel will be selected as a search result.

[2108] Input: SQL query result

[2109] Output: A list of similar novels

[2110] Step 8:

[2111] The server returns the search results to the device.

[2112] The server returns search results in JSON format to the device, including the titles, covers, and summaries of similar novels.

[2113] Input: List of similar novels

[2114] Output: Search results in JSON format to the terminal

[2115] Step 9:

[2116] The terminal presents the search results to the user.

[2117] The terminal analyzes the search results received from the server and displays them on the user interface. The user can review the search results and click on the novel of interest to view detailed information.

[2118] Input: JSON format search results

[2119] Output: Search results displayed in the user interface

[2120] (Application example 3)

[2121] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[2122] Previous systems had limited functionality for searching and presenting related products based on images entered by users, and were particularly inadequate for non-physical products. Even when users had a specific image in mind, it was difficult to find the right product based on that image. Furthermore, the system lacked the functionality to search for similar non-physical products, such as novels, based on the generated image, making it unable to meet user needs.

[2123] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means.

[2124] In this invention, the server includes means for generating an image based on an input from a user, means for searching for products based on the generated image, means for presenting the searched products to the user, means for generating a novel cover and synopsis based on the input image using a generative AI model, means for searching for similar novels based on the generated cover and synopsis, and means for presenting the searched novels to the user. This makes it possible to effectively search for and present physical and non-physical products based on the image input by the user.

[2125] "User input" is information such as text or images provided by system users.

[2126] An "image" is a visual representation generated based on user input.

[2127] A "generative AI model" is an algorithm that uses artificial intelligence techniques to generate content based on user input.

[2128] A "novel cover" is an image that visually represents the contents of a novel.

[2129] A synopsis is a concise summary of the contents of a novel.

[2130] "Product" refers to physical items and non-physical content.

[2131] "Searching" is the act of finding relevant information from a database based on specific criteria.

[2132] "Presenting" is the act of displaying the retrieved information to the user.

[2133] "Similar novels" are novels with similar content and themes based on the generated covers and synopses.

[2134] A system for implementing this invention has the functions of generating an image based on user input, searching for products based on the generated image, and presenting the searched products to the user. It also has the functions of generating a novel cover and synopsis based on the input image using a generative AI model, searching for similar novels based on the generated cover and synopsis, and presenting the searched novels to the user.

[2135] Hardware and software used

[2136] Hardware: Smartphone

[2137] Software: Python, OpenAI API, HTTP request library (requests)

[2138] Data processing and calculation

[2139] 1. User Input: The user inputs a specific image in text form through a smartphone application, for example, a prompt such as "A heartwarming story to read in a quiet forest."

[2140] 2. Use of generative AI model: The server uses the OpenAI API to generate a novel cover and synopsis based on the prompt text entered by the user. The generative AI model analyzes the input text and generates appropriate content based on it.

[2141] 3. Search: Based on the generated cover and synopsis, the server searches for similar novels using an external book search API. The search results include the titles and author information of related novels.

[2142] 4. Displaying Results: The search results are presented to the user, who can then purchase or rent the novels.

[2143] Specific examples

[2144] When a user types "a heartwarming story to read in a quiet forest," the generative AI model generates a novel cover and synopsis based on the prompt. Based on the generated cover and synopsis, the server searches for similar novels and presents the search results to the user. The user can then review the novels and purchase or rent them.

[2145] Prompt Sentence Examples

[2146] "A heartwarming story to read in the quiet forest"

[2147] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[2148] Step 1:

[2149] The user launches an application on their smartphone and inputs a specific image in text format.

[2150] Input: A prompt that the user enters (e.g., "A heartwarming story to read in the quiet woods")

[2151] Output: User input data

[2152] Specific behavior: The user enters a prompt sentence into the application's text input field and presses the submit button.

[2153] Step 2:

[2154] The server receives the user's input data and uses a generative AI model to generate the novel's cover and synopsis.

[2155] Input: User-entered data

[2156] Output: The cover and synopsis of the generated novel

[2157] Specific operation: The server calls the OpenAI API, sends a prompt, and uses the generative AI model to generate a novel cover and synopsis.

[2158] Step 3:

[2159] Based on the cover and synopsis generated by the server, an external book search API is used to search for similar novels.

[2160] Input: Generated novel cover and synopsis

[2161] Output: List of similar novels (title, author information, etc.)

[2162] Specific operation: The server sends the generated cover and synopsis to the book search API to obtain data on similar novels.

[2163] Step 4:

[2164] The server presents the search results to the user.

[2165] Input: List of similar novels

[2166] Output: Search results displayed on the user's smartphone

[2167] Specific operation: The server sends the data of similar novels it has obtained to the user's smartphone, and the application displays it.

[2168] Step 5:

[2169] The user reviews the novels presented and purchases or rents them.

[2170] Input: A novel selected by the user

[2171] Output: Purchase or rental completion notification

[2172] Specific behavior: The user selects a novel of interest from the list of novels presented and presses the purchase or rental button. The server processes the request and sends a completion notification to the user.

[2173] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[2174] "Example 1"

[2175] One embodiment of the present invention is a system that combines an emotion engine that recognizes a user's emotions. This system generates images based on user input, searches for products based on the generated images, and presents the searched products to the user. Furthermore, this system recognizes the user's emotions and generates images and searches for products based on the emotions. Specifically, if the user is expressing joy, an image with bright colors is generated and products are searched for based on the image. On the other hand, if the user is expressing sadness, an image with dark colors is generated and products are searched for based on the image.

[2176] "Example 2"

[2177] Another embodiment of the present invention is a system that has a function of modifying an image based on a modification instruction from a user, and further combines it with an emotion engine that recognizes the user's emotions. This system modifies a generated image in response to the modification instruction from the user, searches for products based on the modified image, and presents the searched products to the user. Furthermore, this system recognizes the user's emotions and modifies the image and searches for products based on the emotion. Specifically, if the user is expressing a happy emotion, the system modifies the image to a brighter color tone and searches for products based on the modified image. On the other hand, if the user is expressing a sad emotion, the system modifies the image to a darker color tone and searches for products based on the modified image.

[2178] "Example 3"

[2179] Yet another embodiment of the present invention is a system that searches for products, whether physical or non-physical, and further combines it with an emotion engine that recognizes a user's emotions. The system searches for physical or non-physical products based on generated images and presents the searched products to the user. The system also recognizes the user's emotions and searches for products based on the emotions. Specifically, if the user expresses a happy emotion, the system searches for physical or non-physical products based on images with bright colors. If the user expresses a sad emotion, the system searches for physical or non-physical products based on images with dark colors.

[2180] The processing flow of each embodiment will be described below.

[2181] "Example 1"

[2182] Step 1: Receive input from the user.

[2183] Step 2: Generate an image based on the input received.

[2184] Step 3: Recognize the user's emotions with the emotion engine.

[2185] Step 4: Generate images and search for products based on the recognized emotions.

[2186] Step 5: Search for products based on the generated image.

[2187] Step 6: Present the searched products to the user.

[2188] "Example 2"

[2189] Step 1: Receive user input and correction instructions.

[2190] Step 2: Generate an image based on the received input and modify the image based on the modification instructions.

[2191] Step 3: Recognize the user's emotions with the emotion engine.

[2192] Step 4: Modify images or search for products based on the recognized emotions.

[2193] Step 5: Search for products based on the modified image.

[2194] Step 6: Present the searched products to the user.

[2195] "Example 3"

[2196] Step 1: Receive input from the user.

[2197] Step 2: Generate an image based on the input received.

[2198] Step 3: Recognize the user's emotions with the emotion engine.

[2199] Step 4: Search for products based on the recognized emotion.

[2200] Step 5: Search for physical and non-physical products based on the generated image.

[2201] Step 6: Present the searched products to the user.

[2202] Example 1

[2203] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[2204] Conventional image generation systems generate images without considering the user's emotions, making it difficult to provide images that match the user's intentions and emotions. Furthermore, they lack the functionality to reflect user feedback on the generated images, which means it takes a lot of effort for the user to obtain an image that satisfies them. Furthermore, when searching for products based on the generated images, they are unable to provide search results that take the user's emotions into account.

[2205] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[2206] In this invention, the server includes means for generating an image based on input from a user, means for searching for products based on the generated image, means for presenting the searched products to the user, means for recognizing the user's emotions and generating an image or searching for products based on the emotions, and means for modifying the image based on feedback from the user. This makes it possible to generate an image that reflects the user's emotions and feedback and to search for products based on the image.

[2207] "User input" refers to information or instructions provided by the user to the system, including keywords such as color, shape, and style.

[2208] "Image" means a visual image generated based on user input and created using a generative AI model.

[2209] A "generative AI model" is an algorithm or program that uses artificial intelligence technology to generate images, for example, using deep learning.

[2210] A "prompt" is a text-based instruction input to a generative AI model, and contains the information necessary to generate an image.

[2211] "Products" are physical or non-physical products or services offered to users, including clothing, furniture, novels, songs, etc.

[2212] "Emotion recognition" is the process of analyzing a user's facial expressions and voice data to determine their emotional state.

[2213] "Feedback" refers to the user's evaluation of the generated image or requests for corrections, which are received and reflected in the system.

[2214] "Revising" is the process of modifying the generated image based on user feedback, again using the generative AI model.

[2215] "Search" is the process of finding related products from a database based on the generated image.

[2216] "Presentation" is the process of displaying searched products and generated images to the user.

[2217] MODE FOR CARRYING OUT THE INVENTION

[2218] This invention is a system that generates an image based on user input, searches for products based on the image, and presents them to the user. It also has the function of recognizing the user's emotions and generating an image or searching for products based on those emotions.

[2219] Hardware and software used

[2220] This system uses the following hardware and software:

[2221] Server: A computer with high-performance computing power that runs generative AI models and manages databases.

[2222] Terminal: A device used by a user, such as a smartphone or PC. It receives input from the user and communicates with the server.

[2223] Generative AI model: An artificial intelligence model for generating images, such as OpenAI's DALL-E or MidJourney.

[2224] Emotion engine: Software that recognizes the user's emotions by analyzing facial expressions and voice data.

[2225] Data processing and calculation

[2226] The server receives input from the user, analyzes it, and converts it into a prompt suitable for the generative AI model. The generative AI model generates an image based on this prompt. The generated image is presented to the user and receives feedback from the user. The server then modifies the image based on this feedback.

[2227] Furthermore, the server uses an emotion engine to recognize the user's emotion and generate images and search for products based on the emotion. For example, if the user expresses joy, it generates an image with bright colors and searches for products based on the image.

[2228] Specific examples

[2229] As a concrete example, consider the case where a user enters the keyword "blue sky and white clouds." The server analyzes this keyword and converts it into a prompt suitable for the generative AI model: "Please generate an image of blue sky and white clouds." The generative AI model generates an image of blue sky and white clouds based on this prompt.

[2230] The generated image is presented to the user via their device, and the user can provide feedback on the image, such as "I'd like more clouds." The server receives this feedback and uses the generative AI model again to generate an image with more clouds.

[2231] Furthermore, if the user smiles while looking at the image, the device's camera captures the facial expression and sends it to the server. The server then uses an emotion engine to recognize that the user is expressing a happy emotion and searches for related products (e.g., brightly colored clothing and accessories) based on the brightly colored image. The searched products are then presented to the user via the device.

[2232] In this way, it becomes possible to generate images that reflect the user's emotions and feedback, and to search for products based on these images.

[2233] The flow of the identification process in the first embodiment will be described with reference to FIG.

[2234] Program processing flow

[2235] Step 1: Receiving User Input

[2236] The user uses a terminal to input keywords such as color, shape, style, etc. to the system, and the terminal sends this input to the server.

[2237] Input: Keywords entered by the user (e.g., "blue sky and white clouds")

[2238] Output: Keywords sent to the server

[2239] Step 2: Keyword analysis

[2240] The server analyzes the received keywords, specifically using natural language processing (NLP) techniques to understand them and convert them into a format suitable for the generative AI model.

[2241] Input: User-submitted keywords

[2242] Output: A prompt suitable for the generative AI model (e.g., "Generate an image of a blue sky with white clouds")

[2243] Step 3: Generate the image

[2244] The server generates an image using a generative AI model based on the analyzed prompt sentence. The generative AI model receives the prompt sentence as input and generates an image based on it.

[2245] Input: A prompt suitable for the generative AI model

[2246] Output: Generated image

[2247] Step 4: Present the image

[2248] The server sends the generated image to the terminal, which displays the image to the user.

[2249] Input: Generated image

[2250] Output: The image presented to the user

[2251] Step 5: Receiving user feedback

[2252] The user can provide feedback on the displayed image, for example by inputting a specific request such as "I'd like more clouds." The device then sends this feedback to the server.

[2253] Input: User feedback

[2254] Output: Feedback sent to the server

[2255] Step 6: Modify the image

[2256] The server receives feedback from the user and again uses the generative AI model to modify the image, which is then sent back to the device and presented to the user.

[2257] Input: User feedback

[2258] Output: The corrected image

[2259] Step 7: Emotion Recognition

[2260] The server recognizes the user's emotions using an emotion engine. The emotion engine analyzes the user's facial expressions and voice data to determine the user's emotional state. For example, if the user expresses joy, the server generates a bright-colored image.

[2261] Input: User facial and voice data

[2262] Output: Perceived emotional state of the user

[2263] Step 8: Product search and presentation

[2264] The server searches for products based on the generated image. For example, based on an image with bright colors, it searches for related products (e.g., bright-colored clothing and accessories). The searched products are presented to the user via the terminal.

[2265] Input: Generated image and perceived user emotional state

[2266] Output: Search results (products) presented to the user

[2267] In this way, a system is realized in which the server, terminals, and users work together to generate image images and search for products.

[2268] (Application example 1)

[2269] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data proce...

Claims

1. a first means for accepting input of color, shape, and style keywords by a user; A second means for generating an image corresponding to the keyword using a generative AI model; A third means for searching a database for related products based on the generated image; a fourth means for presenting the searched products to the user; A system including:

2. When the second means obtains feedback on the product or the image from the user, the second means uses the generative AI model to modify the image based on the feedback; The third means searches the database for related products based on the corrected image. The system of claim 1 .

3. A fifth means for analyzing the emotion of the user from the facial expression and the input contents of the user using an emotion recognition engine; The third means adjusts product search results based on the user's sentiment.

3. A system according to claim 1 or 2.

Images