system

Abstract

We provide the system. [Solution] A means for managing a database of clothing owned by the user, Means for obtaining the user's location information and schedule information, A means for obtaining weather information based on acquired location information and schedule information, A means of suggesting the optimal clothing combination based on the clothes you possess, location information, schedule information, and weather information, A system that includes a means of searching for and suggesting items that are missing from a proposed outfit.

Description

Technical Field

[0005] ,

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a persona chatbot control method performed by at least one processor, including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance as a response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] When deciding today's clothing, there is a problem that it is difficult for users to choose the optimal combination suitable for the weather and schedule. Also, it takes time and effort to find the optimal coordination from the clothes one has. Furthermore, if there are items that are lacking, it takes time and is difficult to appropriately find those items.

Means for Solving the Problems

[0005] The present invention provides a system that includes means for managing a database of clothing owned by a user, means for acquiring the user's location information and schedule information, means for acquiring weather information based on the acquired location information and schedule information, means for suggesting the optimal outfit combination based on the owned clothing, location information, schedule information, and weather information, and means for searching for and suggesting items that are missing from the suggested outfit from an external source. Furthermore, by having means for notifying the user of information on the suggested outfit and missing items, and means for tagging image data of the clothing owned by the user and classifying it based on that tag information, the system enables efficient management of all the clothes owned by the user, presents the optimal outfit based on the day's schedule and weather, and makes it easy to purchase missing items.

[0006] A "user" is an individual or group that uses this system.

[0007] A "database" refers to a collection of information, and in this invention, it specifically refers to a database used to manage information about clothing owned by a user.

[0008] "Location information" refers to information about the user's current location, obtained through GPS or other location information acquisition methods.

[0009] "Schedule information" refers to information about the user's schedule, including events and appointments set on the calendar or manually entered for that day.

[0010] "Weather information" refers to information about weather conditions in a particular area, such as temperature, precipitation, and wind speed, obtained from weather data provision services.

[0011] "Coordination" means suggesting the best combination of clothing from the user's wardrobe, based on the weather and their plans for the day.

[0012] A "tag" is a string of characters or keywords attached to the image data of each garment for identification purposes.

[0013] "Suggestions" refer to the advice this system provides to the user regarding the optimal clothing combination and any missing items.

[0014] "Searching" refers to exploring internal or external databases to find data or items that match specific criteria. [Brief explanation of the drawing]

[0015] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of the data processing device and smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] This shows an emotion map where multiple emotions are mapped. [Figure 11] This is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] This is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13]It is a sequence diagram showing the processing flow of the data processing system in Embodiment 2 when combined with an emotion engine. [Figure 14] It is a sequence diagram showing the processing flow of the data processing system in Application Example 2 when combined with an emotion engine.

Mode for Carrying Out the Invention

[0016] Hereinafter, an example of an embodiment of the system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0017] First, the terms used in the following description will be explained.

[0018] In the following embodiments, the numbered processor (hereinafter simply referred to as "processor") may be one arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be one type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0019] In the following embodiments, the numbered RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0020] In the following embodiments, the numbered storage is one or more non-volatile storage devices that store various programs and various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes, and the like.

[0021] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0022] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0023] [First Embodiment]

[0024] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0025] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0026] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0027] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0028] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0029] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0030] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0031] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0032] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0033] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0034] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0035] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0036] The embodiments for carrying out the present invention will be described in detail below.

[0037] User clothing database management

[0038] The process begins with the user taking photos of their clothes and tagging each photo (e.g., "white T-shirt," "blue jeans," etc.). The device then sends the tagged photo data to a server, which stores the received data in a database. Using the tag information, the clothes are categorized by type, making subsequent processing easier.

[0039] Obtaining schedule and weather information

[0040] When a user enters their daily schedule into the device, the device stores the schedule information linked to the user's current location. Next, the device accesses a weather information API based on the location information to obtain the current weather information. This information is sent from the device to the server, which then prepares to generate the optimal outfit based on this information.

[0041] Coordination suggestions

[0042] The server analyzes the user's schedule, location, and weather information. Based on the analysis, it selects appropriate clothing from the user's database. For example, if the conditions are "business meeting" + "sunny, 25°C," the server will determine that "formal business attire is appropriate" and suggest a combination such as a black jacket, white T-shirt, and blue jeans. This suggestion is sent to the terminal and notified to the user.

[0043] Suggestions for missing items

[0044] Furthermore, the server consults the user's database to identify items needed for an outfit that the user does not possess. For example, if it determines that the user is missing business shoes, the server accesses an external database to search for the item. If a suitable product is found, it sends the details and a purchase link to the user's device, notifying them of the item. For example, the notification might say, "We have business leather shoes available. Purchase them here: [link]".

[0045] Coordination and purchase suggestion notification

[0046] The device clearly notifies the user of the outfit information it has received and the purchase suggestions for any missing items. The notified user can then check the details of the suggested outfit and, if necessary, purchase any missing items via the provided purchase link.

[0047] Specific example

[0048] For example, suppose a user receives the following outfit and purchase suggestions under the conditions of "business meeting" and "sunny, 25℃".

[0049] The device suggested an outfit consisting of a "black jacket," a "white T-shirt," and "blue jeans."

[0050] The device simultaneously displayed a notification stating, "We are short on business shoes. Please purchase them from the link below."

[0051] In this way, users can easily find the perfect outfit for the day and conveniently purchase any missing items as needed.

[0052] The above describes the embodiments for carrying out the present invention. This system not only frees users from the hassle of choosing the most suitable clothing based on their schedule and the weather for the day, but also allows them to easily purchase any missing items.

[0053] The following describes the processing flow.

[0054] Step 1:

[0055] Users take photos of the clothes they own and tag each photo (for example, "white T-shirt," "blue jeans," etc.).

[0056] Step 2:

[0057] The device sends tagged photo data to the server. The transmitted data is in JSON format and includes the photo and tags.

[0058] Step 3:

[0059] The server saves the received data to the database. This allows for systematic management of each user's belongings.

[0060] Step 4:

[0061] The user enters their schedule for the day into the device. Schedule information is obtained through the calendar function or direct input.

[0062] Step 5:

[0063] The device obtains the user's current location information using GPS functionality. The location information is expressed in the form of latitude and longitude.

[0064] Step 6:

[0065] The device accesses a weather information API based on its location to retrieve current weather information (e.g., temperature, weather conditions).

[0066] Step 7:

[0067] The device sends the acquired schedule and weather information to the server. The transmitted data is in JSON format and includes location information, schedule, and weather information.

[0068] Step 8:

[0069] The server analyzes the submitted schedule, location, and weather information to generate the most suitable outfit for the day. Based on the analysis results, the formality level of the clothing (business, casual, etc.) and seasonality are taken into consideration.

[0070] Step 9:

[0071] The server selects appropriate clothing from the user's database. For example, given the conditions "business meeting" + "sunny, 25°C", it would select the combination of "black jacket", "white T-shirt", and "blue jeans".

[0072] Step 10:

[0073] The server sends the generated outfit information to the terminal. The information sent includes the selected clothing items and their details.

[0074] Step 11:

[0075] The device notifies the user of the coordinated outfit information it has received. The notification is displayed using images and text for easy visual understanding.

[0076] Step 12:

[0077] The server refers to the user's database to identify items that are missing from their outfit. It determines specific missing items, such as "business shoes are missing."

[0078] Step 13:

[0079] The server uses an API from an external online shop to search for missing items. The search results include numerous candidate products and their detailed information.

[0080] Step 14:

[0081] The server retrieves product information and sends it to the terminal. This information includes product details, price, and a purchase link.

[0082] Step 15:

[0083] The device notifies the user of purchase offers it has received. The notification will display something like, "We have business shoes available. Purchase them here: [link]".

[0084] Step 16:

[0085] The user reviews the suggested outfit and purchases the suggested items if necessary. For example, they might click a purchase link to access an online shop and buy the items.

[0086] The above outlines the specific processing steps.

[0087] (Example 1)

[0088] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0089] Traditional outfit suggestion systems have been cumbersome to manage due to the complexities of the user's clothing database, and have struggled to efficiently suggest the most suitable outfit based on schedules and weather. Furthermore, they lacked sufficient methods for appropriately suggesting items missing from the suggested outfits, indicating room for improvement in user convenience.

[0090] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0091] In this invention, the server includes means for managing a database of clothing owned by the user; means for acquiring the user's location information and schedule information; means for acquiring weather information based on the acquired location information and schedule information; means for suggesting the optimal outfit combination based on the owned clothing, location information, schedule information, and weather information; means for searching for and suggesting items missing from the suggested outfit from an external source; means for tagging image data in the user's clothing database and classifying clothing based on the tag information; and means for generating prompt sentences to suggest outfits using a generative AI model. As a result, the user can efficiently manage their clothing, quickly receive suggestions for the optimal outfit based on their schedule and the weather, and easily purchase any missing items.

[0092] "User" refers to an individual who uses this system.

[0093] A "clothing database" refers to a database used to manage information about the clothes a user owns (such as photos, tags, and classification information).

[0094] "Location information" refers to data about the physical location where the user is currently located.

[0095] "Schedule information" refers to data about the day's plans and activities that the user has entered in advance.

[0096] "Weather information" refers to meteorological data obtained based on the user's current location.

[0097] "Tags" are keywords that users attach to photos of clothing they have taken, and they refer to information that makes it easier to classify and search for clothing.

[0098] "Optimal outfit combination" refers to the selection of clothing most suitable for the day, suggested based on the user's schedule, location, and weather information.

[0099] "External search" refers to the operation where a server uses the internet or other external databases to find items that the user does not possess.

[0100] A "generative AI model" refers to a model that uses generative artificial intelligence, and is used in this invention to suggest clothing.

[0101] A "prompt statement" is an input statement given to a generative AI model, referring to instructions or questions that guide the model to generate appropriate output.

[0102] A specific embodiment for implementing the system of the present invention will now be described. This system primarily involves data exchange between the user, terminal, and server to suggest the most suitable clothing for the user.

[0103] When users manage their clothing database, they first use a device such as a smartphone or tablet. Users take pictures of their clothes using the device's camera function. At that time, users add tags to each photo (for example, "white shirt," "blue pants," etc.). This tagging makes subsequent processing easier.

[0104] The device sends the captured tagged photo data to the server. The server stores the received data in a database and classifies the clothing by type based on the photos and tag information. MySQL® is one possible database management system to use.

[0105] Next, when the user enters their schedule for the day, the device saves this information and also retrieves the user's current location. Furthermore, the device uses an API to obtain weather information, such as the OpenWeatherMap API. This information is then sent from the device to the server.

[0106] The server analyzes the received schedule information, location information, and weather information. Based on the analysis results, it selects appropriate clothing from the user's clothing database and generates an optimal outfit. The specific analysis method includes analysis using a generative AI model. An example of a prompt statement to instruct this generative AI model is as follows.

[0107] "Please suggest appropriate attire for a business meeting from the user's clothing database. The weather is sunny and the temperature is 25°C."

[0108] The generated outfit is sent from the server to the terminal and notified to the user. At this time, any missing items in the suggested outfit are identified. The server refers to the user's database to determine the missing items and searches for appropriate products using an external database (e.g., Amazon API). The found products are sent to the terminal along with detailed information and a purchase link.

[0109] Finally, the device notifies the user of the received outfit information and purchase suggestions for any missing items. The user can then review the details of the suggested outfit and easily purchase any missing items via the purchase link.

[0110] This reduces the hassle for users in choosing the best outfit based on their schedule and the weather for the day, and also makes purchasing necessary items smoother.

[0111] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0112] Step 1:

[0113] The user takes a picture of their clothing. The input is the user's photo of the clothing and its tag information. The output is the tagged photo data saved on the device. The user uses their smartphone camera to take a picture of, for example, a "white shirt" and tags it "white shirt".

[0114] Step 2:

[0115] The device sends tagged photo data to the server. The input is the tagged photo data generated in step 1. The output is the data sent to the server. The device sends the photo data and tag information together to the cloud server.

[0116] Step 3:

[0117] The server receives data, stores it in a database, and classifies the clothing based on the tag information. The input is tagged photo data sent to the server. The output is the classified clothing data stored in the database. The server stores the photo data and tag information in a database such as MySQL, and classifies the clothing by category based on the tag information.

[0118] Step 4:

[0119] The user enters their schedule for the day into the device. The input is the user's schedule information. The output is the schedule information saved on the device. The user enters, for example, "Business meeting at 2 PM" into their smartphone or tablet.

[0120] Step 5:

[0121] The device obtains the user's current location information. The input is the current location information obtained from the device's location services. The output is the location information stored on the device. The device uses its GPS function to obtain the user's current location.

[0122] Step 6:

[0123] The device accesses a weather information API based on its location to obtain current weather information. The input is the user's location information. The output is the obtained weather information. The device uses the OpenWeatherMap API to obtain weather information such as "Tokyo, sunny, 25°C".

[0124] Step 7:

[0125] The terminal sends the acquired schedule information and weather information to the server. The input is the schedule information and weather information acquired in steps 4 and 6. The output is the information sent to the server. The terminal sends the schedule information and weather information to the server together.

[0126] Step 8:

[0127] The server analyzes the schedule information, location information, and weather information it receives. The input is the schedule information, location information, and weather information sent to the server. The output is the analysis result. Based on the schedule information and weather information, the server might determine, for example, "You need to choose appropriate attire for a business meeting."

[0128] Step 9:

[0129] The server selects appropriate clothing from the user's clothing database based on the analysis results and generates an optimal outfit. The input is schedule information, location information, and weather information analyzed by the server. The output is an optimal outfit suggestion. Using a generative AI model, the server suggests an outfit such as "black jacket, white shirt, blue pants."

[0130] Step 10:

[0131] The generated outfit information is sent to the terminal and notified to the user. The input is the outfit suggestion generated on the server. The output is the outfit information sent to the terminal. On the terminal, the user will see a notification such as, "Today's outfit is a black jacket, white shirt, and blue trousers."

[0132] Step 11:

[0133] The server identifies items needed for an outfit that the user does not possess and searches for those items by referencing an external database. The input is the user's clothing database stored on the server. The output is detailed information about the missing items and a purchase link. For example, if the server determines that "business leather shoes are missing," it will use an external API to find a purchase link for the appropriate product.

[0134] Step 12:

[0135] The system sends purchase information for the suggested missing items to the terminal and notifies the user. The input is the missing item information retrieved by the server from an external database. The output is the purchase information sent to the terminal. The terminal displays "You can purchase business leather shoes. Click this link: [link]".

[0136] This allows users to easily choose the perfect outfit for the day and conveniently purchase any missing items.

[0137] (Application Example 1)

[0138] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0139] Traditional systems offer limited suggestions based on user schedules and weather information, which is insufficient to meet the diverse needs of users. In particular, there is a need for more comprehensive suggestions tailored to users' lifestyles, encompassing not only clothing but also information on food and other items. Furthermore, the search and suggestion functions for items not owned by the user are lacking and need to be streamlined and improved.

[0140] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0141] In this invention, the server includes means for managing a database of items owned by the user, means for acquiring the user's location information and schedule information, means for acquiring external information based on the acquired location information and schedule information, means for making optimal suggestions based on the owned items, location information, schedule information and external information, means for searching for and suggesting items missing from the suggested content from external sources, and means for tagging image data taken by the user and classifying it based on the tag information. As a result, the user can comprehensively manage their belongings and external information and receive suggestions optimized for their lifestyle.

[0142] A "user" refers to an individual who uses the system to manage their belongings and information and receive optimal suggestions.

[0143] "Items" refers to a broad range of items owned by the user (such as clothing or food).

[0144] A "database" refers to an information collection used to store and manage information and image data of items owned by a user.

[0145] "Location information" refers to digital information that indicates the user's current location.

[0146] "Schedule information" refers to information about the user's future plans and schedules.

[0147] "External information" refers to information obtained from external data sources based on the user's location and schedule information, such as weather information and food-related information.

[0148] "Suggestion" refers to the act of recommending the most suitable actions or items based on the user's possessions and acquired external information.

[0149] "Searching" refers to the act of finding missing items that a user does not possess by searching an external database.

[0150] "Image data" refers to photographic data that users have taken and uploaded to the system.

[0151] A "tag" refers to a keyword that indicates attributes or classification information attached to image data.

[0152] "Classification" refers to the act of organizing image data or items into categories based on tag information.

[0153] Specific embodiments of this invention will be described in detail below.

[0154] System Overview

[0155] This invention is a system that manages items owned by a user (e.g., clothing, food, etc.) and makes optimal suggestions based on the user's location and schedule information. Furthermore, it is a system that searches for items that are missing from the suggested items and notifies the user.

[0156] Hardware and software to be used

[0157] Server: Provides the core computing and database management functions of this system. The server runs on Node.js and Express.

[0158] Database: MongoDB is used to store user-owned items and image data.

[0159] Weather Information API: Use the OpenWeatherMap API to obtain current weather information.

[0160] Food Delivery API: Use the Uber Eats API and DoorDash API to retrieve information about food delivery.

[0161] Using the image analysis module: TENSORFLOW(registered trademark).js, tag information is generated from images taken by the user.

[0162] Client device: Users use devices such as smartphones to input location and schedule information and receive suggestions.

[0163] Data flow and processing

[0164] 1. Management of the item database:

[0165] Users take photos of their belongings (e.g., clothing and food items) and tag them. The tagged photo data is sent to a server and stored in a database. This allows users to easily manage their possessions.

[0166] 2. Acquisition of location and schedule information:

[0167] Users input their daily schedule and current location via a device such as a smartphone. This information is sent to a server and stored for subsequent processing.

[0168] 3. Acquisition of external information:

[0169] The server accesses a weather information API based on location data to obtain current weather information. It also obtains food-related information from delivery service APIs.

[0170] 4. Generating the optimal proposal:

[0171] The server analyzes database information, location information, schedule information, and acquired external information to generate the most suitable suggestions for the user. For example, if the user has plans to go out on a nice day, it will suggest nearby food delivery services.

[0172] 5. Search for and suggest missing items:

[0173] If the proposed items are missing, the server accesses an external database to search for them. It then notifies the user of the detailed information and a purchase link.

[0174] Specific example

[0175] For example, if a user wants to eat pizza, they take a picture of it and tag it. The system analyzes the picture and adds it to the user's database. Then, on the day the user plans to go out, the system retrieves the current weather information and suggests the best food delivery service. At this time, the user will be notified with a suggestion such as, "It's sunny, so we recommend delivery from a nearby pizza restaurant. Check Uber Eats or DoorDash." Furthermore, if the system determines that there is a shortage of cheese, a notification will appear saying, "You can purchase cheese. Please purchase it from this link."

[0176] Example of a prompt

[0177] Based on the following information, please generate a program that suggests the most suitable food delivery options to the user.

[0178] Photos and tag information of dishes the user wants to eat.

[0179] Current location and weather information

[0180] Plans to go out

[0181] You also need to provide links to purchase any missing ingredients.

[0182] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0183] Step 1:

[0184] The system takes images of items owned by the user and adds tags to them. The user uses their smartphone to take a picture of an item they own (for example, a dish they want to eat). Then, they add tags such as "pizza" or "hamburger" to that picture. The input is image data and tag information, and the output is tagged image data. The device sends this data to the server.

[0185] Step 2:

[0186] The server receives tagged image data and stores it in a database. The server receives tagged image data sent from the terminal and stores it in a MongoDB database. The input is tagged image data, and the output is the data stored in the database. The image data is analyzed using TensorFlow.js to generate appropriate tags.

[0187] Step 3:

[0188] The user enters their daily schedule and current location into the device. The user uses a smartphone to enter their outing plans and current location. This information is sent to the server. The input is the schedule and location information, and the output is the schedule and location information sent to the server.

[0189] Step 4:

[0190] The server retrieves weather information based on location data. The server uses the location data to access the OpenWeatherMap API and retrieve current weather information. The input is location data, and the output is current weather information. The server saves the weather information to a database.

[0191] Step 5:

[0192] The server analyzes the user's database and acquired external information to generate optimal suggestions. The server analyzes item information, location information, schedule information, and weather information stored in the user's database to generate the most suitable suggestions for the user. For example, on a sunny day, a suggestion such as "You should order from a nearby pizza restaurant" might be generated to match the user's planned outing. The input consists of database information and external information, and the output is the generated suggestion.

[0193] Step 6:

[0194] The server searches for any missing items in the proposed content and generates detailed information and purchase links. The server accesses an external database to search for items missing from the proposed content. For example, if it determines that "cheese is missing," it will generate a purchase link. The input is the proposed content and information from the external database, and the output is detailed information and purchase links.

[0195] Step 7:

[0196] The server notifies the terminal of the proposed content and details of the missing items. The server sends the generated proposed content and details of the missing items to the terminal and notifies the user. The user receives the notification via their smartphone and can take appropriate action. The input is the proposed content and details of the missing items, and the output is the notification to the user.

[0197] Through these steps, users can efficiently manage their possessions and receive optimal suggestions based on external information. This process consistently involves data processing and calculations, providing information optimized for the user's lifestyle.

[0198] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0199] Regarding embodiments for carrying out the present invention, the details of a clothing selection support system that combines an emotion engine will be described.

[0200] User clothing database management

[0201] The process begins with users taking photos of their clothes and tagging each photo (e.g., "white T-shirt," "blue jeans," etc.). The tagged photo data is sent from the device to a server, which stores the received data in a database. This database is used to systematically manage each user's clothing.

[0202] Obtaining schedule and weather information

[0203] When a user enters their daily schedule into their device, the device stores the schedule information linked to the user's current location. Next, the device accesses a weather information API based on the location information to obtain current weather information (e.g., temperature, weather). This information is sent from the device to the server, which analyzes the received information and prepares to generate the optimal outfit.

[0204] Emotion recognition by an emotion engine

[0205] When a user uses the device, the built-in emotion engine recognizes emotions from the user's facial expressions and voice. This emotion data reflects the user's current mood and psychological state. The recognized emotion information is sent from the device to the server.

[0206] Coordination suggestions

[0207] The server analyzes the received schedule information, location information, weather information, and sentiment information. Based on the analysis results, it selects appropriate clothing from the user's database. For example, given conditions such as "business meeting" + "sunny, 25°C" + "user is feeling well," the server determines that "formal business attire is appropriate, and bright-colored items should be chosen," and suggests a combination such as a black jacket, white T-shirt, and blue jeans. The suggested outfit is sent to the terminal and notified to the user.

[0208] Suggestions for missing items

[0209] The server consults the user's database to identify items needed for an outfit that the user does not possess. For example, if it determines that the user is missing business shoes, the server accesses an external database to search for them. If a suitable product is found, it sends detailed information and a purchase link to the user's device and notifies them. The notification content is customized based on the emotions recognized by the emotion engine. For example, if the user is relaxed, a friendly message such as "We have some great business shoes. Purchase them here: link" will be displayed.

[0210] Coordination and purchase suggestion notification

[0211] The system visually and clearly notifies the user of outfit information received by the device and purchase suggestions for missing items. The notified information includes images and text, allowing the user to view the details. Furthermore, the message content is customized based on emotional information generated by an emotion engine, taking care to ensure that the user receives it in a more positive way.

[0212] Specific example

[0213] Suppose a user receives the following outfit and purchase suggestions under the conditions of "business meeting" + "sunny, 25℃" + "user is feeling well".

[0214] The device suggested an outfit consisting of a "black jacket," a "white T-shirt," and "blue jeans."

[0215] The device simultaneously displayed a notification stating, "We are short on business shoes. Please purchase them from the link below."

[0216] In this way, users can easily find the perfect outfit for the day and conveniently purchase any missing items as needed. Furthermore, because the emotion engine adapts notifications to the user's emotions, it is possible to use the system more comfortably.

[0217] The above describes the embodiments for carrying out the present invention. By combining the emotion engine, personalized suggestions tailored to the user's psychological state are realized, significantly improving the user experience.

[0218] The following describes the processing flow.

[0219] Step 1:

[0220] Users take photos of the clothes they own and tag each photo (for example, "white T-shirt," "blue jeans," etc.).

[0221] Step 2:

[0222] The device sends tagged photo data to the server. The transmitted data is in JSON format and includes the photo and tags.

[0223] Step 3:

[0224] The server saves the received data to the database. This allows for systematic management of each user's belongings.

[0225] Step 4:

[0226] The user enters their schedule for the day into the device. Schedule information is obtained through the calendar function or direct input.

[0227] Step 5:

[0228] The device obtains the user's current location information using GPS functionality. The location information is expressed in the form of latitude and longitude.

[0229] Step 6:

[0230] The device accesses a weather information API based on its location to retrieve current weather information (e.g., temperature, weather conditions).

[0231] Step 7:

[0232] The device sends the acquired schedule and weather information to the server. The transmitted data is in JSON format and includes location information, schedule, and weather information.

[0233] Step 8:

[0234] When a user uses the device, the built-in emotion engine recognizes emotions from the user's facial expressions and voice. This emotion data reflects the user's current mood and psychological state. Emotional information is transmitted from the device to the server.

[0235] Step 9:

[0236] The server analyzes the received schedule, location, weather, and sentiment information. Based on the analysis, it selects appropriate clothing from the user's database. For example, given the conditions "business meeting" + "sunny, 25°C" + "user is feeling well," the server would determine that "formal business attire is appropriate, and bright-colored items should be chosen."

[0237] Step 10:

[0238] The server sends generated outfit information (e.g., black jacket, white T-shirt, blue jeans) to the terminal. The information sent includes the selected clothing items and their details.

[0239] Step 11:

[0240] The device notifies the user of the outfit information it has received. The notification is displayed with images and text for easy visual understanding. In addition, the message is customized based on the results of the emotion engine's recognition. If the user is feeling energetic, a positive message such as "This outfit will help you have an energetic day!" will be displayed.

[0241] Step 12:

[0242] The server consults the user's database to identify items needed for an outfit that the user does not possess. If it determines that "business shoes are missing," the server accesses an external database to search for those items.

[0243] Step 13:

[0244] The server retrieves product information from an external database and sends it to the terminal. The transmitted information includes product details, price, and a purchase link.

[0245] Step 14:

[0246] The device notifies the user of purchase suggestions it has received. The notification content is customized based on the user's emotional state. For example, if the user is relaxed, a friendly message such as "We have some great business shoes. Purchase them here: link" will be displayed.

[0247] The above outlines the specific processing steps of the system that combines the emotion engine.

[0248] (Example 2)

[0249] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0250] In modern society, choosing appropriate attire amidst a busy daily life is difficult for many. Efficiently deciding on an outfit that suits the weather and schedule is particularly challenging. Furthermore, clothing choices are often influenced by emotions and psychological states, and these factors cannot be ignored. Conventional systems struggle to integrate these diverse factors to provide optimal suggestions, resulting in a poor user experience.

[0251] The identification processing performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for managing a database of clothing owned by the user, means for acquiring the user's location information and schedule information, means for acquiring weather information based on the acquired location information and schedule information, means for recognizing emotions from the user's facial expressions and voice, means for suggesting the optimal outfit combination based on the owned clothing, location information, schedule information, weather information and emotion information, and means for searching for and suggesting items missing from the suggested outfit from an external source. This enables the integration of diverse information to provide the user with personalized and optimal outfit suggestions.

[0252] "Means for managing a database of clothing owned by a user" refers to systems or software that collect, store, and systematically organize image data and tag information of clothing owned by a user.

[0253] "Means for acquiring user location and schedule information" refers to systems or software used to collect and store a user's current location and schedule for the day.

[0254] "Means for obtaining weather information based on acquired location and schedule information" refers to APIs or systems that obtain weather forecast information based on the user's location and schedule information.

[0255] "Means of recognizing emotions from a user's facial expressions and voice" refers to systems and software that analyze a user's facial expressions and voice to recognize the user's emotional state.

[0256] "Means of suggesting the optimal clothing combination based on the user's clothing, location information, schedule information, weather information, and emotional information" refers to algorithms and systems that integrate and analyze various user information to suggest the optimal clothing combination.

[0257] "Methods for searching for and suggesting items missing from a proposed outfit" refers to systems or algorithms that search an external database for items necessary for a proposed outfit that the user does not own, and then present the results to the user.

[0258] "Means of notifying users of suggested outfits and information on missing items" refers to systems or software that notify users of generated outfit suggestions and information on purchasing missing items.

[0259] "A means of attaching tags to image data of clothing owned by a user and classifying them based on that tag information" refers to a system or software that attaches specific tags to images of clothing taken by a user and categorizes them based on that tag information.

[0260] This invention details a clothing selection support system that incorporates an emotion engine. This system manages the user's clothing data and suggests optimal outfits based on external data such as schedule information and weather information. Furthermore, by recognizing the user's emotions, it provides individually optimized suggestions.

[0261] User clothing database management

[0262] The process begins with the user taking photos of their clothes and attaching tags. The tagged photos are sent from the device to a server, which then stores the received data in a database. This database is used to systematically manage each user's clothing. A smartphone or tablet is required to retrieve the data, and an internet connection is needed to transmit it.

[0263] Obtaining schedule and weather information

[0264] When a user enters their daily schedule into the device, the device stores the schedule information, linking it to the user's current location. This location information is obtained using GPS. Next, the device accesses a weather information API based on the location information to obtain current weather information (e.g., temperature, weather conditions). The obtained information is sent from the device to the server for preparation for analysis. Weather information APIs used include the OpenWeatherMap API and the Weather API.

[0265] Emotion recognition by an emotion engine

[0266] When a user uses the device, the built-in emotion engine (e.g., Microsoft® Azure® Emotion API or Google® Cloud Speech-to-Text) is activated to recognize emotions from the user's facial expressions and voice. This recognized emotion data is then sent from the device to the server.

[0267] Coordination suggestions

[0268] The server performs analysis based on the schedule information, location information, weather information, and sentiment information it receives. Based on this analysis, it selects appropriate clothing from the user's database. For example, given conditions such as "business meeting" + "sunny, 25°C" + "user is feeling well," the server will determine that "formal business attire is appropriate, and bright-colored items should be chosen," and suggest a combination such as a black jacket, white T-shirt, and blue jeans. The suggested result is sent to the terminal and notified to the user.

[0269] Suggestions for missing items

[0270] The server accesses the user's database to identify items needed for an outfit that the user does not possess. For example, if it determines that the user is missing business shoes, it accesses external databases (e.g., Amazon API, Rakuten Market API) to search for the item. If a suitable product is found, it sends the detailed information and purchase link to the user's device and notifies them.

[0271] Coordination and purchase suggestion notification

[0272] The device visually and clearly notifies the user of outfit information received and purchase suggestions for missing items. These notifications include images and text and are customized based on emotional information from an emotion engine. The user can review the suggestions and purchase items as needed.

[0273] Specific example

[0274] For example, consider a scenario where a user receives the following outfit suggestion based on the conditions "business meeting" + "sunny, 25℃" + "user is feeling well".

[0275] The device suggests outfits such as a "black jacket," "white T-shirt," and "blue jeans."

[0276] The device displays a notification that says, "We are short on business shoes. Please purchase them from the link below."

[0277] In this way, users can easily find the perfect outfit for the day and conveniently purchase any missing items as needed.

[0278] Examples of prompts for generative AI models

[0279] The following are examples of prompts to input into the generated AI model.

[0280] Please write a natural language description of a program that provides coordination suggestions to a user under the conditions of "business meeting," "sunny, 25℃," and "user is feeling well."

[0281] Thus, by implementing the present invention, personalized suggestions tailored to the user's psychological state are realized, significantly improving the user experience.

[0282] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0283] Step 1: Register clothing data

[0284] The user takes photos of their own clothes and tags each photo (e.g., "white T-shirt," "blue jeans"). The device then processes this tagged photo data and sends it to the server. The server parses the received data and stores it in a database.

[0285] Input: Photo of clothes and tag information

[0286] Output: Clothes data stored in the database

[0287] Specific operation: The user takes a photo of clothes with a smartphone. The terminal displays a tag input screen, and the user inputs tags. The terminal sends the image and tags to the server. The server stores them in the database.

[0288] Step 2: Obtaining schedule and location information

[0289] The user inputs the schedule for that day into the terminal. The terminal obtains the current location information (GPS) and associates and stores it with the schedule information.

[0290] Input: User's schedule information and location information

[0291] Output: Stored schedule information and location information

[0292] Specific operation: The user inputs the schedule into the terminal's calendar app. The terminal obtains the location information from GPS. The terminal stores the schedule information and location information.

[0293] Step 3: Obtaining weather information

[0294] The terminal accesses the weather information API (e.g., OpenWeatherMap API) based on the location information and obtains the current weather information (e.g., temperature, weather conditions).

[0295] Input: Location information

[0296] Output: Obtained weather information

[0297] Specific operation: The terminal accesses the weather information API and obtains the weather information. The obtained weather information is sent to the server.

[0298] Step 4: Performing emotion recognition

[0299] While the user is using the terminal, the built-in emotion engine (e.g., Microsoft Azure Emotion API) analyzes facial expressions and voice, and recognizes emotions. The recognized emotion data is sent from the terminal to the server.

[0300] Input: User's facial expression and voice data

[0301] Output: Recognized emotion data

[0302] Specific operation: The user takes a selfie with the terminal's camera or speaks towards the microphone. The terminal activates the emotion engine, analyzes facial expressions and voice, and sends the emotion data to the server.

[0303] Step 5: Generation of coordinate proposal

[0304] The server analyzes the received schedule information, location information, weather information, and emotion information to generate optimal coordinates. Selects suitable clothes from the user's database. For example, when conditions such as "business meeting" + "sunny, 25°C" + "the user is in a good mood" are given, it proposes an optimal combination (e.g., black jacket, white T-shirt, blue jeans).

[0305] Input: Schedule information, location information, weather information, emotion information

[0306] Output: Proposed coordinates

[0307] Specific operation: The server analyzes the received information, selects clothing from the user database, generates the proposed clothing, and sends the generation result to the terminal.

[0308] Step 6: Detection of missing items

[0309] The server uses the proposed outfit to determine if any items are missing by referencing the user's database. If missing items are found, it accesses an external database (e.g., Amazon API) to search for them and generates the results.

[0310] Input: Proposed outfits, user database

[0311] Output: Search results for missing items

[0312] Specific operation: The server checks the user's database. It identifies missing items. It accesses an external database to search for the items. It sends the search results to the terminal.

[0313] Step 7: Proposal and Purchase Notification

[0314] The device notifies the user of outfit information it has received and purchase suggestions for missing items. The notification includes images and text and is customized based on emotional information from the emotion engine.

[0315] Input: Suggested outfit, purchase information for missing items

[0316] Output: Visually customized notifications

[0317] Specific actions: The device receives suggestion information from the server. The suggested outfit is displayed as an image and text. Purchase links for missing items are displayed. The message is customized based on information from the emotion engine.

[0318] (Application Example 2)

[0319] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0320] In modern society, users often find it difficult to choose clothing that suits their diverse schedules and lifestyles. Furthermore, some users desire content recommendations based on their current mood and environment. However, no system exists to address these individual needs; there is a need for a system that reduces user effort and provides appropriate suggestions in a streamlined manner.

[0321] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0322] In this invention, the server includes means for managing a database of clothing owned by the user; means for acquiring the user's location information and schedule information; means for acquiring weather information based on the acquired location information and schedule information; means for suggesting the optimal outfit combination based on the owned clothing, location information, schedule information, and weather information; means for searching for and suggesting items missing from the suggested outfit from an external source; and means for recognizing the user's emotional state and recommending content based on that emotion. As a result, the user can not only easily choose the outfit best suited to their schedule and environment for the day, but also receive content recommendations that match their emotions.

[0323] "User" refers to a person or individual who uses this system.

[0324] A "clothing database" is a collection of data used to systematically store and manage information about various types of clothing owned by a user.

[0325] "Location information" refers to information that indicates the user's current location or geographical position.

[0326] "Schedule information" refers to information about plans and schedules that a user will have in the future.

[0327] "Weather information" refers to information about current or future weather and meteorological conditions.

[0328] "Optimal clothing combination" refers to the set of clothes that is most suitable for the user, based on conditions such as weather and schedule.

[0329] "Missing items" refer to clothing or accessories that the user does not currently own but are required for the suggested outfit combination.

[0330] "Emotional state" refers to the user's current emotional and psychological state.

[0331] "Content" is a general term for digital information and entertainment provided to users, such as videos, music, and text.

[0332] The system for implementing this invention manages a database of the user's clothing and suggests the optimal clothing combination and any missing items based on the user's location, schedule, weather, and emotional state. It also recommends digital content according to the user's emotional state.

[0333] User clothing database management

[0334] Users take photos of their clothes and tag each photo (e.g., "white T-shirt" or "blue jeans"). The tagged photo data is sent from the device to the server, which stores the received data in a database. This database systematically manages each user's clothing.

[0335] Acquisition of location and weather information

[0336] When a user enters their daily schedule into the device, the device stores the schedule information linked to the user's current location. Next, the device accesses a weather information API based on the location information to obtain current weather information (temperature and weather conditions). This information is sent from the device to the server, which analyzes the received information and prepares to generate the optimal outfit.

[0337] Emotion recognition by an emotion engine

[0338] When a user uses the device, the built-in emotion engine recognizes emotions from the user's facial expressions and voice. This emotion data reflects the user's current mood and psychological state. The recognized emotion information is sent from the device to the server.

[0339] Coordination suggestions and suggestions for missing items

[0340] The server analyzes the received schedule information, location information, weather information, and sentiment information. Based on the analysis results, it selects appropriate clothing from the user's database. For example, if given conditions such as "business meeting" + "sunny, 25°C" + "user is feeling well," the server will determine that "formal business attire is appropriate, and choose bright-colored items," and the suggested clothing will be sent to the terminal and notified to the user.

[0341] Furthermore, the server consults the user's database to identify items needed for an outfit that the user does not possess. For example, if it determines that the user is missing business shoes, the server accesses an external database to search for the item. If a suitable product is found, it sends detailed information and a purchase link to the user's device and notifies them. The notification content is customized based on the emotions recognized by the emotion engine.

[0342] Digital content recommendations

[0343] Furthermore, the server recommends content appropriate to the user based on their emotional state. For example, if the user is "happy," the server will recommend "comedy movies" or "funny videos." This recommendation is based on a combination of emotional data and weather information.

[0344] Hardware and software to be used

[0345] Hardware: Smartphones (such as Apple iPhone® and Samsung Galaxy)

[0346] Software: Python programs, weather information APIs (such as OpenWeatherMap), emotion recognition APIs (such as Microsoft Azure Emotion API)

[0347] Specific example

[0348] For example, under the conditions of "business meeting" + "sunny, 25℃" + "user is feeling well," a user might receive the following outfit and purchase suggestions.

[0349] The suggested outfit was a "black jacket," a "white T-shirt," and "blue jeans."

[0350] At the same time, a notification appears stating, "We are short on business shoes. Please purchase them from the link below."

[0351] As described above, users can easily find the perfect outfit for their day's schedule and easily obtain the necessary items. Furthermore, because the emotion engine adapts notifications to the user's emotions, they can use the system more comfortably.

[0352] Example of a prompt:

[0353] If the user's emotional state is "happy," the weather is "sunny," and the temperature is "25°C," please generate a list of recommended video content.

[0354] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0355] Step 1:

[0356] The user creates their own clothing database. They take photos of their clothes with their device and tag each photo. The tagged photo data is sent from the device to the server, where it is stored in the database. The input is the user's clothing photos and tags, and the output is the clothing information stored in the server's database.

[0357] Step 2:

[0358] The user enters their schedule for the day into the terminal. The terminal retrieves the entered schedule information and the user's current location. This information is then sent from the terminal to the server. The input is the user's schedule information and location, and the output is the schedule information and location sent to the server.

[0359] Step 3:

[0360] The server accesses a weather information API based on the received location information to obtain current weather information (temperature and weather conditions). The input is location information, and the output is weather information.

[0361] Step 4:

[0362] When a user uses the device, the device's built-in emotion engine analyzes the user's facial image and voice to perform emotion recognition. The recognized emotion information is sent from the device to the server. The input is the user's facial image or voice, and the output is the recognized emotion information.

[0363] Step 5:

[0364] The server analyzes the received schedule information, location information, weather information, and sentiment information. Based on this, it selects the most suitable clothing combination from the user's database. For example, if the user has a "business meeting," the weather is sunny at 25°C, and the user is feeling well, the server will determine that formal attire is appropriate. The input is schedule information, location information, weather information, and sentiment information, and the output is a suggestion for the most suitable clothing.

[0365] Step 6:

[0366] The server consults the user's clothing database to identify any missing items in the suggested outfit. For example, if "business shoes are missing," the server accesses an external database to search for those items. If a suitable product is found, it sends the details and a purchase link to the terminal and notifies the user. The input is the user's clothing database and the suggested outfit, and the output is the search results for the missing items and the notification.

[0367] Step 7:

[0368] The server recommends appropriate digital content based on the user's emotional state. For example, if the user is "happy," the server recommends "comedy movies" or "funny videos." The recommendation list is generated based on a combination of emotional data and weather information. The input is the user's emotional information and weather information, and the output is the recommended content list.

[0369] Step 8:

[0370] The device notifies the user of optimal clothing suggestions received from the server, purchase links for missing items, and recommended digital content. The notification content is displayed in a visually easy-to-understand format, allowing the user to review it. Input is suggestion data and recommended content from the server, and output is the notified information.

[0371] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0372] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0373] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0374] [Second Embodiment]

[0375] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0376] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0377] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0378] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0379] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0380] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0381] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0382] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0383] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0384] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0385] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0386] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0387] The embodiments for carrying out the present invention will be described in detail below.

[0388] User clothing database management

[0389] The process begins with the user taking photos of their clothes and tagging each photo (e.g., "white T-shirt," "blue jeans," etc.). The device then sends the tagged photo data to a server, which stores the received data in a database. Using the tag information, the clothes are categorized by type, making subsequent processing easier.

[0390] Obtaining schedule and weather information

[0391] When a user enters their daily schedule into the device, the device stores the schedule information linked to the user's current location. Next, the device accesses a weather information API based on the location information to obtain the current weather information. This information is sent from the device to the server, which then prepares to generate the optimal outfit based on this information.

[0392] Coordination suggestions

[0393] The server analyzes the user's schedule, location, and weather information. Based on the analysis, it selects appropriate clothing from the user's database. For example, if the conditions are "business meeting" + "sunny, 25°C," the server will determine that "formal business attire is appropriate" and suggest a combination such as a black jacket, white T-shirt, and blue jeans. This suggestion is sent to the terminal and notified to the user.

[0394] Suggestions for missing items

[0395] Furthermore, the server consults the user's database to identify items needed for an outfit that the user does not possess. For example, if it determines that the user is missing business shoes, the server accesses an external database to search for the item. If a suitable product is found, it sends the details and a purchase link to the user's device, notifying them of the item. For example, the notification might say, "We have business leather shoes available. Purchase them here: [link]".

[0396] Coordination and purchase suggestion notification

[0397] The device clearly notifies the user of the outfit information it has received and the purchase suggestions for any missing items. The notified user can then check the details of the suggested outfit and, if necessary, purchase any missing items via the provided purchase link.

[0398] Specific example

[0399] For example, suppose a user receives the following outfit and purchase suggestions under the conditions of "business meeting" and "sunny, 25℃".

[0400] The device suggested an outfit consisting of a "black jacket," a "white T-shirt," and "blue jeans."

[0401] The device simultaneously displayed a notification stating, "We are short on business shoes. Please purchase them from the link below."

[0402] In this way, users can easily find the perfect outfit for the day and conveniently purchase any missing items as needed.

[0403] The above describes the embodiments for carrying out the present invention. This system not only frees users from the hassle of choosing the most suitable clothing based on their schedule and the weather for the day, but also allows them to easily purchase any missing items.

[0404] The following describes the processing flow.

[0405] Step 1:

[0406] Users take photos of the clothes they own and tag each photo (for example, "white T-shirt," "blue jeans," etc.).

[0407] Step 2:

[0408] The device sends tagged photo data to the server. The transmitted data is in JSON format and includes the photo and tags.

[0409] Step 3:

[0410] The server saves the received data to the database. This allows for systematic management of each user's belongings.

[0411] Step 4:

[0412] The user enters their schedule for the day into the device. Schedule information is obtained through the calendar function or direct input.

[0413] Step 5:

[0414] The device obtains the user's current location information using GPS functionality. The location information is expressed in the form of latitude and longitude.

[0415] Step 6:

[0416] The device accesses a weather information API based on its location to retrieve current weather information (e.g., temperature, weather conditions).

[0417] Step 7:

[0418] The device sends the acquired schedule and weather information to the server. The transmitted data is in JSON format and includes location information, schedule, and weather information.

[0419] Step 8:

[0420] The server analyzes the submitted schedule, location, and weather information to generate the most suitable outfit for the day. Based on the analysis results, the formality level of the clothing (business, casual, etc.) and seasonality are taken into consideration.

[0421] Step 9:

[0422] The server selects appropriate clothing from the user's database. For example, given the conditions "business meeting" + "sunny, 25°C", it would select the combination of "black jacket", "white T-shirt", and "blue jeans".

[0423] Step 10:

[0424] The server sends the generated outfit information to the terminal. The information sent includes the selected clothing items and their details.

[0425] Step 11:

[0426] The device notifies the user of the coordinated outfit information it has received. The notification is displayed using images and text for easy visual understanding.

[0427] Step 12:

[0428] The server refers to the user's database to identify items that are missing from their outfit. It determines specific missing items, such as "business shoes are missing."

[0429] Step 13:

[0430] The server uses an API from an external online shop to search for missing items. The search results include numerous candidate products and their detailed information.

[0431] Step 14:

[0432] The server retrieves product information and sends it to the terminal. This information includes product details, price, and a purchase link.

[0433] Step 15:

[0434] The device notifies the user of purchase offers it has received. The notification will display something like, "We have business shoes available. Purchase them here: [link]".

[0435] Step 16:

[0436] The user reviews the suggested outfit and purchases the suggested items if necessary. For example, they might click a purchase link to access an online shop and buy the items.

[0437] The above outlines the specific processing steps.

[0438] (Example 1)

[0439] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0440] Traditional outfit suggestion systems have been cumbersome to manage due to the complexities of the user's clothing database, and have struggled to efficiently suggest the most suitable outfit based on schedules and weather. Furthermore, they lacked sufficient methods for appropriately suggesting items missing from the suggested outfits, indicating room for improvement in user convenience.

[0441] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0442] In this invention, the server includes means for managing a database of clothing owned by the user; means for acquiring the user's location information and schedule information; means for acquiring weather information based on the acquired location information and schedule information; means for suggesting the optimal outfit combination based on the owned clothing, location information, schedule information, and weather information; means for searching for and suggesting items missing from the suggested outfit from an external source; means for tagging image data in the user's clothing database and classifying clothing based on the tag information; and means for generating prompt sentences to suggest outfits using a generative AI model. As a result, the user can efficiently manage their clothing, quickly receive suggestions for the optimal outfit based on their schedule and the weather, and easily purchase any missing items.

[0443] "User" refers to an individual who uses this system.

[0444] A "clothing database" refers to a database used to manage information about the clothes a user owns (such as photos, tags, and classification information).

[0445] "Location information" refers to data about the physical location where the user is currently located.

[0446] "Schedule information" refers to data about the day's plans and activities that the user has entered in advance.

[0447] "Weather information" refers to meteorological data obtained based on the user's current location.

[0448] "Tags" are keywords that users attach to photos of clothing they have taken, and they refer to information that makes it easier to classify and search for clothing.

[0449] "Optimal outfit combination" refers to the selection of clothing most suitable for the day, suggested based on the user's schedule, location, and weather information.

[0450] "External search" refers to the operation where a server uses the internet or other external databases to find items that the user does not possess.

[0451] A "generative AI model" refers to a model that uses generative artificial intelligence, and is used in this invention to suggest clothing.

[0452] A "prompt statement" is an input statement given to a generative AI model, referring to instructions or questions that guide the model to generate appropriate output.

[0453] A specific embodiment for implementing the system of the present invention will now be described. This system primarily involves data exchange between the user, terminal, and server to suggest the most suitable clothing for the user.

[0454] When users manage their clothing database, they first use a device such as a smartphone or tablet. Users take pictures of their clothes using the device's camera function. At that time, users add tags to each photo (for example, "white shirt," "blue pants," etc.). This tagging makes subsequent processing easier.

[0455] The device sends the captured, tagged photo data to the server. The server stores the received data in a database and classifies the clothing by type based on the photos and tag information. MySQL is one possible database management system to use.

[0456] Next, when the user enters their schedule for the day, the device saves this information and also retrieves the user's current location. Furthermore, the device uses an API to obtain weather information, such as the OpenWeatherMap API. This information is then sent from the device to the server.

[0457] The server analyzes the received schedule information, location information, and weather information. Based on the analysis results, it selects appropriate clothing from the user's clothing database and generates an optimal outfit. The specific analysis method includes analysis using a generative AI model. An example of a prompt statement to instruct this generative AI model is as follows.

[0458] "Please suggest appropriate attire for a business meeting from the user's clothing database. The weather is sunny and the temperature is 25°C."

[0459] The generated outfit is sent from the server to the terminal and notified to the user. At this time, any missing items in the suggested outfit are identified. The server refers to the user's database to determine the missing items and searches for appropriate products using an external database (e.g., Amazon API). The found products are sent to the terminal along with detailed information and a purchase link.

[0460] Finally, the device notifies the user of the received outfit information and purchase suggestions for any missing items. The user can then review the details of the suggested outfit and easily purchase any missing items via the purchase link.

[0461] This reduces the hassle for users in choosing the best outfit based on their schedule and the weather for the day, and also makes purchasing necessary items smoother.

[0462] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0463] Step 1:

[0464] The user takes a picture of their clothing. The input is the user's photo of the clothing and its tag information. The output is the tagged photo data saved on the device. The user uses their smartphone camera to take a picture of, for example, a "white shirt" and tags it "white shirt".

[0465] Step 2:

[0466] The device sends tagged photo data to the server. The input is the tagged photo data generated in step 1. The output is the data sent to the server. The device sends the photo data and tag information together to the cloud server.

[0467] Step 3:

[0468] The server receives data, stores it in a database, and classifies the clothing based on the tag information. The input is tagged photo data sent to the server. The output is the classified clothing data stored in the database. The server stores the photo data and tag information in a database such as MySQL, and classifies the clothing by category based on the tag information.

[0469] Step 4:

[0470] The user enters their schedule for the day into the device. The input is the user's schedule information. The output is the schedule information saved on the device. The user enters, for example, "Business meeting at 2 PM" into their smartphone or tablet.

[0471] Step 5:

[0472] The device obtains the user's current location information. The input is the current location information obtained from the device's location services. The output is the location information stored on the device. The device uses its GPS function to obtain the user's current location.

[0473] Step 6:

[0474] The device accesses a weather information API based on its location to obtain current weather information. The input is the user's location information. The output is the obtained weather information. The device uses the OpenWeatherMap API to obtain weather information such as "Tokyo, sunny, 25°C".

[0475] Step 7:

[0476] The terminal sends the acquired schedule information and weather information to the server. The input is the schedule information and weather information acquired in steps 4 and 6. The output is the information sent to the server. The terminal sends the schedule information and weather information to the server together.

[0477] Step 8:

[0478] The server analyzes the schedule information, location information, and weather information it receives. The input is the schedule information, location information, and weather information sent to the server. The output is the analysis result. Based on the schedule information and weather information, the server might determine, for example, "You need to choose appropriate attire for a business meeting."

[0479] Step 9:

[0480] The server selects appropriate clothing from the user's clothing database based on the analysis results and generates an optimal outfit. The input is schedule information, location information, and weather information analyzed by the server. The output is an optimal outfit suggestion. Using a generative AI model, the server suggests an outfit such as "black jacket, white shirt, blue pants."

[0481] Step 10:

[0482] The generated outfit information is sent to the terminal and notified to the user. The input is the outfit suggestion generated on the server. The output is the outfit information sent to the terminal. On the terminal, the user will see a notification such as, "Today's outfit is a black jacket, white shirt, and blue trousers."

[0483] Step 11:

[0484] The server identifies items needed for an outfit that the user does not possess and searches for those items by referencing an external database. The input is the user's clothing database stored on the server. The output is detailed information about the missing items and a purchase link. For example, if the server determines that "business leather shoes are missing," it will use an external API to find a purchase link for the appropriate product.

[0485] Step 12:

[0486] The system sends purchase information for the suggested missing items to the terminal and notifies the user. The input is the missing item information retrieved by the server from an external database. The output is the purchase information sent to the terminal. The terminal displays "You can purchase business leather shoes. Click this link: [link]".

[0487] This allows users to easily choose the perfect outfit for the day and conveniently purchase any missing items.

[0488] (Application Example 1)

[0489] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0490] Traditional systems offer limited suggestions based on user schedules and weather information, which is insufficient to meet the diverse needs of users. In particular, there is a need for more comprehensive suggestions tailored to users' lifestyles, encompassing not only clothing but also information on food and other items. Furthermore, the search and suggestion functions for items not owned by the user are lacking and need to be streamlined and improved.

[0491] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0492] In this invention, the server includes means for managing a database of items owned by the user, means for acquiring the user's location information and schedule information, means for acquiring external information based on the acquired location information and schedule information, means for making optimal suggestions based on the owned items, location information, schedule information and external information, means for searching for and suggesting items missing from the suggested content from external sources, and means for tagging image data taken by the user and classifying it based on the tag information. As a result, the user can comprehensively manage their belongings and external information and receive suggestions optimized for their lifestyle.

[0493] A "user" refers to an individual who uses the system to manage their belongings and information and receive optimal suggestions.

[0494] "Items" refers to a broad range of items owned by the user (such as clothing or food).

[0495] A "database" refers to an information collection used to store and manage information and image data of items owned by a user.

[0496] "Location information" refers to digital information that indicates the user's current location.

[0497] "Schedule information" refers to information about the user's future plans and schedules.

[0498] "External information" refers to information obtained from external data sources based on the user's location and schedule information, such as weather information and food-related information.

[0499] "Suggestion" refers to the act of recommending the most suitable actions or items based on the user's possessions and acquired external information.

[0500] "Searching" refers to the act of finding missing items that a user does not possess by searching an external database.

[0501] "Image data" refers to photographic data that users have taken and uploaded to the system.

[0502] A "tag" refers to a keyword that indicates attributes or classification information attached to image data.

[0503] "Classification" refers to the act of organizing image data or items into categories based on tag information.

[0504] Specific embodiments of this invention will be described in detail below.

[0505] System Overview

[0506] This invention is a system that manages items owned by a user (e.g., clothing, food, etc.) and makes optimal suggestions based on the user's location and schedule information. Furthermore, it is a system that searches for items that are missing from the suggested items and notifies the user.

[0507] Hardware and software to be used

[0508] Server: Provides the core computing and database management functions of this system. The server runs on Node.js and Express.

[0509] Database: MongoDB is used to store user-owned items and image data.

[0510] Weather Information API: Use the OpenWeatherMap API to obtain current weather information.

[0511] Food Delivery API: Use the Uber Eats API and DoorDash API to retrieve information about food delivery.

[0512] Image analysis module: Using TensorFlow.js, tag information is generated from images taken by the user.

[0513] Client device: Users use devices such as smartphones to input location and schedule information and receive suggestions.

[0514] Data flow and processing

[0515] 1. Management of the item database:

[0516] Users take photos of their belongings (e.g., clothing and food items) and tag them. The tagged photo data is sent to a server and stored in a database. This allows users to easily manage their possessions.

[0517] 2. Acquisition of location and schedule information:

[0518] Users input their daily schedule and current location via a device such as a smartphone. This information is sent to a server and stored for subsequent processing.

[0519] 3. Acquisition of external information:

[0520] The server accesses a weather information API based on location data to obtain current weather information. It also obtains food-related information from delivery service APIs.

[0521] 4. Generating the optimal proposal:

[0522] The server analyzes database information, location information, schedule information, and acquired external information to generate the most suitable suggestions for the user. For example, if the user has plans to go out on a nice day, it will suggest nearby food delivery services.

[0523] 5. Search for and suggest missing items:

[0524] If the proposed items are missing, the server accesses an external database to search for them. It then notifies the user of the detailed information and a purchase link.

[0525] Specific example

[0526] For example, if a user wants to eat pizza, they take a picture of it and tag it. The system analyzes the picture and adds it to the user's database. Then, on the day the user plans to go out, the system retrieves the current weather information and suggests the best food delivery service. At this time, the user will be notified with a suggestion such as, "It's sunny, so we recommend delivery from a nearby pizza restaurant. Check Uber Eats or DoorDash." Furthermore, if the system determines that there is a shortage of cheese, a notification will appear saying, "You can purchase cheese. Please purchase it from this link."

[0527] Example of a prompt

[0528] Based on the following information, please generate a program that suggests the most suitable food delivery options to the user.

[0529] Photos and tag information of dishes the user wants to eat.

[0530] Current location and weather information

[0531] Plans to go out

[0532] You also need to provide links to purchase any missing ingredients.

[0533] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0534] Step 1:

[0535] The system takes images of items owned by the user and adds tags to them. The user uses their smartphone to take a picture of an item they own (for example, a dish they want to eat). Then, they add tags such as "pizza" or "hamburger" to that picture. The input is image data and tag information, and the output is tagged image data. The device sends this data to the server.

[0536] Step 2:

[0537] The server receives tagged image data and stores it in a database. The server receives tagged image data sent from the terminal and stores it in a MongoDB database. The input is tagged image data, and the output is the data stored in the database. The image data is analyzed using TensorFlow.js to generate appropriate tags.

[0538] Step 3:

[0539] The user enters their daily schedule and current location into the device. The user uses a smartphone to enter their outing plans and current location. This information is sent to the server. The input is the schedule and location information, and the output is the schedule and location information sent to the server.

[0540] Step 4:

[0541] The server retrieves weather information based on location data. The server uses the location data to access the OpenWeatherMap API and retrieve current weather information. The input is location data, and the output is current weather information. The server saves the weather information to a database.

[0542] Step 5:

[0543] The server analyzes the user's database and acquired external information to generate optimal suggestions. The server analyzes item information, location information, schedule information, and weather information stored in the user's database to generate the most suitable suggestions for the user. For example, on a sunny day, a suggestion such as "You should order from a nearby pizza restaurant" might be generated to match the user's planned outing. The input consists of database information and external information, and the output is the generated suggestion.

[0544] Step 6:

[0545] The server searches for any missing items in the proposed content and generates detailed information and purchase links. The server accesses an external database to search for items missing from the proposed content. For example, if it determines that "cheese is missing," it will generate a purchase link. The input is the proposed content and information from the external database, and the output is detailed information and purchase links.

[0546] Step 7:

[0547] The server notifies the terminal of the proposed content and details of the missing items. The server sends the generated proposed content and details of the missing items to the terminal and notifies the user. The user receives the notification via their smartphone and can take appropriate action. The input is the proposed content and details of the missing items, and the output is the notification to the user.

[0548] Through these steps, users can efficiently manage their possessions and receive optimal suggestions based on external information. This process consistently involves data processing and calculations, providing information optimized for the user's lifestyle.

[0549] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0550] Regarding embodiments for carrying out the present invention, the details of a clothing selection support system that combines an emotion engine will be described.

[0551] User clothing database management

[0552] The process begins with users taking photos of their clothes and tagging each photo (e.g., "white T-shirt," "blue jeans," etc.). The tagged photo data is sent from the device to a server, which stores the received data in a database. This database is used to systematically manage each user's clothing.

[0553] Obtaining schedule and weather information

[0554] When a user enters their daily schedule into their device, the device stores the schedule information linked to the user's current location. Next, the device accesses a weather information API based on the location information to obtain current weather information (e.g., temperature, weather). This information is sent from the device to the server, which analyzes the received information and prepares to generate the optimal outfit.

[0555] Emotion recognition by an emotion engine

[0556] When a user uses the device, the built-in emotion engine recognizes emotions from the user's facial expressions and voice. This emotion data reflects the user's current mood and psychological state. The recognized emotion information is sent from the device to the server.

[0557] Coordination suggestions

[0558] The server analyzes the received schedule information, location information, weather information, and sentiment information. Based on the analysis results, it selects appropriate clothing from the user's database. For example, given conditions such as "business meeting" + "sunny, 25°C" + "user is feeling well," the server determines that "formal business attire is appropriate, and bright-colored items should be chosen," and suggests a combination such as a black jacket, white T-shirt, and blue jeans. The suggested outfit is sent to the terminal and notified to the user.

[0559] Suggestions for missing items

[0560] The server consults the user's database to identify items needed for an outfit that the user does not possess. For example, if it determines that the user is missing business shoes, the server accesses an external database to search for them. If a suitable product is found, it sends detailed information and a purchase link to the user's device and notifies them. The notification content is customized based on the emotions recognized by the emotion engine. For example, if the user is relaxed, a friendly message such as "We have some great business shoes. Purchase them here: link" will be displayed.

[0561] Coordination and purchase suggestion notification

[0562] The system visually and clearly notifies the user of outfit information received by the device and purchase suggestions for missing items. The notified information includes images and text, allowing the user to view the details. Furthermore, the message content is customized based on emotional information generated by an emotion engine, taking care to ensure that the user receives it in a more positive way.

[0563] Specific example

[0564] Suppose a user receives the following outfit and purchase suggestions under the conditions of "business meeting" + "sunny, 25℃" + "user is feeling well".

[0565] The device suggested an outfit consisting of a "black jacket," a "white T-shirt," and "blue jeans."

[0566] The device simultaneously displayed a notification stating, "We are short on business shoes. Please purchase them from the link below."

[0567] In this way, users can easily find the perfect outfit for the day and conveniently purchase any missing items as needed. Furthermore, because the emotion engine adapts notifications to the user's emotions, it is possible to use the system more comfortably.

[0568] The above describes the embodiments for carrying out the present invention. By combining the emotion engine, personalized suggestions tailored to the user's psychological state are realized, significantly improving the user experience.

[0569] The following describes the processing flow.

[0570] Step 1:

[0571] Users take photos of the clothes they own and tag each photo (for example, "white T-shirt," "blue jeans," etc.).

[0572] Step 2:

[0573] The device sends tagged photo data to the server. The transmitted data is in JSON format and includes the photo and tags.

[0574] Step 3:

[0575] The server saves the received data to the database. This allows for systematic management of each user's belongings.

[0576] Step 4:

[0577] The user enters their schedule for the day into the device. Schedule information is obtained through the calendar function or direct input.

[0578] Step 5:

[0579] The device obtains the user's current location information using GPS functionality. The location information is expressed in the form of latitude and longitude.

[0580] Step 6:

[0581] The device accesses a weather information API based on its location to retrieve current weather information (e.g., temperature, weather conditions).

[0582] Step 7:

[0583] The device sends the acquired schedule and weather information to the server. The transmitted data is in JSON format and includes location information, schedule, and weather information.

[0584] Step 8:

[0585] When a user uses the device, the built-in emotion engine recognizes emotions from the user's facial expressions and voice. This emotion data reflects the user's current mood and psychological state. Emotional information is transmitted from the device to the server.

[0586] Step 9:

[0587] The server analyzes the received schedule, location, weather, and sentiment information. Based on the analysis, it selects appropriate clothing from the user's database. For example, given the conditions "business meeting" + "sunny, 25°C" + "user is feeling well," the server would determine that "formal business attire is appropriate, and bright-colored items should be chosen."

[0588] Step 10:

[0589] The server sends generated outfit information (e.g., black jacket, white T-shirt, blue jeans) to the terminal. The information sent includes the selected clothing items and their details.

[0590] Step 11:

[0591] The device notifies the user of the outfit information it has received. The notification is displayed with images and text for easy visual understanding. In addition, the message is customized based on the results of the emotion engine's recognition. If the user is feeling energetic, a positive message such as "This outfit will help you have an energetic day!" will be displayed.

[0592] Step 12:

[0593] The server consults the user's database to identify items needed for an outfit that the user does not possess. If it determines that "business shoes are missing," the server accesses an external database to search for those items.

[0594] Step 13:

[0595] The server retrieves product information from an external database and sends it to the terminal. The transmitted information includes product details, price, and a purchase link.

[0596] Step 14:

[0597] The device notifies the user of purchase suggestions it has received. The notification content is customized based on the user's emotional state. For example, if the user is relaxed, a friendly message such as "We have some great business shoes. Purchase them here: link" will be displayed.

[0598] The above outlines the specific processing steps of the system that combines the emotion engine.

[0599] (Example 2)

[0600] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0601] In modern society, choosing appropriate attire amidst a busy daily life is difficult for many. Efficiently deciding on an outfit that suits the weather and schedule is particularly challenging. Furthermore, clothing choices are often influenced by emotions and psychological states, and these factors cannot be ignored. Conventional systems struggle to integrate these diverse factors to provide optimal suggestions, resulting in a poor user experience.

[0602] The identification processing performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for managing a database of clothing owned by the user, means for acquiring the user's location information and schedule information, means for acquiring weather information based on the acquired location information and schedule information, means for recognizing emotions from the user's facial expressions and voice, means for suggesting the optimal outfit combination based on the owned clothing, location information, schedule information, weather information and emotion information, and means for searching for and suggesting items missing from the suggested outfit from an external source. This enables the integration of diverse information to provide the user with personalized and optimal outfit suggestions.

[0603] "Means for managing a database of clothing owned by a user" refers to systems or software that collect, store, and systematically organize image data and tag information of clothing owned by a user.

[0604] "Means for acquiring user location and schedule information" refers to systems or software used to collect and store a user's current location and schedule for the day.

[0605] "Means for obtaining weather information based on acquired location and schedule information" refers to APIs or systems that obtain weather forecast information based on the user's location and schedule information.

[0606] "Means of recognizing emotions from a user's facial expressions and voice" refers to systems and software that analyze a user's facial expressions and voice to recognize the user's emotional state.

[0607] "Means of suggesting the optimal clothing combination based on the user's clothing, location information, schedule information, weather information, and emotional information" refers to algorithms and systems that integrate and analyze various user information to suggest the optimal clothing combination.

[0608] "Methods for searching for and suggesting items missing from a proposed outfit" refers to systems or algorithms that search an external database for items necessary for a proposed outfit that the user does not own, and then present the results to the user.

[0609] "Means of notifying users of suggested outfits and information on missing items" refers to systems or software that notify users of generated outfit suggestions and information on purchasing missing items.

[0610] "A means of attaching tags to image data of clothing owned by a user and classifying them based on that tag information" refers to a system or software that attaches specific tags to images of clothing taken by a user and categorizes them based on that tag information.

[0611] This invention details a clothing selection support system that incorporates an emotion engine. This system manages the user's clothing data and suggests optimal outfits based on external data such as schedule information and weather information. Furthermore, by recognizing the user's emotions, it provides individually optimized suggestions.

[0612] User clothing database management

[0613] The process begins with the user taking photos of their clothes and attaching tags. The tagged photos are sent from the device to a server, which then stores the received data in a database. This database is used to systematically manage each user's clothing. A smartphone or tablet is required to retrieve the data, and an internet connection is needed to transmit it.

[0614] Obtaining schedule and weather information

[0615] When a user enters their daily schedule into the device, the device stores the schedule information, linking it to the user's current location. This location information is obtained using GPS. Next, the device accesses a weather information API based on the location information to obtain current weather information (e.g., temperature, weather conditions). The obtained information is sent from the device to the server for preparation for analysis. Weather information APIs used include the OpenWeatherMap API and the Weather API.

[0616] Emotion recognition by an emotion engine

[0617] When a user uses a device, the built-in emotion engine (e.g., Microsoft Azure's Emotion API or Google Cloud's Speech-to-Text) is activated to recognize emotions from the user's facial expressions and voice. This recognized emotion data is then sent from the device to a server.

[0618] Coordination suggestions

[0619] The server performs analysis based on the schedule information, location information, weather information, and sentiment information it receives. Based on this analysis, it selects appropriate clothing from the user's database. For example, given conditions such as "business meeting" + "sunny, 25°C" + "user is feeling well," the server will determine that "formal business attire is appropriate, and bright-colored items should be chosen," and suggest a combination such as a black jacket, white T-shirt, and blue jeans. The suggested result is sent to the terminal and notified to the user.

[0620] Suggestions for missing items

[0621] The server accesses the user's database to identify items needed for an outfit that the user does not possess. For example, if it determines that the user is missing business shoes, it accesses external databases (e.g., Amazon API, Rakuten Market API) to search for the item. If a suitable product is found, it sends the detailed information and purchase link to the user's device and notifies them.

[0622] Coordination and purchase suggestion notification

[0623] The device visually and clearly notifies the user of outfit information received and purchase suggestions for missing items. These notifications include images and text and are customized based on emotional information from an emotion engine. The user can review the suggestions and purchase items as needed.

[0624] Specific example

[0625] For example, consider a scenario where a user receives the following outfit suggestion based on the conditions "business meeting" + "sunny, 25℃" + "user is feeling well".

[0626] The device suggests outfits such as a "black jacket," "white T-shirt," and "blue jeans."

[0627] The device displays a notification that says, "We are short on business shoes. Please purchase them from the link below."

[0628] In this way, users can easily find the perfect outfit for the day and conveniently purchase any missing items as needed.

[0629] Examples of prompts for generative AI models

[0630] The following are examples of prompts to input into the generated AI model.

[0631] Please write a natural language description of a program that provides coordination suggestions to a user under the conditions of "business meeting," "sunny, 25℃," and "user is feeling well."

[0632] Thus, by implementing the present invention, personalized suggestions tailored to the user's psychological state are realized, significantly improving the user experience.

[0633] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0634] Step 1: Register clothing data

[0635] The user takes photos of their own clothes and tags each photo (e.g., "white T-shirt," "blue jeans"). The device then processes this tagged photo data and sends it to the server. The server parses the received data and stores it in a database.

[0636] Input: Photos and tag information of clothing

[0637] Output: Clothing data stored in the database

[0638] Specific operation: The user takes a photo of clothing with their smartphone. The device displays a tag input screen, and the user enters the tags. The device sends the image and tags to the server. The server saves them to the database.

[0639] Step 2: Obtain schedule and location information

[0640] The user enters their schedule for the day into their device. The device obtains the user's current location information (GPS) and stores it in conjunction with the schedule information.

[0641] Input: User's schedule information and location information

[0642] Output: Saved schedule information and location information

[0643] Specific operation: The user enters an event into the device's calendar app. The device obtains location information from GPS. The device saves the event information and location information.

[0644] Step 3: Obtain weather information

[0645] The device accesses a weather information API (e.g., OpenWeatherMap API) based on its location to obtain current weather information (e.g., temperature, weather conditions).

[0646] Input: Location information

[0647] Output: Acquired weather information

[0648] Specific operation: The device accesses the weather information API and retrieves weather information. The retrieved weather information is sent to the server.

[0649] Step 4: Performing emotion recognition

[0650] While the user is using the device, the built-in emotion engine (e.g., Microsoft Azure Emotion API) analyzes facial expressions and voice to recognize emotions. The recognized emotion data is then sent from the device to the server.

[0651] Input: User's facial expressions and voice data

[0652] Output: Recognized emotion data

[0653] Specific operation: The user takes a selfie with the device's camera or speaks into the microphone. The device activates its emotion engine and analyzes facial expressions and voice. The emotion data is sent to the server.

[0654] Step 5: Generating Coordination Proposals

[0655] The server analyzes the received schedule information, location information, weather information, and sentiment information to generate the optimal outfit. It selects suitable clothing from the user's database. For example, given conditions such as "business meeting" + "sunny, 25℃" + "user is feeling good," it will suggest the optimal combination (e.g., black jacket, white T-shirt, blue jeans).

[0656] Input: Schedule information, location information, weather information, sentiment information

[0657] Output: Proposed coordination

[0658] Specific operation: The server analyzes the received information. It selects clothing from the user database. It generates the suggested clothing. It sends the generated result to the terminal.

[0659] Step 6: Detecting missing items

[0660] The server uses the proposed outfit to determine if any items are missing by referencing the user's database. If missing items are found, it accesses an external database (e.g., Amazon API) to search for them and generates the results.

[0661] Input: Proposed outfits, user database

[0662] Output: Search results for missing items

[0663] Specific operation: The server checks the user's database. It identifies missing items. It accesses an external database to search for the items. It sends the search results to the terminal.

[0664] Step 7: Proposal and Purchase Notification

[0665] The device notifies the user of outfit information it has received and purchase suggestions for missing items. The notification includes images and text and is customized based on emotional information from the emotion engine.

[0666] Input: Suggested outfit, purchase information for missing items

[0667] Output: Visually customized notifications

[0668] Specific actions: The device receives suggestion information from the server. The suggested outfit is displayed as an image and text. Purchase links for missing items are displayed. The message is customized based on information from the emotion engine.

[0669] (Application Example 2)

[0670] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0671] In modern society, users often find it difficult to choose clothing that suits their diverse schedules and lifestyles. Furthermore, some users desire content recommendations based on their current mood and environment. However, no system exists to address these individual needs; there is a need for a system that reduces user effort and provides appropriate suggestions in a streamlined manner.

[0672] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0673] In this invention, the server includes means for managing a database of clothing owned by the user; means for acquiring the user's location information and schedule information; means for acquiring weather information based on the acquired location information and schedule information; means for suggesting the optimal outfit combination based on the owned clothing, location information, schedule information, and weather information; means for searching for and suggesting items missing from the suggested outfit from an external source; and means for recognizing the user's emotional state and recommending content based on that emotion. As a result, the user can not only easily choose the outfit best suited to their schedule and environment for the day, but also receive content recommendations that match their emotions.

[0674] "User" refers to a person or individual who uses this system.

[0675] A "clothing database" is a collection of data used to systematically store and manage information about various types of clothing owned by a user.

[0676] "Location information" refers to information that indicates the user's current location or geographical position.

[0677] "Schedule information" refers to information about plans and schedules that a user will have in the future.

[0678] "Weather information" refers to information about current or future weather and meteorological conditions.

[0679] "Optimal clothing combination" refers to the set of clothes that is most suitable for the user, based on conditions such as weather and schedule.

[0680] "Missing items" refer to clothing or accessories that the user does not currently own but are required for the suggested outfit combination.

[0681] "Emotional state" refers to the user's current emotional and psychological state.

[0682] "Content" is a general term for digital information and entertainment provided to users, such as videos, music, and text.

[0683] The system for implementing this invention manages a database of the user's clothing and suggests the optimal clothing combination and any missing items based on the user's location, schedule, weather, and emotional state. It also recommends digital content according to the user's emotional state.

[0684] User clothing database management

[0685] Users take photos of their clothes and tag each photo (e.g., "white T-shirt" or "blue jeans"). The tagged photo data is sent from the device to the server, which stores the received data in a database. This database systematically manages each user's clothing.

[0686] Acquisition of location and weather information

[0687] When a user enters their daily schedule into the device, the device stores the schedule information linked to the user's current location. Next, the device accesses a weather information API based on the location information to obtain current weather information (temperature and weather conditions). This information is sent from the device to the server, which analyzes the received information and prepares to generate the optimal outfit.

[0688] Emotion recognition by an emotion engine

[0689] When a user uses the device, the built-in emotion engine recognizes emotions from the user's facial expressions and voice. This emotion data reflects the user's current mood and psychological state. The recognized emotion information is sent from the device to the server.

[0690] Coordination suggestions and suggestions for missing items

[0691] The server analyzes the received schedule information, location information, weather information, and sentiment information. Based on the analysis results, it selects appropriate clothing from the user's database. For example, if given conditions such as "business meeting" + "sunny, 25°C" + "user is feeling well," the server will determine that "formal business attire is appropriate, and choose bright-colored items," and the suggested clothing will be sent to the terminal and notified to the user.

[0692] Furthermore, the server consults the user's database to identify items needed for an outfit that the user does not possess. For example, if it determines that the user is missing business shoes, the server accesses an external database to search for the item. If a suitable product is found, it sends detailed information and a purchase link to the user's device and notifies them. The notification content is customized based on the emotions recognized by the emotion engine.

[0693] Digital content recommendations

[0694] Furthermore, the server recommends content appropriate to the user based on their emotional state. For example, if the user is "happy," the server will recommend "comedy movies" or "funny videos." This recommendation is based on a combination of emotional data and weather information.

[0695] Hardware and software to be used

[0696] Hardware: Smartphones (such as Apple iPhone and Samsung Galaxy)

[0697] Software: Python programs, weather information APIs (such as OpenWeatherMap), emotion recognition APIs (such as Microsoft Azure Emotion API)

[0698] Specific example

[0699] For example, under the conditions of "business meeting" + "sunny, 25℃" + "user is feeling well," a user might receive the following outfit and purchase suggestions.

[0700] The suggested outfit was a "black jacket," a "white T-shirt," and "blue jeans."

[0701] At the same time, a notification appears stating, "We are short on business shoes. Please purchase them from the link below."

[0702] As described above, users can easily find the perfect outfit for their day's schedule and easily obtain the necessary items. Furthermore, because the emotion engine adapts notifications to the user's emotions, they can use the system more comfortably.

[0703] Example of a prompt:

[0704] If the user's emotional state is "happy," the weather is "sunny," and the temperature is "25°C," please generate a list of recommended video content.

[0705] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0706] Step 1:

[0707] The user creates their own clothing database. They take photos of their clothes with their device and tag each photo. The tagged photo data is sent from the device to the server, where it is stored in the database. The input is the user's clothing photos and tags, and the output is the clothing information stored in the server's database.

[0708] Step 2:

[0709] The user enters their schedule for the day into the terminal. The terminal retrieves the entered schedule information and the user's current location. This information is then sent from the terminal to the server. The input is the user's schedule information and location, and the output is the schedule information and location sent to the server.

[0710] Step 3:

[0711] The server accesses a weather information API based on the received location information to obtain current weather information (temperature and weather conditions). The input is location information, and the output is weather information.

[0712] Step 4:

[0713] When a user uses the device, the device's built-in emotion engine analyzes the user's facial image and voice to perform emotion recognition. The recognized emotion information is sent from the device to the server. The input is the user's facial image or voice, and the output is the recognized emotion information.

[0714] Step 5:

[0715] The server analyzes the received schedule information, location information, weather information, and sentiment information. Based on this, it selects the most suitable clothing combination from the user's database. For example, if the user has a "business meeting," the weather is sunny at 25°C, and the user is feeling well, the server will determine that formal attire is appropriate. The input is schedule information, location information, weather information, and sentiment information, and the output is a suggestion for the most suitable clothing.

[0716] Step 6:

[0717] The server consults the user's clothing database to identify any missing items in the suggested outfit. For example, if "business shoes are missing," the server accesses an external database to search for those items. If a suitable product is found, it sends the details and a purchase link to the terminal and notifies the user. The input is the user's clothing database and the suggested outfit, and the output is the search results for the missing items and the notification.

[0718] Step 7:

[0719] The server recommends appropriate digital content based on the user's emotional state. For example, if the user is "happy," the server recommends "comedy movies" or "funny videos." The recommendation list is generated based on a combination of emotional data and weather information. The input is the user's emotional information and weather information, and the output is the recommended content list.

[0720] Step 8:

[0721] The device notifies the user of optimal clothing suggestions received from the server, purchase links for missing items, and recommended digital content. The notification content is displayed in a visually easy-to-understand format, allowing the user to review it. Input is suggestion data and recommended content from the server, and output is the notified information.

[0722] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0723] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0724] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0725] [Third Embodiment]

[0726] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0727] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0728] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0729] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0730] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0731] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0732] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0733] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0734] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0735] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0736] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0737] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0738] The embodiments for carrying out the present invention will be described in detail below.

[0739] User clothing database management

[0740] The process begins with the user taking photos of their clothes and tagging each photo (e.g., "white T-shirt," "blue jeans," etc.). The device then sends the tagged photo data to a server, which stores the received data in a database. Using the tag information, the clothes are categorized by type, making subsequent processing easier.

[0741] Obtaining schedule and weather information

[0742] When a user enters their daily schedule into the device, the device stores the schedule information linked to the user's current location. Next, the device accesses a weather information API based on the location information to obtain the current weather information. This information is sent from the device to the server, which then prepares to generate the optimal outfit based on this information.

[0743] Coordination suggestions

[0744] The server analyzes the user's schedule, location, and weather information. Based on the analysis, it selects appropriate clothing from the user's database. For example, if the conditions are "business meeting" + "sunny, 25°C," the server will determine that "formal business attire is appropriate" and suggest a combination such as a black jacket, white T-shirt, and blue jeans. This suggestion is sent to the terminal and notified to the user.

[0745] Suggestions for missing items

[0746] Furthermore, the server consults the user's database to identify items needed for an outfit that the user does not possess. For example, if it determines that the user is missing business shoes, the server accesses an external database to search for the item. If a suitable product is found, it sends the details and a purchase link to the user's device, notifying them of the item. For example, the notification might say, "We have business leather shoes available. Purchase them here: [link]".

[0747] Coordination and purchase suggestion notification

[0748] The device clearly notifies the user of the outfit information it has received and the purchase suggestions for any missing items. The notified user can then check the details of the suggested outfit and, if necessary, purchase any missing items via the provided purchase link.

[0749] Specific example

[0750] For example, suppose a user receives the following outfit and purchase suggestions under the conditions of "business meeting" and "sunny, 25℃".

[0751] The device suggested an outfit consisting of a "black jacket," a "white T-shirt," and "blue jeans."

[0752] The device simultaneously displayed a notification stating, "We are short on business shoes. Please purchase them from the link below."

[0753] In this way, users can easily find the perfect outfit for the day and conveniently purchase any missing items as needed.

[0754] The above describes the embodiments for carrying out the present invention. This system not only frees users from the hassle of choosing the most suitable clothing based on their schedule and the weather for the day, but also allows them to easily purchase any missing items.

[0755] The following describes the processing flow.

[0756] Step 1:

[0757] Users take photos of the clothes they own and tag each photo (for example, "white T-shirt," "blue jeans," etc.).

[0758] Step 2:

[0759] The device sends tagged photo data to the server. The transmitted data is in JSON format and includes the photo and tags.

[0760] Step 3:

[0761] The server saves the received data to the database. This allows for systematic management of each user's belongings.

[0762] Step 4:

[0763] The user enters their schedule for the day into the device. Schedule information is obtained through the calendar function or direct input.

[0764] Step 5:

[0765] The device obtains the user's current location information using GPS functionality. The location information is expressed in the form of latitude and longitude.

[0766] Step 6:

[0767] The device accesses a weather information API based on its location to retrieve current weather information (e.g., temperature, weather conditions).

[0768] Step 7:

[0769] The device sends the acquired schedule and weather information to the server. The transmitted data is in JSON format and includes location information, schedule, and weather information.

[0770] Step 8:

[0771] The server analyzes the submitted schedule, location, and weather information to generate the most suitable outfit for the day. Based on the analysis results, the formality level of the clothing (business, casual, etc.) and seasonality are taken into consideration.

[0772] Step 9:

[0773] The server selects appropriate clothing from the user's database. For example, given the conditions "business meeting" + "sunny, 25°C", it would select the combination of "black jacket", "white T-shirt", and "blue jeans".

[0774] Step 10:

[0775] The server sends the generated outfit information to the terminal. The information sent includes the selected clothing items and their details.

[0776] Step 11:

[0777] The device notifies the user of the coordinated outfit information it has received. The notification is displayed using images and text for easy visual understanding.

[0778] Step 12:

[0779] The server refers to the user's database to identify items that are missing from their outfit. It determines specific missing items, such as "business shoes are missing."

[0780] Step 13:

[0781] The server uses an API from an external online shop to search for missing items. The search results include numerous candidate products and their detailed information.

[0782] Step 14:

[0783] The server retrieves product information and sends it to the terminal. This information includes product details, price, and a purchase link.

[0784] Step 15:

[0785] The device notifies the user of purchase offers it has received. The notification will display something like, "We have business shoes available. Purchase them here: [link]".

[0786] Step 16:

[0787] The user reviews the suggested outfit and purchases the suggested items if necessary. For example, they might click a purchase link to access an online shop and buy the items.

[0788] The above outlines the specific processing steps.

[0789] (Example 1)

[0790] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0791] Traditional outfit suggestion systems have been cumbersome to manage due to the complexities of the user's clothing database, and have struggled to efficiently suggest the most suitable outfit based on schedules and weather. Furthermore, they lacked sufficient methods for appropriately suggesting items missing from the suggested outfits, indicating room for improvement in user convenience.

[0792] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0793] In this invention, the server includes means for managing a database of clothing owned by the user; means for acquiring the user's location information and schedule information; means for acquiring weather information based on the acquired location information and schedule information; means for suggesting the optimal outfit combination based on the owned clothing, location information, schedule information, and weather information; means for searching for and suggesting items missing from the suggested outfit from an external source; means for tagging image data in the user's clothing database and classifying clothing based on the tag information; and means for generating prompt sentences to suggest outfits using a generative AI model. As a result, the user can efficiently manage their clothing, quickly receive suggestions for the optimal outfit based on their schedule and the weather, and easily purchase any missing items.

[0794] "User" refers to an individual who uses this system.

[0795] A "clothing database" refers to a database used to manage information about the clothes a user owns (such as photos, tags, and classification information).

[0796] "Location information" refers to data about the physical location where the user is currently located.

[0797] "Schedule information" refers to data about the day's plans and activities that the user has entered in advance.

[0798] "Weather information" refers to meteorological data obtained based on the user's current location.

[0799] "Tags" are keywords that users attach to photos of clothing they have taken, and they refer to information that makes it easier to classify and search for clothing.

[0800] "Optimal outfit combination" refers to the selection of clothing most suitable for the day, suggested based on the user's schedule, location, and weather information.

[0801] "External search" refers to the operation where a server uses the internet or other external databases to find items that the user does not possess.

[0802] A "generative AI model" refers to a model that uses generative artificial intelligence, and is used in this invention to suggest clothing.

[0803] A "prompt statement" is an input statement given to a generative AI model, referring to instructions or questions that guide the model to generate appropriate output.

[0804] A specific embodiment for implementing the system of the present invention will now be described. This system primarily involves data exchange between the user, terminal, and server to suggest the most suitable clothing for the user.

[0805] When users manage their clothing database, they first use a device such as a smartphone or tablet. Users take pictures of their clothes using the device's camera function. At that time, users add tags to each photo (for example, "white shirt," "blue pants," etc.). This tagging makes subsequent processing easier.

[0806] The device sends the captured, tagged photo data to the server. The server stores the received data in a database and classifies the clothing by type based on the photos and tag information. MySQL is one possible database management system to use.

[0807] Next, when the user enters their schedule for the day, the device saves this information and also retrieves the user's current location. Furthermore, the device uses an API to obtain weather information, such as the OpenWeatherMap API. This information is then sent from the device to the server.

[0808] The server analyzes the received schedule information, location information, and weather information. Based on the analysis results, it selects appropriate clothing from the user's clothing database and generates an optimal outfit. The specific analysis method includes analysis using a generative AI model. An example of a prompt statement to instruct this generative AI model is as follows.

[0809] "Please suggest appropriate attire for a business meeting from the user's clothing database. The weather is sunny and the temperature is 25°C."

[0810] The generated outfit is sent from the server to the terminal and notified to the user. At this time, any missing items in the suggested outfit are identified. The server refers to the user's database to determine the missing items and searches for appropriate products using an external database (e.g., Amazon API). The found products are sent to the terminal along with detailed information and a purchase link.

[0811] Finally, the device notifies the user of the received outfit information and purchase suggestions for any missing items. The user can then review the details of the suggested outfit and easily purchase any missing items via the purchase link.

[0812] This reduces the hassle for users in choosing the best outfit based on their schedule and the weather for the day, and also makes purchasing necessary items smoother.

[0813] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0814] Step 1:

[0815] The user takes a picture of their clothing. The input is the user's photo of the clothing and its tag information. The output is the tagged photo data saved on the device. The user uses their smartphone camera to take a picture of, for example, a "white shirt" and tags it "white shirt".

[0816] Step 2:

[0817] The device sends tagged photo data to the server. The input is the tagged photo data generated in step 1. The output is the data sent to the server. The device sends the photo data and tag information together to the cloud server.

[0818] Step 3:

[0819] The server receives data, stores it in a database, and classifies the clothing based on the tag information. The input is tagged photo data sent to the server. The output is the classified clothing data stored in the database. The server stores the photo data and tag information in a database such as MySQL, and classifies the clothing by category based on the tag information.

[0820] Step 4:

[0821] The user enters their schedule for the day into the device. The input is the user's schedule information. The output is the schedule information saved on the device. The user enters, for example, "Business meeting at 2 PM" into their smartphone or tablet.

[0822] Step 5:

[0823] The device obtains the user's current location information. The input is the current location information obtained from the device's location services. The output is the location information stored on the device. The device uses its GPS function to obtain the user's current location.

[0824] Step 6:

[0825] The device accesses a weather information API based on its location to obtain current weather information. The input is the user's location information. The output is the obtained weather information. The device uses the OpenWeatherMap API to obtain weather information such as "Tokyo, sunny, 25°C".

[0826] Step 7:

[0827] The terminal sends the acquired schedule information and weather information to the server. The input is the schedule information and weather information acquired in steps 4 and 6. The output is the information sent to the server. The terminal sends the schedule information and weather information to the server together.

[0828] Step 8:

[0829] The server analyzes the schedule information, location information, and weather information it receives. The input is the schedule information, location information, and weather information sent to the server. The output is the analysis result. Based on the schedule information and weather information, the server might determine, for example, "You need to choose appropriate attire for a business meeting."

[0830] Step 9:

[0831] The server selects appropriate clothing from the user's clothing database based on the analysis results and generates an optimal outfit. The input is schedule information, location information, and weather information analyzed by the server. The output is an optimal outfit suggestion. Using a generative AI model, the server suggests an outfit such as "black jacket, white shirt, blue pants."

[0832] Step 10:

[0833] The generated outfit information is sent to the terminal and notified to the user. The input is the outfit suggestion generated on the server. The output is the outfit information sent to the terminal. On the terminal, the user will see a notification such as, "Today's outfit is a black jacket, white shirt, and blue trousers."

[0834] Step 11:

[0835] The server identifies items needed for an outfit that the user does not possess and searches for those items by referencing an external database. The input is the user's clothing database stored on the server. The output is detailed information about the missing items and a purchase link. For example, if the server determines that "business leather shoes are missing," it will use an external API to find a purchase link for the appropriate product.

[0836] Step 12:

[0837] The system sends purchase information for the suggested missing items to the terminal and notifies the user. The input is the missing item information retrieved by the server from an external database. The output is the purchase information sent to the terminal. The terminal displays "You can purchase business leather shoes. Click this link: [link]".

[0838] This allows users to easily choose the perfect outfit for the day and conveniently purchase any missing items.

[0839] (Application Example 1)

[0840] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0841] Traditional systems offer limited suggestions based on user schedules and weather information, which is insufficient to meet the diverse needs of users. In particular, there is a need for more comprehensive suggestions tailored to users' lifestyles, encompassing not only clothing but also information on food and other items. Furthermore, the search and suggestion functions for items not owned by the user are lacking and need to be streamlined and improved.

[0842] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0843] In this invention, the server includes means for managing a database of items owned by the user, means for acquiring the user's location information and schedule information, means for acquiring external information based on the acquired location information and schedule information, means for making optimal suggestions based on the owned items, location information, schedule information and external information, means for searching for and suggesting items missing from the suggested content from external sources, and means for tagging image data taken by the user and classifying it based on the tag information. As a result, the user can comprehensively manage their belongings and external information and receive suggestions optimized for their lifestyle.

[0844] A "user" refers to an individual who uses the system to manage their belongings and information and receive optimal suggestions.

[0845] "Items" refers to a broad range of items owned by the user (such as clothing or food).

[0846] A "database" refers to an information collection used to store and manage information and image data of items owned by a user.

[0847] "Location information" refers to digital information that indicates the user's current location.

[0848] "Schedule information" refers to information about the user's future plans and schedules.

[0849] "External information" refers to information obtained from external data sources based on the user's location and schedule information, such as weather information and food-related information.

[0850] "Suggestion" refers to the act of recommending the most suitable actions or items based on the user's possessions and acquired external information.

[0851] "Searching" refers to the act of finding missing items that a user does not possess by searching an external database.

[0852] "Image data" refers to photographic data that users have taken and uploaded to the system.

[0853] A "tag" refers to a keyword that indicates attributes or classification information attached to image data.

[0854] "Classification" refers to the act of organizing image data or items into categories based on tag information.

[0855] Specific embodiments of this invention will be described in detail below.

[0856] System Overview

[0857] This invention is a system that manages items owned by a user (e.g., clothing, food, etc.) and makes optimal suggestions based on the user's location and schedule information. Furthermore, it is a system that searches for items that are missing from the suggested items and notifies the user.

[0858] Hardware and software to be used

[0859] Server: Provides the core computing and database management functions of this system. The server runs on Node.js and Express.

[0860] Database: MongoDB is used to store user-owned items and image data.

[0861] Weather Information API: Use the OpenWeatherMap API to obtain current weather information.

[0862] Food Delivery API: Use the Uber Eats API and DoorDash API to retrieve information about food delivery.

[0863] Image analysis module: Using TensorFlow.js, tag information is generated from images taken by the user.

[0864] Client device: Users use devices such as smartphones to input location and schedule information and receive suggestions.

[0865] Data flow and processing

[0866] 1. Management of the item database:

[0867] Users take photos of their belongings (e.g., clothing and food items) and tag them. The tagged photo data is sent to a server and stored in a database. This allows users to easily manage their possessions.

[0868] 2. Acquisition of location and schedule information:

[0869] Users input their daily schedule and current location via a device such as a smartphone. This information is sent to a server and stored for subsequent processing.

[0870] 3. Acquisition of external information:

[0871] The server accesses a weather information API based on location data to obtain current weather information. It also obtains food-related information from delivery service APIs.

[0872] 4. Generating the optimal proposal:

[0873] The server analyzes database information, location information, schedule information, and acquired external information to generate the most suitable suggestions for the user. For example, if the user has plans to go out on a nice day, it will suggest nearby food delivery services.

[0874] 5. Search for and suggest missing items:

[0875] If the proposed items are missing, the server accesses an external database to search for them. It then notifies the user of the detailed information and a purchase link.

[0876] Specific example

[0877] For example, if a user wants to eat pizza, they take a picture of it and tag it. The system analyzes the picture and adds it to the user's database. Then, on the day the user plans to go out, the system retrieves the current weather information and suggests the best food delivery service. At this time, the user will be notified with a suggestion such as, "It's sunny, so we recommend delivery from a nearby pizza restaurant. Check Uber Eats or DoorDash." Furthermore, if the system determines that there is a shortage of cheese, a notification will appear saying, "You can purchase cheese. Please purchase it from this link."

[0878] Example of a prompt

[0879] Based on the following information, please generate a program that suggests the most suitable food delivery options to the user.

[0880] Photos and tag information of dishes the user wants to eat.

[0881] Current location and weather information

[0882] Plans to go out

[0883] You also need to provide links to purchase any missing ingredients.

[0884] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0885] Step 1:

[0886] The system takes images of items owned by the user and adds tags to them. The user uses their smartphone to take a picture of an item they own (for example, a dish they want to eat). Then, they add tags such as "pizza" or "hamburger" to that picture. The input is image data and tag information, and the output is tagged image data. The device sends this data to the server.

[0887] Step 2:

[0888] The server receives tagged image data and stores it in a database. The server receives tagged image data sent from the terminal and stores it in a MongoDB database. The input is tagged image data, and the output is the data stored in the database. The image data is analyzed using TensorFlow.js to generate appropriate tags.

[0889] Step 3:

[0890] The user enters their daily schedule and current location into the device. The user uses a smartphone to enter their outing plans and current location. This information is sent to the server. The input is the schedule and location information, and the output is the schedule and location information sent to the server.

[0891] Step 4:

[0892] The server retrieves weather information based on location data. The server uses the location data to access the OpenWeatherMap API and retrieve current weather information. The input is location data, and the output is current weather information. The server saves the weather information to a database.

[0893] Step 5:

[0894] The server analyzes the user's database and acquired external information to generate optimal suggestions. The server analyzes item information, location information, schedule information, and weather information stored in the user's database to generate the most suitable suggestions for the user. For example, on a sunny day, a suggestion such as "You should order from a nearby pizza restaurant" might be generated to match the user's planned outing. The input consists of database information and external information, and the output is the generated suggestion.

[0895] Step 6:

[0896] The server searches for any missing items in the proposed content and generates detailed information and purchase links. The server accesses an external database to search for items missing from the proposed content. For example, if it determines that "cheese is missing," it will generate a purchase link. The input is the proposed content and information from the external database, and the output is detailed information and purchase links.

[0897] Step 7:

[0898] The server notifies the terminal of the proposed content and details of the missing items. The server sends the generated proposed content and details of the missing items to the terminal and notifies the user. The user receives the notification via their smartphone and can take appropriate action. The input is the proposed content and details of the missing items, and the output is the notification to the user.

[0899] Through these steps, users can efficiently manage their possessions and receive optimal suggestions based on external information. This process consistently involves data processing and calculations, providing information optimized for the user's lifestyle.

[0900] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0901] Regarding embodiments for carrying out the present invention, the details of a clothing selection support system that combines an emotion engine will be described.

[0902] User clothing database management

[0903] The process begins with users taking photos of their clothes and tagging each photo (e.g., "white T-shirt," "blue jeans," etc.). The tagged photo data is sent from the device to a server, which stores the received data in a database. This database is used to systematically manage each user's clothing.

[0904] Obtaining schedule and weather information

[0905] When a user enters their daily schedule into their device, the device stores the schedule information linked to the user's current location. Next, the device accesses a weather information API based on the location information to obtain current weather information (e.g., temperature, weather). This information is sent from the device to the server, which analyzes the received information and prepares to generate the optimal outfit.

[0906] Emotion recognition by an emotion engine

[0907] When a user uses the device, the built-in emotion engine recognizes emotions from the user's facial expressions and voice. This emotion data reflects the user's current mood and psychological state. The recognized emotion information is sent from the device to the server.

[0908] Coordination suggestions

[0909] The server analyzes the received schedule information, location information, weather information, and sentiment information. Based on the analysis results, it selects appropriate clothing from the user's database. For example, given conditions such as "business meeting" + "sunny, 25°C" + "user is feeling well," the server determines that "formal business attire is appropriate, and bright-colored items should be chosen," and suggests a combination such as a black jacket, white T-shirt, and blue jeans. The suggested outfit is sent to the terminal and notified to the user.

[0910] Suggestions for missing items

[0911] The server consults the user's database to identify items needed for an outfit that the user does not possess. For example, if it determines that the user is missing business shoes, the server accesses an external database to search for them. If a suitable product is found, it sends detailed information and a purchase link to the user's device and notifies them. The notification content is customized based on the emotions recognized by the emotion engine. For example, if the user is relaxed, a friendly message such as "We have some great business shoes. Purchase them here: link" will be displayed.

[0912] Coordination and purchase suggestion notification

[0913] The system visually and clearly notifies the user of outfit information received by the device and purchase suggestions for missing items. The notified information includes images and text, allowing the user to view the details. Furthermore, the message content is customized based on emotional information generated by an emotion engine, taking care to ensure that the user receives it in a more positive way.

[0914] Specific example

[0915] Suppose a user receives the following outfit and purchase suggestions under the conditions of "business meeting" + "sunny, 25℃" + "user is feeling well".

[0916] The device suggested an outfit consisting of a "black jacket," a "white T-shirt," and "blue jeans."

[0917] The device simultaneously displayed a notification stating, "We are short on business shoes. Please purchase them from the link below."

[0918] In this way, users can easily find the perfect outfit for the day and conveniently purchase any missing items as needed. Furthermore, because the emotion engine adapts notifications to the user's emotions, it is possible to use the system more comfortably.

[0919] The above describes the embodiments for carrying out the present invention. By combining the emotion engine, personalized suggestions tailored to the user's psychological state are realized, significantly improving the user experience.

[0920] The following describes the processing flow.

[0921] Step 1:

[0922] Users take photos of the clothes they own and tag each photo (for example, "white T-shirt," "blue jeans," etc.).

[0923] Step 2:

[0924] The device sends tagged photo data to the server. The transmitted data is in JSON format and includes the photo and tags.

[0925] Step 3:

[0926] The server saves the received data to the database. This allows for systematic management of each user's belongings.

[0927] Step 4:

[0928] The user enters their schedule for the day into the device. Schedule information is obtained through the calendar function or direct input.

[0929] Step 5:

[0930] The device obtains the user's current location information using GPS functionality. The location information is expressed in the form of latitude and longitude.

[0931] Step 6:

[0932] The device accesses a weather information API based on its location to retrieve current weather information (e.g., temperature, weather conditions).

[0933] Step 7:

[0934] The device sends the acquired schedule and weather information to the server. The transmitted data is in JSON format and includes location information, schedule, and weather information.

[0935] Step 8:

[0936] When a user uses the device, the built-in emotion engine recognizes emotions from the user's facial expressions and voice. This emotion data reflects the user's current mood and psychological state. Emotional information is transmitted from the device to the server.

[0937] Step 9:

[0938] The server analyzes the received schedule, location, weather, and sentiment information. Based on the analysis, it selects appropriate clothing from the user's database. For example, given the conditions "business meeting" + "sunny, 25°C" + "user is feeling well," the server would determine that "formal business attire is appropriate, and bright-colored items should be chosen."

[0939] Step 10:

[0940] The server sends generated outfit information (e.g., black jacket, white T-shirt, blue jeans) to the terminal. The information sent includes the selected clothing items and their details.

[0941] Step 11:

[0942] The device notifies the user of the outfit information it has received. The notification is displayed with images and text for easy visual understanding. In addition, the message is customized based on the results of the emotion engine's recognition. If the user is feeling energetic, a positive message such as "This outfit will help you have an energetic day!" will be displayed.

[0943] Step 12:

[0944] The server consults the user's database to identify items needed for an outfit that the user does not possess. If it determines that "business shoes are missing," the server accesses an external database to search for those items.

[0945] Step 13:

[0946] The server retrieves product information from an external database and sends it to the terminal. The transmitted information includes product details, price, and a purchase link.

[0947] Step 14:

[0948] The device notifies the user of purchase suggestions it has received. The notification content is customized based on the user's emotional state. For example, if the user is relaxed, a friendly message such as "We have some great business shoes. Purchase them here: link" will be displayed.

[0949] The above outlines the specific processing steps of the system that combines the emotion engine.

[0950] (Example 2)

[0951] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0952] In modern society, choosing appropriate attire amidst a busy daily life is difficult for many. Efficiently deciding on an outfit that suits the weather and schedule is particularly challenging. Furthermore, clothing choices are often influenced by emotions and psychological states, and these factors cannot be ignored. Conventional systems struggle to integrate these diverse factors to provide optimal suggestions, resulting in a poor user experience.

[0953] The identification processing performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for managing a database of clothing owned by the user, means for acquiring the user's location information and schedule information, means for acquiring weather information based on the acquired location information and schedule information, means for recognizing emotions from the user's facial expressions and voice, means for suggesting the optimal outfit combination based on the owned clothing, location information, schedule information, weather information and emotion information, and means for searching for and suggesting items missing from the suggested outfit from an external source. This enables the integration of diverse information to provide the user with personalized and optimal outfit suggestions.

[0954] "Means for managing a database of clothing owned by a user" refers to systems or software that collect, store, and systematically organize image data and tag information of clothing owned by a user.

[0955] "Means for acquiring user location and schedule information" refers to systems or software used to collect and store a user's current location and schedule for the day.

[0956] "Means for obtaining weather information based on acquired location and schedule information" refers to APIs or systems that obtain weather forecast information based on the user's location and schedule information.

[0957] "Means of recognizing emotions from a user's facial expressions and voice" refers to systems and software that analyze a user's facial expressions and voice to recognize the user's emotional state.

[0958] "Means of suggesting the optimal clothing combination based on the user's clothing, location information, schedule information, weather information, and emotional information" refers to algorithms and systems that integrate and analyze various user information to suggest the optimal clothing combination.

[0959] "Methods for searching for and suggesting items missing from a proposed outfit" refers to systems or algorithms that search an external database for items necessary for a proposed outfit that the user does not own, and then present the results to the user.

[0960] "Means of notifying users of suggested outfits and information on missing items" refers to systems or software that notify users of generated outfit suggestions and information on purchasing missing items.

[0961] "A means of attaching tags to image data of clothing owned by a user and classifying them based on that tag information" refers to a system or software that attaches specific tags to images of clothing taken by a user and categorizes them based on that tag information.

[0962] This invention details a clothing selection support system that incorporates an emotion engine. This system manages the user's clothing data and suggests optimal outfits based on external data such as schedule information and weather information. Furthermore, by recognizing the user's emotions, it provides individually optimized suggestions.

[0963] User clothing database management

[0964] The process begins with the user taking photos of their clothes and attaching tags. The tagged photos are sent from the device to a server, which then stores the received data in a database. This database is used to systematically manage each user's clothing. A smartphone or tablet is required to retrieve the data, and an internet connection is needed to transmit it.

[0965] Obtaining schedule and weather information

[0966] When a user enters their daily schedule into the device, the device stores the schedule information, linking it to the user's current location. This location information is obtained using GPS. Next, the device accesses a weather information API based on the location information to obtain current weather information (e.g., temperature, weather conditions). The obtained information is sent from the device to the server for preparation for analysis. Weather information APIs used include the OpenWeatherMap API and the Weather API.

[0967] Emotion recognition by an emotion engine

[0968] When a user uses a device, the built-in emotion engine (e.g., Microsoft Azure's Emotion API or Google Cloud's Speech-to-Text) is activated to recognize emotions from the user's facial expressions and voice. This recognized emotion data is then sent from the device to a server.

[0969] Coordination suggestions

[0970] The server performs analysis based on the schedule information, location information, weather information, and sentiment information it receives. Based on this analysis, it selects appropriate clothing from the user's database. For example, given conditions such as "business meeting" + "sunny, 25°C" + "user is feeling well," the server will determine that "formal business attire is appropriate, and bright-colored items should be chosen," and suggest a combination such as a black jacket, white T-shirt, and blue jeans. The suggested result is sent to the terminal and notified to the user.

[0971] Suggestions for missing items

[0972] The server accesses the user's database to identify items needed for an outfit that the user does not possess. For example, if it determines that the user is missing business shoes, it accesses external databases (e.g., Amazon API, Rakuten Market API) to search for the item. If a suitable product is found, it sends the detailed information and purchase link to the user's device and notifies them.

[0973] Coordination and purchase suggestion notification

[0974] The device visually and clearly notifies the user of outfit information received and purchase suggestions for missing items. These notifications include images and text and are customized based on emotional information from an emotion engine. The user can review the suggestions and purchase items as needed.

[0975] Specific example

[0976] For example, consider a scenario where a user receives the following outfit suggestion based on the conditions "business meeting" + "sunny, 25℃" + "user is feeling well".

[0977] The device suggests outfits such as a "black jacket," "white T-shirt," and "blue jeans."

[0978] The device displays a notification that says, "We are short on business shoes. Please purchase them from the link below."

[0979] In this way, users can easily find the perfect outfit for the day and conveniently purchase any missing items as needed.

[0980] Examples of prompts for generative AI models

[0981] The following are examples of prompts to input into the generated AI model.

[0982] Please write a natural language description of a program that provides coordination suggestions to a user under the conditions of "business meeting," "sunny, 25℃," and "user is feeling well."

[0983] Thus, by implementing the present invention, personalized suggestions tailored to the user's psychological state are realized, significantly improving the user experience.

[0984] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0985] Step 1: Register clothing data

[0986] The user takes photos of their own clothes and tags each photo (e.g., "white T-shirt," "blue jeans"). The device then processes this tagged photo data and sends it to the server. The server parses the received data and stores it in a database.

[0987] Input: Photos and tag information of clothing

[0988] Output: Clothing data stored in the database

[0989] Specific operation: The user takes a photo of clothing with their smartphone. The device displays a tag input screen, and the user enters the tags. The device sends the image and tags to the server. The server saves them to the database.

[0990] Step 2: Obtain schedule and location information

[0991] The user enters their schedule for the day into their device. The device obtains the user's current location information (GPS) and stores it in conjunction with the schedule information.

[0992] Input: User's schedule information and location information

[0993] Output: Saved schedule information and location information

[0994] Specific operation: The user enters an event into the device's calendar app. The device obtains location information from GPS. The device saves the event information and location information.

[0995] Step 3: Obtain weather information

[0996] The device accesses a weather information API (e.g., OpenWeatherMap API) based on its location to obtain current weather information (e.g., temperature, weather conditions).

[0997] Input: Location information

[0998] Output: Acquired weather information

[0999] Specific operation: The device accesses the weather information API and retrieves weather information. The retrieved weather information is sent to the server.

[1000] Step 4: Performing emotion recognition

[1001] While the user is using the device, the built-in emotion engine (e.g., Microsoft Azure Emotion API) analyzes facial expressions and voice to recognize emotions. The recognized emotion data is then sent from the device to the server.

[1002] Input: User's facial expressions and voice data

[1003] Output: Recognized emotion data

[1004] Specific operation: The user takes a selfie with the device's camera or speaks into the microphone. The device activates its emotion engine and analyzes facial expressions and voice. The emotion data is sent to the server.

[1005] Step 5: Generating Coordination Proposals

[1006] The server analyzes the received schedule information, location information, weather information, and sentiment information to generate the optimal outfit. It selects suitable clothing from the user's database. For example, given conditions such as "business meeting" + "sunny, 25℃" + "user is feeling good," it will suggest the optimal combination (e.g., black jacket, white T-shirt, blue jeans).

[1007] Input: Schedule information, location information, weather information, sentiment information

[1008] Output: Proposed coordination

[1009] Specific operation: The server analyzes the received information. It selects clothing from the user database. It generates the suggested clothing. It sends the generated result to the terminal.

[1010] Step 6: Detecting missing items

[1011] The server uses the proposed outfit to determine if any items are missing by referencing the user's database. If missing items are found, it accesses an external database (e.g., Amazon API) to search for them and generates the results.

[1012] Input: Proposed outfits, user database

[1013] Output: Search results for missing items

[1014] Specific operation: The server checks the user's database. It identifies missing items. It accesses an external database to search for the items. It sends the search results to the terminal.

[1015] Step 7: Proposal and Purchase Notification

[1016] The device notifies the user of outfit information it has received and purchase suggestions for missing items. The notification includes images and text and is customized based on emotional information from the emotion engine.

[1017] Input: Suggested outfit, purchase information for missing items

[1018] Output: Visually customized notifications

[1019] Specific actions: The device receives suggestion information from the server. The suggested outfit is displayed as an image and text. Purchase links for missing items are displayed. The message is customized based on information from the emotion engine.

[1020] (Application Example 2)

[1021] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1022] In modern society, users often find it difficult to choose clothing that suits their diverse schedules and lifestyles. Furthermore, some users desire content recommendations based on their current mood and environment. However, no system exists to address these individual needs; there is a need for a system that reduces user effort and provides appropriate suggestions in a streamlined manner.

[1023] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[1024] In this invention, the server includes means for managing a database of clothing owned by the user; means for acquiring the user's location information and schedule information; means for acquiring weather information based on the acquired location information and schedule information; means for suggesting the optimal outfit combination based on the owned clothing, location information, schedule information, and weather information; means for searching for and suggesting items missing from the suggested outfit from an external source; and means for recognizing the user's emotional state and recommending content based on that emotion. As a result, the user can not only easily choose the outfit best suited to their schedule and environment for the day, but also receive content recommendations that match their emotions.

[1025] "User" refers to a person or individual who uses this system.

[1026] A "clothing database" is a collection of data used to systematically store and manage information about various types of clothing owned by a user.

[1027] "Location information" refers to information that indicates the user's current location or geographical position.

[1028] "Schedule information" refers to information about plans and schedules that a user will have in the future.

[1029] "Weather information" refers to information about current or future weather and meteorological conditions.

[1030] "Optimal clothing combination" refers to the set of clothes that is most suitable for the user, based on conditions such as weather and schedule.

[1031] "Missing items" refer to clothing or accessories that the user does not currently own but are required for the suggested outfit combination.

[1032] "Emotional state" refers to the user's current emotional and psychological state.

[1033] "Content" is a general term for digital information and entertainment provided to users, such as videos, music, and text.

[1034] The system for implementing this invention manages a database of the user's clothing and suggests the optimal clothing combination and any missing items based on the user's location, schedule, weather, and emotional state. It also recommends digital content according to the user's emotional state.

[1035] User clothing database management

[1036] Users take photos of their clothes and tag each photo (e.g., "white T-shirt" or "blue jeans"). The tagged photo data is sent from the device to the server, which stores the received data in a database. This database systematically manages each user's clothing.

[1037] Acquisition of location and weather information

[1038] When a user enters their daily schedule into the device, the device stores the schedule information linked to the user's current location. Next, the device accesses a weather information API based on the location information to obtain current weather information (temperature and weather conditions). This information is sent from the device to the server, which analyzes the received information and prepares to generate the optimal outfit.

[1039] Emotion recognition by an emotion engine

[1040] When a user uses the device, the built-in emotion engine recognizes emotions from the user's facial expressions and voice. This emotion data reflects the user's current mood and psychological state. The recognized emotion information is sent from the device to the server.

[1041] Coordination suggestions and suggestions for missing items

[1042] The server analyzes the received schedule information, location information, weather information, and sentiment information. Based on the analysis results, it selects appropriate clothing from the user's database. For example, if given conditions such as "business meeting" + "sunny, 25°C" + "user is feeling well," the server will determine that "formal business attire is appropriate, and choose bright-colored items," and the suggested clothing will be sent to the terminal and notified to the user.

[1043] Furthermore, the server consults the user's database to identify items needed for an outfit that the user does not possess. For example, if it determines that the user is missing business shoes, the server accesses an external database to search for the item. If a suitable product is found, it sends detailed information and a purchase link to the user's device and notifies them. The notification content is customized based on the emotions recognized by the emotion engine.

[1044] Digital content recommendations

[1045] Furthermore, the server recommends content appropriate to the user based on their emotional state. For example, if the user is "happy," the server will recommend "comedy movies" or "funny videos." This recommendation is based on a combination of emotional data and weather information.

[1046] Hardware and software to be used

[1047] Hardware: Smartphones (such as Apple iPhone and Samsung Galaxy)

[1048] Software: Python programs, weather information APIs (such as OpenWeatherMap), emotion recognition APIs (such as Microsoft Azure Emotion API)

[1049] Specific example

[1050] For example, under the conditions of "business meeting" + "sunny, 25℃" + "user is feeling well," a user might receive the following outfit and purchase suggestions.

[1051] The suggested outfit was a "black jacket," a "white T-shirt," and "blue jeans."

[1052] At the same time, a notification appears stating, "We are short on business shoes. Please purchase them from the link below."

[1053] As described above, users can easily find the perfect outfit for their day's schedule and easily obtain the necessary items. Furthermore, because the emotion engine adapts notifications to the user's emotions, they can use the system more comfortably.

[1054] Example of a prompt:

[1055] If the user's emotional state is "happy," the weather is "sunny," and the temperature is "25°C," please generate a list of recommended video content.

[1056] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[1057] Step 1:

[1058] The user creates their own clothing database. They take photos of their clothes with their device and tag each photo. The tagged photo data is sent from the device to the server, where it is stored in the database. The input is the user's clothing photos and tags, and the output is the clothing information stored in the server's database.

[1059] Step 2:

[1060] The user enters their schedule for the day into the terminal. The terminal retrieves the entered schedule information and the user's current location. This information is then sent from the terminal to the server. The input is the user's schedule information and location, and the output is the schedule information and location sent to the server.

[1061] Step 3:

[1062] The server accesses a weather information API based on the received location information to obtain current weather information (temperature and weather conditions). The input is location information, and the output is weather information.

[1063] Step 4:

[1064] When a user uses the device, the device's built-in emotion engine analyzes the user's facial image and voice to perform emotion recognition. The recognized emotion information is sent from the device to the server. The input is the user's facial image or voice, and the output is the recognized emotion information.

[1065] Step 5:

[1066] The server analyzes the received schedule information, location information, weather information, and sentiment information. Based on this, it selects the most suitable clothing combination from the user's database. For example, if the user has a "business meeting," the weather is sunny at 25°C, and the user is feeling well, the server will determine that formal attire is appropriate. The input is schedule information, location information, weather information, and sentiment information, and the output is a suggestion for the most suitable clothing.

[1067] Step 6:

[1068] The server consults the user's clothing database to identify any missing items in the suggested outfit. For example, if "business shoes are missing," the server accesses an external database to search for those items. If a suitable product is found, it sends the details and a purchase link to the terminal and notifies the user. The input is the user's clothing database and the suggested outfit, and the output is the search results for the missing items and the notification.

[1069] Step 7:

[1070] The server recommends appropriate digital content based on the user's emotional state. For example, if the user is "happy," the server recommends "comedy movies" or "funny videos." The recommendation list is generated based on a combination of emotional data and weather information. The input is the user's emotional information and weather information, and the output is the recommended content list.

[1071] Step 8:

[1072] The device notifies the user of optimal clothing suggestions received from the server, purchase links for missing items, and recommended digital content. The notification content is displayed in a visually easy-to-understand format, allowing the user to review it. Input is suggestion data and recommended content from the server, and output is the notified information.

[1073] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[1074] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1075] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[1076] [Fourth Embodiment]

[1077] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[1078] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1079] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1080] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[1081] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[1082] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[1083] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[1084] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[1085] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[1086] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1087] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1088] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[1089] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1090] The embodiments for carrying out the present invention will be described in detail below.

[1091] User clothing database management

[1092] The process begins with the user taking photos of their clothes and tagging each photo (e.g., "white T-shirt," "blue jeans," etc.). The device then sends the tagged photo data to a server, which stores the received data in a database. Using the tag information, the clothes are categorized by type, making subsequent processing easier.

[1093] Obtaining schedule and weather information

[1094] When a user enters their daily schedule into the device, the device stores the schedule information linked to the user's current location. Next, the device accesses a weather information API based on the location information to obtain the current weather information. This information is sent from the device to the server, which then prepares to generate the optimal outfit based on this information.

[1095] Coordination suggestions

[1096] The server analyzes the user's schedule, location, and weather information. Based on the analysis, it selects appropriate clothing from the user's database. For example, if the conditions are "business meeting" + "sunny, 25°C," the server will determine that "formal business attire is appropriate" and suggest a combination such as a black jacket, white T-shirt, and blue jeans. This suggestion is sent to the terminal and notified to the user.

[1097] Suggestions for missing items

[1098] Furthermore, the server consults the user's database to identify items needed for an outfit that the user does not possess. For example, if it determines that the user is missing business shoes, the server accesses an external database to search for the item. If a suitable product is found, it sends the details and a purchase link to the user's device, notifying them of the item. For example, the notification might say, "We have business leather shoes available. Purchase them here: [link]".

[1099] Coordination and purchase suggestion notification

[1100] The device clearly notifies the user of the outfit information it has received and the purchase suggestions for any missing items. The notified user can then check the details of the suggested outfit and, if necessary, purchase any missing items via the provided purchase link.

[1101] Specific example

[1102] For example, suppose a user receives the following outfit and purchase suggestions under the conditions of "business meeting" and "sunny, 25℃".

[1103] The device suggested an outfit consisting of a "black jacket," a "white T-shirt," and "blue jeans."

[1104] The device simultaneously displayed a notification stating, "We are short on business shoes. Please purchase them from the link below."

[1105] In this way, users can easily find the perfect outfit for the day and conveniently purchase any missing items as needed.

[1106] The above describes the embodiments for carrying out the present invention. This system not only frees users from the hassle of choosing the most suitable clothing based on their schedule and the weather for the day, but also allows them to easily purchase any missing items.

[1107] The following describes the processing flow.

[1108] Step 1:

[1109] Users take photos of the clothes they own and tag each photo (for example, "white T-shirt," "blue jeans," etc.).

[1110] Step 2:

[1111] The device sends tagged photo data to the server. The transmitted data is in JSON format and includes the photo and tags.

[1112] Step 3:

[1113] The server saves the received data to the database. This allows for systematic management of each user's belongings.

[1114] Step 4:

[1115] The user enters their schedule for the day into the device. Schedule information is obtained through the calendar function or direct input.

[1116] Step 5:

[1117] The device obtains the user's current location information using GPS functionality. The location information is expressed in the form of latitude and longitude.

[1118] Step 6:

[1119] The device accesses a weather information API based on its location to retrieve current weather information (e.g., temperature, weather conditions).

[1120] Step 7:

[1121] The device sends the acquired schedule and weather information to the server. The transmitted data is in JSON format and includes location information, schedule, and weather information.

[1122] Step 8:

[1123] The server analyzes the submitted schedule, location, and weather information to generate the most suitable outfit for the day. Based on the analysis results, the formality level of the clothing (business, casual, etc.) and seasonality are taken into consideration.

[1124] Step 9:

[1125] The server selects appropriate clothing from the user's database. For example, given the conditions "business meeting" + "sunny, 25°C", it would select the combination of "black jacket", "white T-shirt", and "blue jeans".

[1126] Step 10:

[1127] The server sends the generated outfit information to the terminal. The information sent includes the selected clothing items and their details.

[1128] Step 11:

[1129] The device notifies the user of the coordinated outfit information it has received. The notification is displayed using images and text for easy visual understanding.

[1130] Step 12:

[1131] The server refers to the user's database to identify items that are missing from their outfit. It determines specific missing items, such as "business shoes are missing."

[1132] Step 13:

[1133] The server uses an API from an external online shop to search for missing items. The search results include numerous candidate products and their detailed information.

[1134] Step 14:

[1135] The server retrieves product information and sends it to the terminal. This information includes product details, price, and a purchase link.

[1136] Step 15:

[1137] The device notifies the user of purchase offers it has received. The notification will display something like, "We have business shoes available. Purchase them here: [link]".

[1138] Step 16:

[1139] The user reviews the suggested outfit and purchases the suggested items if necessary. For example, they might click a purchase link to access an online shop and buy the items.

[1140] The above outlines the specific processing steps.

[1141] (Example 1)

[1142] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1143] Traditional outfit suggestion systems have been cumbersome to manage due to the complexities of the user's clothing database, and have struggled to efficiently suggest the most suitable outfit based on schedules and weather. Furthermore, they lacked sufficient methods for appropriately suggesting items missing from the suggested outfits, indicating room for improvement in user convenience.

[1144] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[1145] In this invention, the server includes means for managing a database of clothing owned by the user; means for acquiring the user's location information and schedule information; means for acquiring weather information based on the acquired location information and schedule information; means for suggesting the optimal outfit combination based on the owned clothing, location information, schedule information, and weather information; means for searching for and suggesting items missing from the suggested outfit from an external source; means for tagging image data in the user's clothing database and classifying clothing based on the tag information; and means for generating prompt sentences to suggest outfits using a generative AI model. As a result, the user can efficiently manage their clothing, quickly receive suggestions for the optimal outfit based on their schedule and the weather, and easily purchase any missing items.

[1146] "User" refers to an individual who uses this system.

[1147] A "clothing database" refers to a database used to manage information about the clothes a user owns (such as photos, tags, and classification information).

[1148] "Location information" refers to data about the physical location where the user is currently located.

[1149] "Schedule information" refers to data about the day's plans and activities that the user has entered in advance.

[1150] "Weather information" refers to meteorological data obtained based on the user's current location.

[1151] "Tags" are keywords that users attach to photos of clothing they have taken, and they refer to information that makes it easier to classify and search for clothing.

[1152] "Optimal outfit combination" refers to the selection of clothing most suitable for the day, suggested based on the user's schedule, location, and weather information.

[1153] "External search" refers to the operation where a server uses the internet or other external databases to find items that the user does not possess.

[1154] A "generative AI model" refers to a model that uses generative artificial intelligence, and is used in this invention to suggest clothing.

[1155] A "prompt statement" is an input statement given to a generative AI model, referring to instructions or questions that guide the model to generate appropriate output.

[1156] A specific embodiment for implementing the system of the present invention will now be described. This system primarily involves data exchange between the user, terminal, and server to suggest the most suitable clothing for the user.

[1157] When users manage their clothing database, they first use a device such as a smartphone or tablet. Users take pictures of their clothes using the device's camera function. At that time, users add tags to each photo (for example, "white shirt," "blue pants," etc.). This tagging makes subsequent processing easier.

[1158] The device sends the captured, tagged photo data to the server. The server stores the received data in a database and classifies the clothing by type based on the photos and tag information. MySQL is one possible database management system to use.

[1159] Next, when the user enters their schedule for the day, the device saves this information and also retrieves the user's current location. Furthermore, the device uses an API to obtain weather information, such as the OpenWeatherMap API. This information is then sent from the device to the server.

[1160] The server analyzes the received schedule information, location information, and weather information. Based on the analysis results, it selects appropriate clothing from the user's clothing database and generates an optimal outfit. The specific analysis method includes analysis using a generative AI model. An example of a prompt statement to instruct this generative AI model is as follows.

[1161] "Please suggest appropriate attire for a business meeting from the user's clothing database. The weather is sunny and the temperature is 25°C."

[1162] The generated outfit is sent from the server to the terminal and notified to the user. At this time, any missing items in the suggested outfit are identified. The server refers to the user's database to determine the missing items and searches for appropriate products using an external database (e.g., Amazon API). The found products are sent to the terminal along with detailed information and a purchase link.

[1163] Finally, the device notifies the user of the received outfit information and purchase suggestions for any missing items. The user can then review the details of the suggested outfit and easily purchase any missing items via the purchase link.

[1164] This reduces the hassle for users in choosing the best outfit based on their schedule and the weather for the day, and also makes purchasing necessary items smoother.

[1165] The flow of the specific processing in Example 1 will be explained using Figure 11.

[1166] Step 1:

[1167] The user takes a picture of their clothing. The input is the user's photo of the clothing and its tag information. The output is the tagged photo data saved on the device. The user uses their smartphone camera to take a picture of, for example, a "white shirt" and tags it "white shirt".

[1168] Step 2:

[1169] The device sends tagged photo data to the server. The input is the tagged photo data generated in step 1. The output is the data sent to the server. The device sends the photo data and tag information together to the cloud server.

[1170] Step 3:

[1171] The server receives data, stores it in a database, and classifies the clothing based on the tag information. The input is tagged photo data sent to the server. The output is the classified clothing data stored in the database. The server stores the photo data and tag information in a database such as MySQL, and classifies the clothing by category based on the tag information.

[1172] Step 4:

[1173] The user enters their schedule for the day into the device. The input is the user's schedule information. The output is the schedule information saved on the device. The user enters, for example, "Business meeting at 2 PM" into their smartphone or tablet.

[1174] Step 5:

[1175] The device obtains the user's current location information. The input is the current location information obtained from the device's location services. The output is the location information stored on the device. The device uses its GPS function to obtain the user's current location.

[1176] Step 6:

[1177] The device accesses a weather information API based on its location to obtain current weather information. The input is the user's location information. The output is the obtained weather information. The device uses the OpenWeatherMap API to obtain weather information such as "Tokyo, sunny, 25°C".

[1178] Step 7:

[1179] The terminal sends the acquired schedule information and weather information to the server. The input is the schedule information and weather information acquired in steps 4 and 6. The output is the information sent to the server. The terminal sends the schedule information and weather information to the server together.

[1180] Step 8:

[1181] The server analyzes the schedule information, location information, and weather information it receives. The input is the schedule information, location information, and weather information sent to the server. The output is the analysis result. Based on the schedule information and weather information, the server might determine, for example, "You need to choose appropriate attire for a business meeting."

[1182] Step 9:

[1183] The server selects appropriate clothing from the user's clothing database based on the analysis results and generates an optimal outfit. The input is schedule information, location information, and weather information analyzed by the server. The output is an optimal outfit suggestion. Using a generative AI model, the server suggests an outfit such as "black jacket, white shirt, blue pants."

[1184] Step 10:

[1185] The generated outfit information is sent to the terminal and notified to the user. The input is the outfit suggestion generated on the server. The output is the outfit information sent to the terminal. On the terminal, the user will see a notification such as, "Today's outfit is a black jacket, white shirt, and blue trousers."

[1186] Step 11:

[1187] The server identifies items needed for an outfit that the user does not possess and searches for those items by referencing an external database. The input is the user's clothing database stored on the server. The output is detailed information about the missing items and a purchase link. For example, if the server determines that "business leather shoes are missing," it will use an external API to find a purchase link for the appropriate product.

[1188] Step 12:

[1189] The system sends purchase information for the suggested missing items to the terminal and notifies the user. The input is the missing item information retrieved by the server from an external database. The output is the purchase information sent to the terminal. The terminal displays "You can purchase business leather shoes. Click this link: [link]".

[1190] This allows users to easily choose the perfect outfit for the day and conveniently purchase any missing items.

[1191] (Application Example 1)

[1192] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1193] Traditional systems offer limited suggestions based on user schedules and weather information, which is insufficient to meet the diverse needs of users. In particular, there is a need for more comprehensive suggestions tailored to users' lifestyles, encompassing not only clothing but also information on food and other items. Furthermore, the search and suggestion functions for items not owned by the user are lacking and need to be streamlined and improved.

[1194] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[1195] In this invention, the server includes means for managing a database of items owned by the user, means for acquiring the user's location information and schedule information, means for acquiring external information based on the acquired location information and schedule information, means for making optimal suggestions based on the owned items, location information, schedule information and external information, means for searching for and suggesting items missing from the suggested content from external sources, and means for tagging image data taken by the user and classifying it based on the tag information. As a result, the user can comprehensively manage their belongings and external information and receive suggestions optimized for their lifestyle.

[1196] A "user" refers to an individual who uses the system to manage their belongings and information and receive optimal suggestions.

[1197] "Items" refers to a broad range of items owned by the user (such as clothing or food).

[1198] A "database" refers to an information collection used to store and manage information and image data of items owned by a user.

[1199] "Location information" refers to digital information that indicates the user's current location.

[1200] "Schedule information" refers to information about the user's future plans and schedules.

[1201] "External information" refers to information obtained from external data sources based on the user's location and schedule information, such as weather information and food-related information.

[1202] "Suggestion" refers to the act of recommending the most suitable actions or items based on the user's possessions and acquired external information.

[1203] "Searching" refers to the act of finding missing items that a user does not possess by searching an external database.

[1204] "Image data" refers to photographic data that users have taken and uploaded to the system.

[1205] A "tag" refers to a keyword that indicates attributes or classification information attached to image data.

[1206] "Classification" refers to the act of organizing image data or items into categories based on tag information.

[1207] Specific embodiments of this invention will be described in detail below.

[1208] System Overview

[1209] This invention is a system that manages items owned by a user (e.g., clothing, food, etc.) and makes optimal suggestions based on the user's location and schedule information. Furthermore, it is a system that searches for items that are missing from the suggested items and notifies the user.

[1210] Hardware and software to be used

[1211] Server: Provides the core computing and database management functions of this system. The server runs on Node.js and Express.

[1212] Database: MongoDB is used to store user-owned items and image data.

[1213] Weather Information API: Use the OpenWeatherMap API to obtain current weather information.

[1214] Food Delivery API: Use the Uber Eats API and DoorDash API to retrieve information about food delivery.

[1215] Image analysis module: Using TensorFlow.js, tag information is generated from images taken by the user.

[1216] Client device: Users use devices such as smartphones to input location and schedule information and receive suggestions.

[1217] Data flow and processing

[1218] 1. Management of the item database:

[1219] Users take photos of their belongings (e.g., clothing and food items) and tag them. The tagged photo data is sent to a server and stored in a database. This allows users to easily manage their possessions.

[1220] 2. Acquisition of location and schedule information:

[1221] Users input their daily schedule and current location via a device such as a smartphone. This information is sent to a server and stored for subsequent processing.

[1222] 3. Acquisition of external information:

[1223] The server accesses a weather information API based on location data to obtain current weather information. It also obtains food-related information from delivery service APIs.

[1224] 4. Generating the optimal proposal:

[1225] The server analyzes database information, location information, schedule information, and acquired external information to generate the most suitable suggestions for the user. For example, if the user has plans to go out on a nice day, it will suggest nearby food delivery services.

[1226] 5. Search for and suggest missing items:

[1227] If the proposed items are missing, the server accesses an external database to search for them. It then notifies the user of the detailed information and a purchase link.

[1228] Specific example

[1229] For example, if a user wants to eat pizza, they take a picture of it and tag it. The system analyzes the picture and adds it to the user's database. Then, on the day the user plans to go out, the system retrieves the current weather information and suggests the best food delivery service. At this time, the user will be notified with a suggestion such as, "It's sunny, so we recommend delivery from a nearby pizza restaurant. Check Uber Eats or DoorDash." Furthermore, if the system determines that there is a shortage of cheese, a notification will appear saying, "You can purchase cheese. Please purchase it from this link."

[1230] Example of a prompt

[1231] Based on the following information, please generate a program that suggests the most suitable food delivery options to the user.

[1232] Photos and tag information of dishes the user wants to eat.

[1233] Current location and weather information

[1234] Plans to go out

[1235] You also need to provide links to purchase any missing ingredients.

[1236] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[1237] Step 1:

[1238] The system takes images of items owned by the user and adds tags to them. The user uses their smartphone to take a picture of an item they own (for example, a dish they want to eat). Then, they add tags such as "pizza" or "hamburger" to that picture. The input is image data and tag information, and the output is tagged image data. The device sends this data to the server.

[1239] Step 2:

[1240] The server receives tagged image data and stores it in a database. The server receives tagged image data sent from the terminal and stores it in a MongoDB database. The input is tagged image data, and the output is the data stored in the database. The image data is analyzed using TensorFlow.js to generate appropriate tags.

[1241] Step 3:

[1242] The user enters their daily schedule and current location into the device. The user uses a smartphone to enter their outing plans and current location. This information is sent to the server. The input is the schedule and location information, and the output is the schedule and location information sent to the server.

[1243] Step 4:

[1244] The server retrieves weather information based on location data. The server uses the location data to access the OpenWeatherMap API and retrieve current weather information. The input is location data, and the output is current weather information. The server saves the weather information to a database.

[1245] Step 5:

[1246] The server analyzes the user's database and acquired external information to generate optimal suggestions. The server analyzes item information, location information, schedule information, and weather information stored in the user's database to generate the most suitable suggestions for the user. For example, on a sunny day, a suggestion such as "You should order from a nearby pizza restaurant" might be generated to match the user's planned outing. The input consists of database information and external information, and the output is the generated suggestion.

[1247] Step 6:

[1248] The server searches for any missing items in the proposed content and generates detailed information and purchase links. The server accesses an external database to search for items missing from the proposed content. For example, if it determines that "cheese is missing," it will generate a purchase link. The input is the proposed content and information from the external database, and the output is detailed information and purchase links.

[1249] Step 7:

[1250] The server notifies the terminal of the proposed content and details of the missing items. The server sends the generated proposed content and details of the missing items to the terminal and notifies the user. The user receives the notification via their smartphone and can take appropriate action. The input is the proposed content and details of the missing items, and the output is the notification to the user.

[1251] Through these steps, users can efficiently manage their possessions and receive optimal suggestions based on external information. This process consistently involves data processing and calculations, providing information optimized for the user's lifestyle.

[1252] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[1253] Regarding embodiments for carrying out the present invention, the details of a clothing selection support system that combines an emotion engine will be described.

[1254] User clothing database management

[1255] The process begins with users taking photos of their clothes and tagging each photo (e.g., "white T-shirt," "blue jeans," etc.). The tagged photo data is sent from the device to a server, which stores the received data in a database. This database is used to systematically manage each user's clothing.

[1256] Obtaining schedule and weather information

[1257] When a user enters their daily schedule into their device, the device stores the schedule information linked to the user's current location. Next, the device accesses a weather information API based on the location information to obtain current weather information (e.g., temperature, weather). This information is sent from the device to the server, which analyzes the received information and prepares to generate the optimal outfit.

[1258] Emotion recognition by an emotion engine

[1259] When a user uses the device, the built-in emotion engine recognizes emotions from the user's facial expressions and voice. This emotion data reflects the user's current mood and psychological state. The recognized emotion information is sent from the device to the server.

[1260] Coordination suggestions

[1261] The server analyzes the received schedule information, location information, weather information, and sentiment information. Based on the analysis results, it selects appropriate clothing from the user's database. For example, given conditions such as "business meeting" + "sunny, 25°C" + "user is feeling well," the server determines that "formal business attire is appropriate, and bright-colored items should be chosen," and suggests a combination such as a black jacket, white T-shirt, and blue jeans. The suggested outfit is sent to the terminal and notified to the user.

[1262] Suggestions for missing items

[1263] The server consults the user's database to identify items needed for an outfit that the user does not possess. For example, if it determines that the user is missing business shoes, the server accesses an external database to search for them. If a suitable product is found, it sends detailed information and a purchase link to the user's device and notifies them. The notification content is customized based on the emotions recognized by the emotion engine. For example, if the user is relaxed, a friendly message such as "We have some great business shoes. Purchase them here: link" will be displayed.

[1264] Coordination and purchase suggestion notification

[1265] The system visually and clearly notifies the user of outfit information received by the device and purchase suggestions for missing items. The notified information includes images and text, allowing the user to view the details. Furthermore, the message content is customized based on emotional information generated by an emotion engine, taking care to ensure that the user receives it in a more positive way.

[1266] Specific example

[1267] Suppose a user receives the following outfit and purchase suggestions under the conditions of "business meeting" + "sunny, 25℃" + "user is feeling well".

[1268] The device suggested an outfit consisting of a "black jacket," a "white T-shirt," and "blue jeans."

[1269] The device simultaneously displayed a notification stating, "We are short on business shoes. Please purchase them from the link below."

[1270] In this way, users can easily find the perfect outfit for the day and conveniently purchase any missing items as needed. Furthermore, because the emotion engine adapts notifications to the user's emotions, it is possible to use the system more comfortably.

[1271] The above describes the embodiments for carrying out the present invention. By combining the emotion engine, personalized suggestions tailored to the user's psychological state are realized, significantly improving the user experience.

[1272] The following describes the processing flow.

[1273] Step 1:

[1274] Users take photos of the clothes they own and tag each photo (for example, "white T-shirt," "blue jeans," etc.).

[1275] Step 2:

[1276] The device sends tagged photo data to the server. The transmitted data is in JSON format and includes the photo and tags.

[1277] Step 3:

[1278] The server saves the received data to the database. This allows for systematic management of each user's belongings.

[1279] Step 4:

[1280] The user enters their schedule for the day into the device. Schedule information is obtained through the calendar function or direct input.

[1281] Step 5:

[1282] The device obtains the user's current location information using GPS functionality. The location information is expressed in the form of latitude and longitude.

[1283] Step 6:

[1284] The device accesses a weather information API based on its location to retrieve current weather information (e.g., temperature, weather conditions).

[1285] Step 7:

[1286] The device sends the acquired schedule and weather information to the server. The transmitted data is in JSON format and includes location information, schedule, and weather information.

[1287] Step 8:

[1288] When a user uses the device, the built-in emotion engine recognizes emotions from the user's facial expressions and voice. This emotion data reflects the user's current mood and psychological state. Emotional information is transmitted from the device to the server.

[1289] Step 9:

[1290] The server analyzes the received schedule, location, weather, and sentiment information. Based on the analysis, it selects appropriate clothing from the user's database. For example, given the conditions "business meeting" + "sunny, 25°C" + "user is feeling well," the server would determine that "formal business attire is appropriate, and bright-colored items should be chosen."

[1291] Step 10:

[1292] The server sends generated outfit information (e.g., black jacket, white T-shirt, blue jeans) to the terminal. The information sent includes the selected clothing items and their details.

[1293] Step 11:

[1294] The device notifies the user of the outfit information it has received. The notification is displayed with images and text for easy visual understanding. In addition, the message is customized based on the results of the emotion engine's recognition. If the user is feeling energetic, a positive message such as "This outfit will help you have an energetic day!" will be displayed.

[1295] Step 12:

[1296] The server consults the user's database to identify items needed for an outfit that the user does not possess. If it determines that "business shoes are missing," the server accesses an external database to search for those items.

[1297] Step 13:

[1298] The server retrieves product information from an external database and sends it to the terminal. The transmitted information includes product details, price, and a purchase link.

[1299] Step 14:

[1300] The device notifies the user of purchase suggestions it has received. The notification content is customized based on the user's emotional state. For example, if the user is relaxed, a friendly message such as "We have some great business shoes. Purchase them here: link" will be displayed.

[1301] The above outlines the specific processing steps of the system that combines the emotion engine.

[1302] (Example 2)

[1303] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1304] In modern society, choosing appropriate attire amidst a busy daily life is difficult for many. Efficiently deciding on an outfit that suits the weather and schedule is particularly challenging. Furthermore, clothing choices are often influenced by emotions and psychological states, and these factors cannot be ignored. Conventional systems struggle to integrate these diverse factors to provide optimal suggestions, resulting in a poor user experience.

[1305] The identification processing performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for managing a database of clothing owned by the user, means for acquiring the user's location information and schedule information, means for acquiring weather information based on the acquired location information and schedule information, means for recognizing emotions from the user's facial expressions and voice, means for suggesting the optimal outfit combination based on the owned clothing, location information, schedule information, weather information and emotion information, and means for searching for and suggesting items missing from the suggested outfit from an external source. This enables the integration of diverse information to provide the user with personalized and optimal outfit suggestions.

[1306] "Means for managing a database of clothing owned by a user" refers to systems or software that collect, store, and systematically organize image data and tag information of clothing owned by a user.

[1307] "Means for acquiring user location and schedule information" refers to systems or software used to collect and store a user's current location and schedule for the day.

[1308] "Means for obtaining weather information based on acquired location and schedule information" refers to APIs or systems that obtain weather forecast information based on the user's location and schedule information.

[1309] "Means of recognizing emotions from a user's facial expressions and voice" refers to systems and software that analyze a user's facial expressions and voice to recognize the user's emotional state.

[1310] "Means of suggesting the optimal clothing combination based on the user's clothing, location information, schedule information, weather information, and emotional information" refers to algorithms and systems that integrate and analyze various user information to suggest the optimal clothing combination.

[1311] "Methods for searching for and suggesting items missing from a proposed outfit" refers to systems or algorithms that search an external database for items necessary for a proposed outfit that the user does not own, and then present the results to the user.

[1312] "Means of notifying users of suggested outfits and information on missing items" refers to systems or software that notify users of generated outfit suggestions and information on purchasing missing items.

[1313] "A means of attaching tags to image data of clothing owned by a user and classifying them based on that tag information" refers to a system or software that attaches specific tags to images of clothing taken by a user and categorizes them based on that tag information.

[1314] This invention details a clothing selection support system that incorporates an emotion engine. This system manages the user's clothing data and suggests optimal outfits based on external data such as schedule information and weather information. Furthermore, by recognizing the user's emotions, it provides individually optimized suggestions.

[1315] User clothing database management

[1316] The process begins with the user taking photos of their clothes and attaching tags. The tagged photos are sent from the device to a server, which then stores the received data in a database. This database is used to systematically manage each user's clothing. A smartphone or tablet is required to retrieve the data, and an internet connection is needed to transmit it.

[1317] Obtaining schedule and weather information

[1318] When a user enters their daily schedule into the device, the device stores the schedule information, linking it to the user's current location. This location information is obtained using GPS. Next, the device accesses a weather information API based on the location information to obtain current weather information (e.g., temperature, weather conditions). The obtained information is sent from the device to the server for preparation for analysis. Weather information APIs used include the OpenWeatherMap API and the Weather API.

[1319] Emotion recognition by an emotion engine

[1320] When a user uses a device, the built-in emotion engine (e.g., Microsoft Azure's Emotion API or Google Cloud's Speech-to-Text) is activated to recognize emotions from the user's facial expressions and voice. This recognized emotion data is then sent from the device to a server.

[1321] Coordination suggestions

[1322] The server performs analysis based on the schedule information, location information, weather information, and sentiment information it receives. Based on this analysis, it selects appropriate clothing from the user's database. For example, given conditions such as "business meeting" + "sunny, 25°C" + "user is feeling well," the server will determine that "formal business attire is appropriate, and bright-colored items should be chosen," and suggest a combination such as a black jacket, white T-shirt, and blue jeans. The suggested result is sent to the terminal and notified to the user.

[1323] Suggestions for missing items

[1324] The server accesses the user's database to identify items needed for an outfit that the user does not possess. For example, if it determines that the user is missing business shoes, it accesses external databases (e.g., Amazon API, Rakuten Market API) to search for the item. If a suitable product is found, it sends the detailed information and purchase link to the user's device and notifies them.

[1325] Coordination and purchase suggestion notification

[1326] The device visually and clearly notifies the user of outfit information received and purchase suggestions for missing items. These notifications include images and text and are customized based on emotional information from an emotion engine. The user can review the suggestions and purchase items as needed.

[1327] Specific example

[1328] For example, consider a scenario where a user receives the following outfit suggestion based on the conditions "business meeting" + "sunny, 25℃" + "user is feeling well".

[1329] The device suggests outfits such as a "black jacket," "white T-shirt," and "blue jeans."

[1330] The device displays a notification that says, "We are short on business shoes. Please purchase them from the link below."

[1331] In this way, users can easily find the perfect outfit for the day and conveniently purchase any missing items as needed.

[1332] Examples of prompts for generative AI models

[1333] The following are examples of prompts to input into the generated AI model.

[1334] Please write a natural language description of a program that provides coordination suggestions to a user under the conditions of "business meeting," "sunny, 25℃," and "user is feeling well."

[1335] Thus, by implementing the present invention, personalized suggestions tailored to the user's psychological state are realized, significantly improving the user experience.

[1336] The flow of the specific processing in Example 2 will be explained using Figure 13.

[1337] Step 1: Register clothing data

[1338] The user takes photos of their own clothes and tags each photo (e.g., "white T-shirt," "blue jeans"). The device then processes this tagged photo data and sends it to the server. The server parses the received data and stores it in a database.

[1339] Input: Photos and tag information of clothing

[1340] Output: Clothing data stored in the database

[1341] Specific operation: The user takes a photo of clothing with their smartphone. The device displays a tag input screen, and the user enters the tags. The device sends the image and tags to the server. The server saves them to the database.

[1342] Step 2: Obtain schedule and location information

[1343] The user enters their schedule for the day into their device. The device obtains the user's current location information (GPS) and stores it in conjunction with the schedule information.

[1344] Input: User's schedule information and location information

[1345] Output: Saved schedule information and location information

[1346] Specific operation: The user enters an event into the device's calendar app. The device obtains location information from GPS. The device saves the event information and location information.

[1347] Step 3: Obtain weather information

[1348] The device accesses a weather information API (e.g., OpenWeatherMap API) based on its location to obtain current weather information (e.g., temperature, weather conditions).

[1349] Input: Location information

[1350] Output: Acquired weather information

[1351] Specific operation: The device accesses the weather information API and retrieves weather information. The retrieved weather information is sent to the server.

[1352] Step 4: Performing emotion recognition

[1353] While the user is using the device, the built-in emotion engine (e.g., Microsoft Azure Emotion API) analyzes facial expressions and voice to recognize emotions. The recognized emotion data is then sent from the device to the server.

[1354] Input: User's facial expressions and voice data

[1355] Output: Recognized emotion data

[1356] Specific operation: The user takes a selfie with the device's camera or speaks into the microphone. The device activates its emotion engine and analyzes facial expressions and voice. The emotion data is sent to the server.

[1357] Step 5: Generating Coordination Proposals

[1358] The server analyzes the received schedule information, location information, weather information, and sentiment information to generate the optimal outfit. It selects suitable clothing from the user's database. For example, given conditions such as "business meeting" + "sunny, 25℃" + "user is feeling good," it will suggest the optimal combination (e.g., black jacket, white T-shirt, blue jeans).

[1359] Input: Schedule information, location information, weather information, sentiment information

[1360] Output: Proposed coordination

[1361] Specific operation: The server analyzes the received information. It selects clothing from the user database. It generates the suggested clothing. It sends the generated result to the terminal.

[1362] Step 6: Detecting missing items

[1363] The server uses the proposed outfit to determine if any items are missing by referencing the user's database. If missing items are found, it accesses an external database (e.g., Amazon API) to search for them and generates the results.

[1364] Input: Proposed outfits, user database

[1365] Output: Search results for missing items

[1366] Specific operation: The server checks the user's database. It identifies missing items. It accesses an external database to search for the items. It sends the search results to the terminal.

[1367] Step 7: Proposal and Purchase Notification

[1368] The device notifies the user of outfit information it has received and purchase suggestions for missing items. The notification includes images and text and is customized based on emotional information from the emotion engine.

[1369] Input: Suggested outfit, purchase information for missing items

[1370] Output: Visually customized notifications

[1371] Specific actions: The device receives suggestion information from the server. The suggested outfit is displayed as an image and text. Purchase links for missing items are displayed. The message is customized based on information from the emotion engine.

[1372] (Application Example 2)

[1373] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1374] In modern society, users often find it difficult to choose clothing that suits their diverse schedules and lifestyles. Furthermore, some users desire content recommendations based on their current mood and environment. However, no system exists to address these individual needs; there is a need for a system that reduces user effort and provides appropriate suggestions in a streamlined manner.

[1375] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[1376] In this invention, the server includes means for managing a database of clothing owned by the user; means for acquiring the user's location information and schedule information; means for acquiring weather information based on the acquired location information and schedule information; means for suggesting the optimal outfit combination based on the owned clothing, location information, schedule information, and weather information; means for searching for and suggesting items missing from the suggested outfit from an external source; and means for recognizing the user's emotional state and recommending content based on that emotion. As a result, the user can not only easily choose the outfit best suited to their schedule and environment for the day, but also receive content recommendations that match their emotions.

[1377] "User" refers to a person or individual who uses this system.

[1378] A "clothing database" is a collection of data used to systematically store and manage information about various types of clothing owned by a user.

[1379] "Location information" refers to information that indicates the user's current location or geographical position.

[1380] "Schedule information" refers to information about plans and schedules that a user will have in the future.

[1381] "Weather information" refers to information about current or future weather and meteorological conditions.

[1382] "Optimal clothing combination" refers to the set of clothes that is most suitable for the user, based on conditions such as weather and schedule.

[1383] "Missing items" refer to clothing or accessories that the user does not currently own but are required for the suggested outfit combination.

[1384] "Emotional state" refers to the user's current emotional and psychological state.

[1385] "Content" is a general term for digital information and entertainment provided to users, such as videos, music, and text.

[1386] The system for implementing this invention manages a database of the user's clothing and suggests the optimal clothing combination and any missing items based on the user's location, schedule, weather, and emotional state. It also recommends digital content according to the user's emotional state.

[1387] User clothing database management

[1388] Users take photos of their clothes and tag each photo (e.g., "white T-shirt" or "blue jeans"). The tagged photo data is sent from the device to the server, which stores the received data in a database. This database systematically manages each user's clothing.

[1389] Acquisition of location and weather information

[1390] When a user enters their daily schedule into the device, the device stores the schedule information linked to the user's current location. Next, the device accesses a weather information API based on the location information to obtain current weather information (temperature and weather conditions). This information is sent from the device to the server, which analyzes the received information and prepares to generate the optimal outfit.

[1391] Emotion recognition by an emotion engine

[1392] When a user uses the device, the built-in emotion engine recognizes emotions from the user's facial expressions and voice. This emotion data reflects the user's current mood and psychological state. The recognized emotion information is sent from the device to the server.

[1393] Coordination suggestions and suggestions for missing items

[1394] The server analyzes the received schedule information, location information, weather information, and sentiment information. Based on the analysis results, it selects appropriate clothing from the user's database. For example, if given conditions such as "business meeting" + "sunny, 25°C" + "user is feeling well," the server will determine that "formal business attire is appropriate, and choose bright-colored items," and the suggested clothing will be sent to the terminal and notified to the user.

[1395] Furthermore, the server consults the user's database to identify items needed for an outfit that the user does not possess. For example, if it determines that the user is missing business shoes, the server accesses an external database to search for the item. If a suitable product is found, it sends detailed information and a purchase link to the user's device and notifies them. The notification content is customized based on the emotions recognized by the emotion engine.

[1396] Digital content recommendations

[1397] Furthermore, the server recommends content appropriate to the user based on their emotional state. For example, if the user is "happy," the server will recommend "comedy movies" or "funny videos." This recommendation is based on a combination of emotional data and weather information.

[1398] Hardware and software to be used

[1399] Hardware: Smartphones (such as Apple iPhone and Samsung Galaxy)

[1400] Software: Python programs, weather information APIs (such as OpenWeatherMap), emotion recognition APIs (such as Microsoft Azure Emotion API)

[1401] Specific example

[1402] For example, under the conditions of "business meeting" + "sunny, 25℃" + "user is feeling well," a user might receive the following outfit and purchase suggestions.

[1403] The suggested outfit was a "black jacket," a "white T-shirt," and "blue jeans."

[1404] At the same time, a notification appears stating, "We are short on business shoes. Please purchase them from the link below."

[1405] As described above, users can easily find the perfect outfit for their day's schedule and easily obtain the necessary items. Furthermore, because the emotion engine adapts notifications to the user's emotions, they can use the system more comfortably.

[1406] Example of a prompt:

[1407] If the user's emotional state is "happy," the weather is "sunny," and the temperature is "25°C," please generate a list of recommended video content.

[1408] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[1409] Step 1:

[1410] The user creates their own clothing database. They take photos of their clothes with their device and tag each photo. The tagged photo data is sent from the device to the server, where it is stored in the database. The input is the user's clothing photos and tags, and the output is the clothing information stored in the server's database.

[1411] Step 2:

[1412] The user enters their schedule for the day into the terminal. The terminal retrieves the entered schedule information and the user's current location. This information is then sent from the terminal to the server. The input is the user's schedule information and location, and the output is the schedule information and location sent to the server.

[1413] Step 3:

[1414] The server accesses a weather information API based on the received location information to obtain current weather information (temperature and weather conditions). The input is location information, and the output is weather information.

[1415] Step 4:

[1416] When a user uses the device, the device's built-in emotion engine analyzes the user's facial image and voice to perform emotion recognition. The recognized emotion information is sent from the device to the server. The input is the user's facial image or voice, and the output is the recognized emotion information.

[1417] Step 5:

[1418] The server analyzes the received schedule information, location information, weather information, and sentiment information. Based on this, it selects the most suitable clothing combination from the user's database. For example, if the user has a "business meeting," the weather is sunny at 25°C, and the user is feeling well, the server will determine that formal attire is appropriate. The input is schedule information, location information, weather information, and sentiment information, and the output is a suggestion for the most suitable clothing.

[1419] Step 6:

[1420] The server consults the user's clothing database to identify any missing items in the suggested outfit. For example, if "business shoes are missing," the server accesses an external database to search for those items. If a suitable product is found, it sends the details and a purchase link to the terminal and notifies the user. The input is the user's clothing database and the suggested outfit, and the output is the search results for the missing items and the notification.

[1421] Step 7:

[1422] The server recommends appropriate digital content based on the user's emotional state. For example, if the user is "happy," the server recommends "comedy movies" or "funny videos." The recommendation list is generated based on a combination of emotional data and weather information. The input is the user's emotional information and weather information, and the output is the recommended content list.

[1423] Step 8:

[1424] The device notifies the user of optimal clothing suggestions received from the server, purchase links for missing items, and recommended digital content. The notification content is displayed in a visually easy-to-understand format, allowing the user to review it. Input is suggestion data and recommended content from the server, and output is the notified information.

[1425] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[1426] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1427] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[1428] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1429] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[1430] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[1431] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[1432] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[1433] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[1434] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[1435] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[1436] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[1437] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[1438] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1439] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[1440] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[1441] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[1442] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[1443] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[1444] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[1445] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[1446] The following is further disclosed regarding the embodiments described above.

[1447] (Claim 1)

[1448] A means of managing a database of clothing owned by a user,

[1449] Means for obtaining the user's location information and schedule information,

[1450] A means for obtaining weather information based on acquired location information and schedule information,

[1451] A means of suggesting the optimal clothing combination based on the clothes you possess, location information, schedule information, and weather information,

[1452] A system that includes a means of searching for and suggesting items that are missing from a proposed outfit.

[1453] (Claim 2)

[1454] The system according to claim 1, comprising means for notifying the user of suggested clothing and information on missing items.

[1455] (Claim 3)

[1456] The system according to claim 1, which includes means for attaching tags to image data of clothing owned by a user and classifying them based on the tag information.

[1457] "Example 1"

[1458] (Claim 1)

[1459] A means of managing a database of clothing owned by a user,

[1460] Means for obtaining the user's location information and schedule information,

[1461] A means for obtaining weather information based on acquired location information and schedule information,

[1462] A means of suggesting the optimal clothing combination based on the clothes you possess, location information, schedule information, and weather information,

[1463] A method for searching for and suggesting items that are missing from the proposed outfit from an external source,

[1464] A means of assigning tags to image data in a user's clothing database and classifying clothing based on the tag information,

[1465] A system including means for generating prompt sentences to suggest clothing using a generative AI model.

[1466] (Claim 2)

[1467] The system according to claim 1, comprising means for notifying the user of suggested clothing and information on missing items.

[1468] (Claim 3)

[1469] The system according to claim 1, which includes means for attaching tags to image data of clothing owned by a user and classifying the clothing based on the tag information.

[1470] "Application Example 1"

[1471] (Claim 1)

[1472] A means of managing a database of items owned by a user,

[1473] Means for obtaining the user's location information and schedule information,

[1474] A means for acquiring external information based on acquired location information and schedule information,

[1475] A means of providing optimal suggestions based on the items possessed, location information, schedule information, and external information,

[1476] A method for searching for and suggesting items that are missing from the proposed content from an external source,

[1477] A system that includes a means of tagging image data taken by users and classifying it based on that tag information.

[1478] (Claim 2)

[1479] The system according to claim 1, comprising means for notifying the user of the proposed content and information on missing items.

[1480] (Claim 3)

[1481] The system according to claim 1, which includes means for acquiring weather information and food-related information as external information and making optimal suggestions.

[1482] "Example 2 of combining an emotion engine"

[1483] (Claim 1)

[1484] A means of managing a database of clothing owned by a user,

[1485] Means for obtaining the user's location information and schedule information,

[1486] A means for obtaining weather information based on acquired location information and schedule information,

[1487] A means of recognizing emotions from the user's facial expressions and voice,

[1488] A means of suggesting the optimal clothing combination based on the clothes one possesses, location information, schedule information, weather information, and emotional information,

[1489] A system that includes a means of searching for and suggesting items that are missing from a proposed outfit.

[1490] (Claim 2)

[1491] The system according to claim 1, comprising means for notifying the user of suggested clothing and information on missing items.

[1492] (Claim 3)

[1493] The system according to claim 1, which includes means for attaching tags to image data of clothing owned by a user and classifying them based on the tag information.

[1494] "Application example 2 when combining with an emotional engine"

[1495] (Claim 1)

[1496] A means of managing a database of clothing owned by a user,

[1497] Means for obtaining the user's location information and schedule information,

[1498] A means for obtaining weather information based on acquired location information and schedule information,

[1499] A means of suggesting the optimal clothing combination based on the clothes you possess, location information, schedule information, and weather information,

[1500] A method for searching for and suggesting items that are missing from the proposed outfit from an external source,

[1501] A means of recognizing the user's emotional state and recommending content based on that emotion,

[1502] A system that includes this.

[1503] (Claim 2)

[1504] The system according to claim 1, comprising means for notifying the user of suggested clothing and information on missing items.

[1505] (Claim 3)

[1506] The system according to claim 1, which includes means for attaching tags to image data of clothing owned by a user and classifying them based on the tag information.

[1507] (Claim 4)

[1508] The system according to claim 1, comprising means for notifying a user of recommended content based on emotional state and weather information. [Explanation of Symbols]

[1509] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. A means of managing a database of clothing owned by a user, Means for obtaining the user's location information and schedule information, A means for obtaining weather information based on acquired location information and schedule information, A means of suggesting the optimal clothing combination based on the clothes you possess, location information, schedule information, and weather information, A system that includes a means of searching for and suggesting items that are missing from a proposed outfit.

2. The system according to claim 1, comprising means for notifying the user of the suggested clothing and information on any missing items.

3. The system according to claim 1, which includes means for attaching tags to image data of clothing owned by a user and classifying them based on the tag information.

Images