Information processing system

Abstract

The present invention provides an information processing system comprising: means for inputting user information; means for acquiring store information; means for generating a menu based on the user information and the store information; means for displaying the generated recipe to a user; the invention relates to a device for guiding a desired food material.

Description

Technical Field

[0001] The technology disclosed herein relates to an information processing system. Background Technology

[0002] Japanese Patent Application Publication No. 2022-180282 discloses a method for controlling a role-based chatbot executed by at least one processor. The method includes the following steps: receiving a user's speech; adding the user's speech to a prompt word, the prompt word containing instruction statements associated with an explanation of the chatbot's role; encoding the prompt word; and inputting the encoded prompt word into a language model to generate a chatbot response to the user's speech.

[0003] When recommending products and recipes to users, existing supermarkets or stores often fail to fully consider individual user needs, health conditions, dietary preferences, and the store's actual inventory and information on soon-to-expire items. This leads to increased food waste, low sales efficiency, and a poor user experience. Furthermore, personalized recommendations for adding items to the shopping cart are difficult to implement, limiting in-store promotions and advertising by food manufacturers. Therefore, how to efficiently generate and recommend personalized recipes by combining user data with real-time store inventory information to effectively reduce food waste and boost sales is a pressing technical problem that needs to be solved. Summary of the Invention

[0004] This invention provides an information processing system, comprising: a device for inputting user information; a device for acquiring store information; a device for generating recipes based on the user information and store information; a device for displaying the generated recipes to the user; and a device for guiding users to the required ingredients. The system can also identify products nearing their expiration date or those with excess inventory, prioritizing the use of these products to generate recommended recipes, thereby promoting sales and reducing waste. Simultaneously, the system can intelligently recommend suitable beverages, candies, and other add-on items based on the user's purchase history and preferences, improving user satisfaction. Furthermore, the system can enable recipe integration and recommendation of advertised products from manufacturers, facilitating product promotion.

[0005] "User information" refers to data related to an individual or family using the system, including but not limited to name, number of family members, dietary preferences, budget, health status, cooking experience, etc.

[0006] "Store information" refers to various data related to retail stores, including information such as product inventory, prices, shelf life, promotional items, and product placement.

[0007] "Recipe" refers to a recipe plan generated based on user information and store information, which includes the required ingredients, quantities, preparation steps, and nutritional balance.

[0008] "Input device" refers to hardware or software tools used to receive and collect user information, such as smartphones, tablets and other terminals and their related software.

[0009] "Acquisition device" refers to hardware or software components used to collect and receive information about store merchandise, including APIs, sensing devices, etc.

[0010] "Generation device" refers to a system unit that can automatically generate recipe content based on different combinations of input conditions, and usually includes artificial intelligence algorithms, server applications, etc.

[0011] "Display device" refers to the hardware and software that presents the generated recipe content to users in a visual way, such as terminal screens, user interfaces, push notifications, etc.

[0012] "Guidance devices" refer to various hardware and software measures that provide users with the specific location of the ingredients they need in the store, navigation, and purchase tips.

[0013] "Products nearing their expiration date" refers to products in the store's inventory that are close to their expiration date and are about to expire.

[0014] "Overstocked goods" refers to goods that are in excessive quantities in stores and are selling slowly.

[0015] "Add-to-cart items" refer to supplementary products such as beverages and candies that are recommended by the system based on user habits and needs, aiming to enhance the user's shopping experience and increase the transaction amount per transaction.

[0016] "Advertised products" refer to specific products that are promoted by food manufacturers or related companies and pushed to users by the system in the form of recommended recipes. Attached Figure Description

[0017] Figure 1 This is a conceptual diagram illustrating an example of the configuration of the data processing system according to the first embodiment.

[0018] Figure 2 This is a conceptual diagram illustrating an example of the main functions of the data processing apparatus and smart device according to the first embodiment.

[0019] Figure 3 This is a conceptual diagram illustrating an example of the configuration of the data processing system according to the second embodiment.

[0020] Figure 4 This is a conceptual diagram illustrating an example of the main functions of the data processing device and smart glasses according to the second embodiment.

[0021] Figure 5 This is a conceptual diagram illustrating an example of the configuration of the data processing system according to the third embodiment.

[0022] Figure 6 This is a conceptual diagram illustrating an example of the main functions of the data processing apparatus and head-mounted terminal according to the third embodiment.

[0023] Figure 7 This is a conceptual diagram illustrating an example of the configuration of the data processing system according to the fourth embodiment.

[0024] Figure 8 This is a conceptual diagram illustrating an example of the main functions of the data processing device and robot according to the fourth embodiment.

[0025] Figure 9 This represents an emotion map that maps multiple emotions.

[0026] Figure 10 This represents an emotion map that maps multiple emotions.

[0027] Figure 11 This is a sequence diagram illustrating the processing flow of the data processing system of the first embodiment.

[0028] Figure 12 This is a sequence diagram illustrating the processing flow of the data processing system in Application Example 1.

[0029] Figure 13 This is a sequence diagram illustrating the processing flow of the data processing system of the second embodiment.

[0030] Figure 14 This is a sequence diagram illustrating the processing flow of the data processing system in Application Example 2. Detailed Implementation

[0031] Hereinafter, an example of an implementation of the system according to the present disclosure will be described with reference to the accompanying drawings.

[0032] First, let me explain the terminology used in the following instructions.

[0033] In the following embodiments, the processor (hereinafter referred to as "processor") with reference numerals may be a single computing device or a combination of multiple computing devices. Furthermore, the processor may be a single computing device or a combination of multiple computing devices. Examples of computing devices include CPU (Central Processing Unit), GPU (Graphics Processing Unit), GPGPU (General-Purpose computing on Graphics Processing Units), APU (Accelerated Processing Unit), etc.

[0034] In the following embodiments, RAM (Random Access Memory), as indicated in the figures, is a memory that temporarily stores information and is used as working memory by the processor.

[0035] In the following embodiments, the memory, as indicated by the reference numerals, is one or more non-volatile storage devices that store various programs and parameters. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), disks (e.g., hard disks), or magnetic tapes.

[0036] In the following embodiments, the communication I / F (Interface) with reference numerals is an interface that includes a communication processor and an antenna, etc. The communication I / F is responsible for communication between multiple computers. As an example of a communication specification applicable to the communication I / F, wireless communication specifications such as 5G (5th Generation Mobile Communication System), Wi-Fi (wireless fidelity) (registered trademark), or Bluetooth (registered trademark) can be listed.

[0037] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B". That is, "A and / or B" means that it can be only A, only B, or a combination of A and B. Furthermore, in this specification, when "and / or" connects to express more than three items, the same interpretation as "A and / or B" applies.

[0038] First Implementation Method

[0039] Figure 1 An example of the configuration of the data processing system 10 according to the first embodiment is shown.

[0040] like Figure 1 As shown, the data processing system 10 includes a data processing device 12 and an intelligent device 14. A server can be cited as an example of the data processing device 12.

[0041] The data processing apparatus 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" as understood in this disclosure. The computer 22 includes a processor 28, RAM 30, and memory 32. The processor 28, RAM 30, and memory 32 are connected to a bus 34. Furthermore, the database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0042] The smart device 14 includes a computer 36, a receiving device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, RAM 48, and memory 50. The processor 46, RAM 48, and memory 50 are connected to a bus 52. In addition, the receiving device 38, output device 40, camera 42, and communication I / F 44 are also connected to the bus 52.

[0043] The receiving device 38 includes a touchscreen 38A and a microphone 38B, and receives user input. The touchscreen 38A receives user input via touch by detecting contact with an indicator (e.g., a pen or finger). The microphone 38B receives user input via sound by detecting the user's voice. The control unit 46A in the processor 46 sends data representing the user input received by the touchscreen 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data representing the user input.

[0044] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting data in a form perceptible to the user 20 (e.g., sound and / or text). The display 40A displays visual information such as text and images according to instructions from the processor 46. The speaker 40B outputs sound according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0045] Communication I / F44 is connected to network 54. Communication I / F44 and 26 are responsible for sending and receiving various information between processor 46 and processor 28 via network 54.

[0046] Figure 2The diagram shows an example of the main functions of the data processing device 12 and the smart device 14.

[0047] like Figure 2 As shown, in the data processing apparatus 12, specific processing is performed by the processor 28. A specific processing program 56 is stored in the memory 32. The specific processing program 56 is an example of a "program" as understood in this disclosure. The processor 28 reads the specific processing program 56 from the memory 32 and executes the read specific processing program 56 on the RAM 30. Specific processing is implemented by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0048] The memory 32 stores a data generation model 58 and an emotion-specific model 59. The data generation model 58 and the emotion-specific model 59 are used by the specific processing unit 290. The specific processing unit 290 can use the emotion-specific model 59 to infer the user's emotions and perform specific processing based on the user's emotions. In the emotion inference function (emotion-specific function) using the emotion-specific model 59, various inferences and predictions related to the user's emotions are performed, including inferences and predictions of the user's emotions, but this is not limited to this example. Furthermore, emotion inference and prediction may also include, for example, emotion analysis (parsing).

[0049] In the smart device 14, the processor 46 performs the acceptance output processing. The memory 50 stores the acceptance output program 60. The acceptance output program 60 is used in conjunction with the data processing system 10 and the specific processing program 56. The processor 46 reads the acceptance output program 60 from the memory 50 and executes the read acceptance output program 60 on the RAM 48. The acceptance output processing is implemented by the processor 46 acting as the control unit 46A according to the acceptance output program 60 executed on the RAM 48. Furthermore, the smart device 14 has the same data generation model and emotion-specific model as the data generation model 58 and the emotion-specific model 59, and these models can also be used to perform the same processing as the specific processing unit 290. The acceptance output processing is implemented by the processor 46 acting as the control unit 46A according to the acceptance output program 60 executed on the RAM 48.

[0050] Alternatively, other devices besides the data processing device 12 may also have the data generation model 58. For example, a server device (e.g., a generation server) may have the data generation model 58. In this case, the data processing device 12 obtains the processing results (prediction results, etc.) using the data generation model 58 by communicating with the server device that has the data generation model 58. Furthermore, the data processing device 12 may be a server device or a user-held terminal device (e.g., a mobile phone, robot, home appliance, etc.). Next, an example of the processing of the data processing system 10 of the first embodiment will be described.

[0051] Example 1

[0052] The flow of a specific process in Example 1 will be described. Each part of the system described below is implemented by the data processing device 12 and the smart device 14. Furthermore, the data processing device 12 is referred to as the "server," and the smart device 14 is referred to as the "terminal."

[0053] In the modern supply chain and retail industry, food waste is becoming increasingly serious due to problems such as unsold goods nearing their expiration date, increased inventory management difficulties, and insufficient personalization of consumer demand. Furthermore, consumers struggle to access personalized recipes and product information tailored to their health conditions, dietary preferences, and budgets while shopping, and also lack efficient auxiliary recommendations and in-store guidance services. Existing technologies fail to effectively integrate and process multi-source, heterogeneous user information and store inventory information, and cannot achieve efficient matching and resource utilization between consumers and suppliers through intelligent means. Therefore, there is an urgent need for an intelligent system that can comprehensively process user and supplier attribute information, and automatically generate personalized cooking instructions, dynamic product recommendations, and store guidance using generative artificial intelligence models, in order to reduce food waste, improve the consumer shopping experience, and optimize inventory turnover efficiency.

[0054] The specific processing performed by the specific processing unit 290 of the data processing apparatus 12 in Embodiment 1 is achieved by the following means.

[0055] In this invention, the server includes an information processing device for acquiring and managing user attribute information, an information acquisition and data analysis device for periodically acquiring and parsing supply location attribute information, a generative artificial intelligence model invocation device for automatically generating personalized cooking instructions and recommendation messages based on the information, an information provision and guidance device for data push from the information processing terminal and in-store shopping guide display, and an analysis device for product status identification and priority recommendation processing. This enables efficient integration of multi-source attribute information from users and stores, dynamically provides customized cooking solutions and personalized product recommendations through a generative artificial intelligence model, intelligently analyzes product inventory and shelf-life status, prioritizes pushing near-expiry or overstocked products, assists users in efficient shopping, significantly reduces food waste, and improves supply chain management efficiency.

[0056] "User attribute information" refers to information that characterizes the features of an individual user, including but not limited to family composition, dietary preferences, budget, cooking experience, health status, allergens, etc.

[0057] "Supply location attribute information" refers to various data related to the supply location of goods, including commodity inventory information, price information, shelf life information, and promotional priority information.

[0058] "Information processing device" means any data processing unit used to collect, process, store or transmit information, including but not limited to computer systems or servers.

[0059] "Generative AI models" refer to AI algorithm systems that can automatically generate text or decision suggestions based on input data, such as natural language generation models based on deep learning.

[0060] "Cooking instruction information" refers to the guidance information automatically generated by the system that provides users with specific methods for preparing dishes and a list of required ingredients.

[0061] "Information processing terminal" refers to a device used by users to receive, display, and interact with system information, including but not limited to smartphones, tablets, and personal computers.

[0062] A "data parsing device" refers to a data processing unit that can analyze and structure the collected data.

[0063] "Guidance device" refers to a functional unit that provides users with helpful information such as product location, store guidance, and route planning.

[0064] "Input generation device" refers to a device that generates data formats and prompt statements that can be used as input to a model based on the collected information.

[0065] "Supplementary item information" refers to supplementary product information, such as beverages and snacks, that the system intelligently recommends to users outside the main shopping list based on their attributes and purchase history.

[0066] This invention provides an information processing system based on a generative artificial intelligence model, which can efficiently integrate user attribute information and supply location attribute information to automatically generate personalized cooking instructions, product recommendations, and store guide information, thereby reducing food waste and improving the user shopping experience. The following describes specific embodiments of this invention in conjunction with hardware and software.

[0067] In this invention, the server employs standard computer hardware, which can be a physical server or cloud computing resources, such as a cloud server environment. On the software side, the server is equipped with an operating system (such as Linux), a database management system (such as MySQL), backend services (such as using Node.js or Python Flask), and a scheduled task tool (such as Cron). The server also integrates generative artificial intelligence models (such as large-scale natural language processing models, including GPT-3 or GPT-4).

[0068] A terminal refers to a smart terminal device operated by the user, primarily a smartphone or tablet, running a mobile operating system (such as Android or iOS) that includes client applications. Applications can utilize cross-platform development frameworks (such as React Native) or native development environments. The terminal communicates with the server via a secure network (such as HTTPS).

[0069] In actual operation, users input their personal attribute information, such as family structure, budget, dietary preferences, health status, and cooking skills, through the terminal and upload it to the server via the application interface. The server periodically obtains information on the inventory, product prices, and shelf life of each supply location through API and stores it in a structured database.

[0070] The server combines user input and store information to dynamically generate prompts tailored to specific users and inventory status. It then uses an API to call a generative AI model to automatically generate cooking instructions suitable for various constraints, including user preferences, inventory levels, and health considerations. The server receives the text results generated by the AI ​​model, parses them, stores them in a database, and then sends them to the user's terminal via push notification.

[0071] When the device receives a push notification, a pop-up message automatically appears. After opening the app, the user can view recommended recipes, a detailed ingredient list, and the location of products in the store. The device automatically generates a shopping list based on the acquired data, which the user can edit according to their own needs.

[0072] In addition, the server uses the shelf life and quantity of inventory to intelligently screen for near-expiry and excess inventory products, and automatically generates recipe suggestions that prioritize the use of such products. At the same time, the server analyzes users' historical purchase records and profiles to customize personalized recommendations for additional products (such as beverages and snacks) for each user, and pushes these recommendations to the terminal.

[0073] For example, when user A opens the app, they enter the following information: "Our family consists of four people, our budget is 500 yuan per week, our child is allergic to milk and prefers a light taste." At this time, the server automatically collects the inventory information of the stores the user recently shopped at, organizes it into the following prompt, and sends it to the generative artificial intelligence model:

[0074] "Please create a healthy and light dinner recipe for a family of four who are allergic to milk, using seasonal vegetables and chicken, and provide a detailed list of the required ingredients and steps."

[0075] For example, if cucumbers and tofu in the inventory are nearing their expiration date, the server can automatically generate the following prompt:

[0076] "Please use cucumber and tofu as the main ingredients to generate a simple and easy-to-make home-style recipe suitable for spring and summer for family users, with detailed instructions and required ingredients."

[0077] This invention also supports the server generating the following recommendations based on the user's shopping history:

[0078] "Based on your past shopping habits, we recommend this week's special matcha drinks and snacks. Would you like to add them to your shopping list?"

[0079] Through the above methods, the entire information processing flow is completed through the coordinated interaction between the server, terminal, and user, realizing the functions of multi-source data integration, intelligent prompt generation, and personalized information push.

[0080] use Figure 11 The processing flow is explained.

[0081] Step 1:

[0082] The user opens the application on the terminal, enters user attribute information such as family composition, dietary preferences, budget, health status, and cooking experience, and then clicks submit. The input is information manually filled in by the user, and the output is structured JSON data. The terminal packages this data and sends it to the server interface via HTTPS protocol.

[0083] Step 2:

[0084] The server receives user attribute information from the terminal and performs field validation and integrity verification on the data. The input is user data in JSON format, and the output is updated records in the database. The server writes the parsed user information into the relevant tables in the database.

[0085] Step 3:

[0086] The server initiates API requests to the data interface of the supply locations via scheduled tasks to obtain real-time supply location attribute information such as product inventory, price, shelf life, and featured products. The input is the API request, and the output is the raw inventory data. After parsing the obtained data, the server transfers the valid information to the inventory table in the database.

[0087] Step 4:

[0088] The server analyzes the current inventory status and user demand based on saved user attribute information and supply location attribute information, and generates personalized prompts. The input is user attributes and inventory information, and the output is targeted text prompts. After processing these natural language prompts, the server prepares to call the generative artificial intelligence model.

[0089] Step 5:

[0090] The server invokes a generative artificial intelligence model, taking the generated prompts as input to obtain personalized cooking instructions containing the recipe, necessary ingredients, and preparation methods. The input is the personalized prompts, and the output is the detailed recipe text. The server then formats and checks the results before storing them in a database.

[0091] Step 6:

[0092] The server sends the newly generated recipe data to the corresponding user's terminal via a push notification service (such as using a push API). The input is the generated recipe data, and the output is the push notification received by the terminal. After receiving the push notification, the terminal prompts the user to click to view it via a pop-up window in its local notification system.

[0093] Step 7:

[0094] After the user clicks the notification, the terminal automatically requests detailed information about the recipe and the store location of related products from the server. The input is the recipe information request, and the output is the recipe details and store navigation data. The terminal displays the recipe details on the interface and presents the product locations on the screen as icons or text.

[0095] Step 8:

[0096] The terminal automatically generates a shopping list based on the displayed recipes, allowing users to check, delete, and add ingredients. Input is the user's commands (such as checking or deleting), and output is a dynamically updated shopping list. The terminal saves and updates the list's status in real time.

[0097] Step 9:

[0098] The server periodically analyzes the shelf life and inventory of the stock table, filters out near-expiry or surplus products, and uses these as the core ingredients to generate new personalized prompts. It then calls a generative AI model to generate new recipes that prioritize the consumption of these products. The input is inventory data, and the output is the new recipes and corresponding push notifications.

[0099] Step 10:

[0100] Based on the user's historical shopping records and attribute profile, the server uses a recommendation algorithm to select suitable auxiliary products (such as beverages and snacks) for the user and generates relevant recommendation messages to push to the terminal. The input is the user's historical data, and the output is the auxiliary product recommendation content.

[0101] Step 11:

[0102] After viewing the supplementary product recommendations on the terminal, users can choose whether to add the recommended products to their current shopping list. The input is the user's selection, and the output is the final updated shopping list. The terminal synchronizes and organizes the updated shopping list to facilitate efficient shopping for users in the store.

[0103] Application Example 1

[0104] The process flow corresponding to the specific processing in Use Case 1 will be described below. The various parts of the system described below are implemented by the data processing device 12 and the intelligent device 14. Furthermore, the data processing device 12 is referred to as the "server" and the intelligent device 14 is referred to as the "terminal".

[0105] Existing product recommendation systems generally suffer from the following problems: difficulty in generating optimal personalized recommendations based on diverse information such as individual user circumstances and preferences, real-time store inventory, and expiration dates; inability to dynamically collect and utilize users' emotional states for intelligent recommendations; low efficiency in handling near-expiration and excess inventory products, leading to food waste and lost sales opportunities; and a lack of targeted shopping path guidance, resulting in low user shopping efficiency. Furthermore, traditional systems lack accuracy in recommending additional products, offering a weak personalized experience and failing to effectively improve customer satisfaction and product sales.

[0106] The specific processing performed by the specific processing unit 290 of the data processing apparatus 12 in Application Example 1 is achieved by the following means.

[0107] In this invention, the server includes devices for inputting individual information, acquiring location information, generating and inputting prompts into a generative artificial intelligence model, displaying proposal information, guiding item acquisition, collecting and analyzing emotional states, and optimizing recommendations accordingly. This allows for the generation of more accurate and efficient personalized product combination solutions for users by combining individual states, real-time inventory, product attributes, and user emotional dynamics, thereby improving the sales rate of related products and store operational efficiency, while also providing users with intelligent shopping path guidance and high-value-added integrated services.

[0108] "Individual information" refers to information entered by users or users about an individual's basic situation, preferences, health status, budget, family structure, cooking experience, etc.

[0109] "Providing location information" refers to real-time or periodic data related to the places where goods are provided (such as stores, warehouses, etc.), such as inventory, prices, expiration dates, promotional activities, and product distribution.

[0110] "Information generation device" refers to a hardware or software system that can process and organize input data and output information suitable for a target purpose.

[0111] "Prompt statements" refer to specific formatted texts that are automatically generated by the system and input into generative artificial intelligence models to guide the models to output specific solutions or suggestions.

[0112] "Generative artificial intelligence models" refer to artificial intelligence models that can autonomously generate relevant text, information, or data results based on input prompts, including but not limited to large-scale language models.

[0113] "Proposal information" refers to user-oriented recommendation content generated based on individual information and location information, and analyzed by a generative artificial intelligence model. This includes product combinations, recipes, shopping suggestions, etc.

[0114] "Emotion recognition device" refers to a software or hardware system that can collect and analyze unstructured data such as an individual's facial expressions, voice, and behavior through cameras, sensors, microphones, or other input devices, and infer emotions or psychological states.

[0115] "State information" refers to data describing an individual's current psychological or emotional state obtained after analysis by an emotion recognition device.

[0116] "Optimized proposal information" refers to the recommended content output to users after the initial proposal information has been adjusted or personalized based on the individual's status information.

[0117] "Expiration date" refers to the expiration date by which a product is safe to consume or use.

[0118] "Inventory Quantity" refers to the current available quantity of a specific product at a given location.

[0119] "Additional items" refer to related products that are suggested in addition to the main recommended products, based on the user's history, attributes, and preferences, including beverages, snacks, etc.

[0120] "Acquisition records" refer to the historical purchase, browsing, and collection data records related to an individual in the storage system.

[0121] One embodiment of the present invention relates to a personalized recommendation system that combines generative artificial intelligence models and emotion recognition technology. Based on user information, location information, and the user's emotional state, it can dynamically provide users with optimal product proposals, route navigation, and additional product recommendations. The implementation of the present invention is described in detail below.

[0122] This invention can be composed of multiple hardware and software modules, including a user terminal device, a server device, a database system, a generative artificial intelligence model, a prompt generation unit, and an emotion recognition device. The user terminal device can be a smartphone, tablet, wearable device, or self-service terminal. The server device can be based on a general-purpose computer architecture, running a Linux operating system, and the core business logic is recommended to be developed using Python. The database system is recommended to use relational databases such as MySQL or PostgreSQL. The generative artificial intelligence model can adopt a general-purpose large-scale language model interface, such as an API service based on OpenAIGPT. The emotion recognition device can be a camera / microphone integrated into the terminal, working in conjunction with an AI emotion recognition service (such as a cloud-based emotion recognition API) for parsing user state information.

[0123] In practical applications, users open a dedicated application on their terminals and input personal information such as family structure, dietary preferences, budget, and health status. After preprocessing the data on the terminal, the data is transmitted to the server via HTTPS or a secure API. The server periodically or in real-time retrieves information such as inventory, prices, expiration dates, and promotions from the providing locations through the interface, preprocesses the data (including standardization, format conversion, and data cleaning), and stores it in the database.

[0124] The server generates prompts based on user information and location information provided in the database. For example, if a family of four prefers Japanese food, has a budget of 5000 yen, and the restaurant offers mackerel, cabbage, tofu, etc., the server generates the following prompt:

[0125] Please generate a Japanese family recipe for the user based on the following conditions.

[0126] User's family members: 4 people

[0127] Dietary preferences: Japanese cuisine

[0128] Budget: 5000 yen

[0129] Cooking experience: Intermediate

[0130] Physical condition: No special requirements

[0131] Available ingredients for the store: mackerel, cabbage, tofu, scallions, soy sauce

[0132] Please list the required ingredients and specific preparation steps in detail.

[0133] The server submits the prompt to a generative artificial intelligence model, which retrieves the recipe text via an API call. The server processes the output text, parsing out the dish name, ingredient list, and preparation process, and records them in the database. Simultaneously, it analyzes the store layout and location of the required ingredients.

[0134] After obtaining the recommended menu from the server, the terminal displays detailed recipes, the inventory of the required products, and their shelf location to the user. If the system detects tension, fatigue, or other emotions in the user's facial expressions, voice, or past operations, the emotion recognition device analyzes the emotion data. Based on this, the server adjusts the prompt message to: "The user is currently under stress. Please recommend a light and home-style menu to help relieve stress, with mackerel and tofu as the main ingredients. Avoid greasy and high-calorie foods, and provide a simple and easy-to-prepare process."

[0135] The server calls the generative AI model again to obtain a new, optimized recipe and pushes it to the user.

[0136] In addition, the server prioritizes clearing out items nearing their expiration date or with excess inventory based on stock levels, generating recommendations centered around these items. The system can also recommend supplementary items such as beverages and desserts based on user history, improving user experience and sales. For example: Based on your shopping list, we suggest pairing green tea and wagashi (Japanese sweets) as dessert. Would you like to add them to your shopping list?

[0137] These functional modules interact securely and efficiently via APIs, ensuring the system accurately reflects individual circumstances and external supply information, enabling personalized and intelligent product recommendations and shopping guidance. Users can directly select, accept, adjust, and rate recommended content through their terminals, ultimately optimizing the entire shopping experience and resource utilization efficiency.

[0138] use Figure 12 The processing flow is explained.

[0139] Step 1:

[0140] Users open the application on their devices and actively input individual information such as family member details, dietary preferences, budget, cooking experience, and health status. After undergoing basic validation on the device (e.g., mandatory field checks, formatting compliance), the input data is packaged as structured data. The input is the information provided by the user, and the output is standardized JSON or form data.

[0141] Step 2:

[0142] The terminal sends the user's input personal information to the server via secure protocols such as HTTPS. After receiving the data, the server performs data integrity and format validity checks, and data processing includes field cleaning and default completion. The input is the user information submitted by the terminal, and the output is the user profile record in the system database.

[0143] Step 3:

[0144] The server retrieves raw data such as inventory, prices, product expiration dates, and promotional information from providing locations (e.g., stores) via API interfaces, following periodic scheduled tasks or real-time triggering mechanisms. The server performs format standardization and anomaly cleanup on the raw data, removing invalid fields. The input is raw inventory data from the external interface, and the output is standardized store information saved in the database.

[0145] Step 4:

[0146] The server extracts the current user's profile information and the corresponding store's inventory and attribute data from the database. Based on these two types of data, it automatically constructs prompts for a generative artificial intelligence model. The input is the user profile and store inventory information; the generation logic matches key fields and concatenates the text. The output is a complete natural language prompt.

[0147] Step 5:

[0148] The server transmits the generated prompts to a generative artificial intelligence model (such as OpenAI GPT). The server waits for the model to return and, based on its algorithm, parses the returned text into a structured recommended menu, a list of required ingredients, and detailed cooking instructions. The input is the natural language prompts, and the output is the recommended recipe and its component structure data generated by the model's inference.

[0149] Step 6:

[0150] The terminal receives recipes, ingredient lists, and related content from the server and displays them on the user interface. If the system integrates an emotion recognition device, the terminal collects sensory data such as the user's facial expressions and voice while the user browses or operates the system. The input consists of the menu output by the generative AI and the emotion data collected by the terminal; the output consists of the content displayed on the interface and the raw emotion information collected.

[0151] Step 7:

[0152] The terminal uploads the collected user emotion-related data to the server via API. The server then calls the emotion recognition engine to model and analyze the emotion data, inferring the user's state (such as stress, happiness, fatigue, etc.). The input is raw emotion data such as video / audio / behavioral data, and the output is structured emotion state information.

[0153] Step 8:

[0154] Based on the user's latest emotional state, combined with user profile and inventory data, the server automatically regenerates new prompts that better match the current mood, and then calls the generative AI model again to obtain optimized recommendations. If the user is under stress, the server prefers to recommend light meals and relaxing menus; if the user is happy, the server recommends party-themed or innovative products. The inputs are profile, inventory, and emotional state; the processing is emotionally appropriate prompts; and the output is the updated AI menu recommendations.

[0155] Step 9:

[0156] The server automatically analyzes the expiration dates and inventory levels of store products, prioritizes generating new menu prompts that include near-expiration or surplus items, and uses a generative artificial intelligence model to obtain effective recommended recipes, thus optimizing resource consumption. The input is current inventory and sales information, and the output is recipe recommendations centered around specific products.

[0157] Step 10:

[0158] Based on the user's past purchasing or browsing behavior, individual preferences, and real-time emotional state, the server automatically identifies supplementary products (such as beverages and desserts) related to the main recommendations, generates recommendation text, and pushes it to the terminal. Inputs include historical records, preferences, sentiment, and other multi-dimensional data; output is supplementary product push information.

[0159] Step 11:

[0160] The terminal aggregates all newly recommended menus, ingredients, products, and shopping lists, presenting them uniformly to the user. It also collects user feedback on each recommendation (such as likes, adding to shopping lists, and comments). The input consists of server-calculated and feedback data, while the output is the user's actual selections and interaction data.

[0161] Step 12:

[0162] The terminal transmits the user's selections and feedback back to the server. The server records the user's new choices and evaluations, serving as the data foundation for subsequent personalized recommendations, system optimization, and model retraining. The input is the terminal user's feedback data, and the output is the updated behavior and label records in the database.

[0163] Alternatively, an emotion engine for inferring user emotions can be combined. That is, the specific processing unit 290 can also use the emotion-specific model 59 to infer user emotions and perform specific processing using user emotions.

[0164] Example 2

[0165] The flow of a specific process in Example 2 will be described. Each part of the system described below is implemented by the data processing device 12 and the smart device 14. The data processing device 12 will be referred to as the "server," and the smart device 14 as the "terminal."

[0166] With the development of a consumer society, food waste has become a significant social problem, while consumers' demand for personalized shopping experiences and healthy eating continues to increase. Existing systems struggle to combine diverse user needs, real-time emotional states, and actual store inventory to achieve intelligent recommendations, promote rational consumption, effectively reduce food waste, and flexibly promote new or slow-moving products. Therefore, there is an urgent need for a system that can integrate user attribute information, real-time inventory information, and emotional data, utilizing generative artificial intelligence models to achieve personalized, dynamic, and intelligent recommendations and boost sales.

[0167] The specific processing performed by the specific processing unit 290 of the data processing apparatus 12 in Embodiment 2 is achieved by the following means.

[0168] In this invention, the server includes: a device for inputting user attribute data; a device for acquiring business information; a device for inputting user attribute data and business information into a generative artificial intelligence model to generate recipe instruction information; a device for displaying the recipe instruction information to the user through a terminal; a device for guiding users to desired consumer goods and shopping assistance information; a device for acquiring and analyzing the user's emotional state and adjusting the recipe based on the emotional state; a device for recommending additional consumer goods; and a device for acquiring and pushing information on new or key products. This allows for intelligent, timely, and accurate product and recipe recommendations tailored to users' personalized needs, emotional changes, and store inventory dynamics, thereby improving user experience, effectively promoting the rational circulation and consumption of food, reducing waste, and supporting flexible marketing of new products.

[0169] "User attribute data" refers to data that includes basic information related to users, such as user identity, family structure, dietary preferences, budget, health status, and cooking experience.

[0170] "Business information" refers to comprehensive data information in a store regarding the inventory, price, expiration date, promotions, and new products of goods.

[0171] "Generative artificial intelligence models" refer to artificial intelligence algorithms and systems that can automatically generate text content or solutions based on input prompts and conditions.

[0172] "Prompt statements" refer to information descriptions that contain specific tasks and requirements input to generative artificial intelligence models.

[0173] "Recipe instruction information" refers to recipe content information generated based on comprehensive data, which includes the names of side dishes, ingredients, preparation steps, etc.

[0174] "Terminal" refers to intelligent devices that allow users to operate, receive, and display system information, including smartphones, tablets, etc.

[0175] "Consumer goods" refers to various foods, beverages, or daily necessities that are available for purchase, consumption, or use by users.

[0176] "Emotional state information" refers to information about a user's current psychological or emotional state obtained by analyzing data such as facial expressions or voice.

[0177] "Additional consumer goods" refer to beverages, food, or hobbies that are recommended as an additional item outside of the main shopping list, based on the user's history and preferences.

[0178] "New or key products" refers to products that have been newly launched in the store or that the company is focusing on promoting.

[0179] To implement this invention, the system consists of core hardware such as servers, terminals, and databases, as well as various software modules. The server can run on a general-purpose computing platform (such as a cloud server or physical server), and it is recommended to install an operating system such as Linux or Windows Server. The terminal can be a user's smartphone, tablet, or other device with dedicated application software installed. The database uses a relational database system, such as MySQL or PostgreSQL, to store user information, business information, recipe content, etc. The server needs to integrate generative artificial intelligence model APIs, such as OpenAI GPT-4, ChatGPT, and Baidu Wenxin Yiyan, to handle recipe generation, text understanding, and content reasoning. Furthermore, the server or terminal also needs to integrate an emotion recognition module, which can call existing open-source emotion analysis libraries (such as OpenFace) or cloud-based emotion APIs (such as Microsoft Azure Emotion Analysis and Baidu Voice Emotion Analysis).

[0180] The server is responsible for receiving and parsing user attribute data from the terminal, and automatically retrieving shop's business information, including inventory, price, expiration date, and promotional information, via API on a regular schedule. Through an integrated data processing module, the server structures and validates the acquired data, and dynamically generates prompts based on user and shop information. It then combines necessary user preferences and inventory status, sending the results to a generative AI model to obtain personalized recipe instructions. The server writes the recipe content, required consumables, and shopping assistance information into the database and feeds it back to the terminal.

[0181] The terminal displays various information pushed by the server, including new recipes, ingredients, cooking methods, shelf locations of needed products, recommendations for near-expiry ingredients, and additional consumer goods. The terminal also guides users in filling out personal profiles, uploading emotional data collected through facial expressions or voice recordings, and promptly transmits this information to the server. The terminal synchronizes data with the server in real time for each user action, such as confirming addition to a shopping list or selecting new product recommendations.

[0182] The server uses an emotion recognition module to analyze the user's current psychological state in real time. For example, if "stress" is detected, the server will regenerate and adjust the recipe instructions, prioritizing recipes that soothe emotions; if "happiness" is detected, it will recommend energetic and delicious meals. The server also pushes appropriate add-on products (such as drinks and snacks) or new and limited-time products that merchants are promoting, based on the user's purchase history and preference data, to comprehensively improve the user experience and store sales.

[0183] Here's a specific example: User A logs into the application via their mobile phone, filling in "likes Japanese food, doesn't eat eggs, family of three, daily budget 200 yuan." The terminal uploads this information to the server. The server simultaneously retrieves the shop's current inventory of "salmon, cabbage, tofu" and automatically generates the following prompt: "Based on the user's preferences (likes Japanese food, doesn't eat eggs) and the shop's inventory (salmon, cabbage, tofu), please generate three Japanese recipes suitable for dinner." The server sends this prompt to a generative artificial intelligence model (such as GPT-4), receives the returned recipes, and stores them in the database. The user smiles at the screen with their phone; the terminal captures the expression and uploads it to the server. The server's sentiment analysis module identifies "high stress" and generates a new prompt: "For users experiencing high stress, please recommend Japanese recipes to soothe nerves." After obtaining information on new seasonings, the server can generate something like: "Combined with the new seasoning 'miso,' please generate a recommended Japanese main dish for the family."

[0184] Typical prompt statements are shown below:

[0185] "Please generate a suitable recipe based on the user profile (e.g., likes Japanese food, allergic to eggs) and the store's current inventory (e.g., has salmon, cabbage, tofu)."

[0186] "For users currently experiencing high levels of stress, we recommend a Japanese recipe that promotes relaxation."

[0187] "Please use soon-to-expire items (such as tofu and salmon) to create new family recipes and reduce waste."

[0188] "Integrate new products from food manufacturers (such as a new seasoning from a certain brand) to create a Japanese-style family recipe."

[0189] This invention focuses on server-side data fusion and computation to achieve fully automated, dynamically updated, and highly personalized recipe and product recommendations, significantly improving the consumer experience and product circulation efficiency.

[0190] use Figure 13 The processing flow is explained.

[0191] Step 1:

[0192] Users launch the application on their devices and enter basic personal information, including family structure, dietary preferences, budget, health status, and cooking experience. The input consists of manually entered text and options. The device validates the user's input, and after confirming all required fields are complete, sends it to the server in encrypted form. The output is a data packet containing the user's attribute data.

[0193] Step 2:

[0194] The server receives and parses user attribute data packets from the terminal, verifying data integrity and validity. The server then calls a database interface (such as MySQL) to write the user attribute data into the user information table. The input is the data packet uploaded by the terminal; after data parsing and correction, the output is the newly added or updated user records in the database.

[0195] Step 3:

[0196] The server periodically calls the store management system via API to automatically retrieve the latest inventory, pricing, expiration dates, promotions, and new product information. The input is the API request; the server parses the API response, performs structured processing and validity checks on the product data, and the output is an updated business information database table.

[0197] Step 4:

[0198] The server combines user attribute data and business information from the database, automatically concatenating prompts to synthesize user needs with information such as inventory and pricing into a generated instruction. The server uses this prompt as text input and calls the API of a generative artificial intelligence model (such as GPT-4). The input is the processed prompt based on user attributes and business information. The server receives the returned recipe text, stores it in a structured format (e.g., dish name, ingredients, steps), and writes it to the recipe database. The output is the generated recipe instruction information.

[0199] Step 5:

[0200] The terminal receives recipe information pushed by the server and proactively notifies the user of new recipe recommendations via pop-up windows or push notifications. When the user clicks to view the recipe on the terminal, it displays detailed recipe content, required consumables, corresponding store shelf locations, and shopping assistance information. The input is the recipe information pushed by the server, and the terminal can generate a personalized shopping list based on the user's selections. The output is a visual display of recipe and consumable information for the user.

[0201] Step 6:

[0202] The terminal collects facial expression or voice data based on user authorization, generating raw emotion data. The terminal then uploads this emotion data to the server. Upon receiving the data, the server invokes an emotion analysis module (such as the Azure Emotion API or OpenFace) to parse and identify the user's emotional state (e.g., stress, happiness). The input is raw facial or voice data; the server processes the data and performs emotion recognition, outputting the user's current emotional state.

[0203] Step 7:

[0204] The server automatically adjusts its recipe recommendation logic based on the user's emotional state, regenerates prompts that match the user's mood, and then generates new recipes again using a generative artificial intelligence model. The inputs are the user's emotional state, existing inventory, and user profile. The server automatically generates new prompts and calls the AI ​​to generate recipes. The output is adjusted recipe information that matches the current emotional state.

[0205] Step 8:

[0206] The server analyzes trade information, proactively identifies near-expiry or slow-moving inventory, and prioritizes these items in notification prompts. The server uses a generative artificial intelligence model to generate new recipes that prioritize the use of near-expiry inventory and pushes the results to the terminal. The input is inventory data, which is filtered and processed through a database; the output is recipe recommendations that prioritize near-expiry items.

[0207] Step 9:

[0208] Based on the user's shopping history and preferences, combined with their current emotional state, the server automatically filters add-on consumer goods such as beverages and snacks, and generates recommendations. The server pushes add-to-cart suggestions to the user's device, allowing them to add these items to their shopping list with a single click. Inputs include shopping history, preferences, and emotional data; the server performs data mining and recommendation logic calculations, outputting personalized add-on consumer goods recommendations.

[0209] Step 10:

[0210] The server retrieves key product information from stores, such as new arrivals and promotional items, and writes it into prompts. It then uses a generative AI model to generate recipe recommendations that include these new products. The server pushes relevant information to users who meet certain criteria, and the terminal displays details, allowing users to view the information and add it to their shopping cart. The input includes new product information and user preferences; the system uses reasoning to generate recipes related to the new products; and the output is a recipe containing the new products along with recommendations.

[0211] Application Example 2

[0212] The process flow corresponding to the specific processing in Use Case 2 will be described below. The various parts of the system described below are implemented by the data processing device 12 and the intelligent device 14. In addition, the data processing device 12 is referred to as the "server" and the intelligent device 14 is referred to as the "terminal".

[0213] Existing recipe recommendation systems primarily generate recommendations based on static information such as users' dietary preferences and health status. This makes it difficult to dynamically reflect changes in users' emotions and real-time inventory and expiration dates at sales points, resulting in limited user satisfaction and hindering efforts to effectively reduce food waste and boost sales. Furthermore, existing systems struggle to promptly recommend products nearing their expiration date or with excess inventory, lack the ability to adjust recommendations based on users' emotional states, and cannot achieve personalized, refined recommendations based on generative artificial intelligence models.

[0214] The specific processing performed by the specific processing unit 290 of the data processing apparatus 12 in Application Example 2 is achieved by the following means.

[0215] In this invention, the server includes: an information acquisition device for inputting user information; an information acquisition device for acquiring point-of-sale information; an information processing device for generating and sending prompts to a generative artificial intelligence model and obtaining cooking plan information; an information output device for outputting cooking plan information to the user; an information providing device for indicating the required raw materials; an emotion acquisition device for detecting and acquiring the user's emotional state; an information processing device for adjusting the cooking plan based on the emotional state; and an information processing device for prioritizing the use of items with near expiration dates and recommending supplementary products based on purchase history, preferences, and emotions. This enables the dynamic acquisition and fusion of multi-source information such as user, point-of-sale, and emotional information, and, with the help of a generative artificial intelligence model, achieves personalized, real-time, and emotion-aware recipe and product recommendations, improving user satisfaction, reducing food waste, and increasing the sales efficiency of points of sale.

[0216] "User information" refers to various data input by users that are related to recipe recommendations, including personal basic information, family members, dietary preferences, budget, cooking experience, health status, etc.

[0217] "Point of sale information" refers to real-time sales-related data such as product inventory, prices, expiration dates, and promotional items at sales locations.

[0218] "Prompt statements" refer to instructions or contextual descriptions that have been written through information fusion and are used as input to generative artificial intelligence models to guide the model to output desired format and content.

[0219] "Generative AI models" refer to AI model tools that can automatically generate content (such as recipes) by inputting prompts, including but not limited to natural language generation models based on deep learning.

[0220] "Cooking plan information" refers to recipe content output by a generative artificial intelligence model based on user information and point-of-sale information, including preparation steps and required raw materials.

[0221] "Information acquisition device" refers to a data input terminal that realizes data input and collection, including but not limited to mobile device applications, web forms, etc.

[0222] "Information processing device" refers to a data processing unit that can process and analyze diverse types of data and provide output results, such as a server or cloud computing platform.

[0223] "Information output device" refers to the interface or device that presents the processing results to the user, including smartphone displays, computer display terminals, etc.

[0224] "Information providing device" refers to a functional module used to output auxiliary information such as lists and navigation related to the required raw materials to users.

[0225] "Emotion acquisition device" refers to a functional module or tool that collects and determines the user's emotional state through facial expressions, voice, and other means.

[0226] "Emotional state" refers to the category of a user's current psychological feeling detected and determined by the emotion acquisition device, including but not limited to happiness, stress, sadness, fatigue, etc.

[0227] "Add-on products" refer to beverages, snacks, and other items recommended based on user preferences, shopping history, and emotional state, in addition to the main recipe ingredients.

[0228] "Raw material information" refers to the information on the various ingredients required in the cooking plan and their corresponding sales point locations.

[0229] This invention relates to a system and its implementation method for personalized recipe and product recommendations based on user information, point-of-sale information, and emotional state using a generative artificial intelligence model. The following details the implementation of this invention with examples of hardware and software configuration and practical operation.

[0230] The system consists of a server, a terminal (such as a smartphone), a database, and an artificial intelligence model. The server can be a general-purpose cloud server, such as a general-purpose enterprise-grade server or a cloud computing platform; the database can be a relational database, such as MySQL or PostgreSQL; the terminal is a data input / output device capable of running applications, such as an Android phone or an iPhone; and the artificial intelligence model can adopt existing mainstream generative artificial intelligence models, such as large-scale language models or natural language generation models.

[0231] In practice, users install and launch a dedicated application (App) on their devices. Users enter basic information such as family members, dietary preferences, budget, cooking experience, and health status into the App. The device guides the user through the information entry process via a graphical interface and uploads it to the server via a standard API interface. Upon receiving the information, the server stores it in its database to generate a user profile.

[0232] The server periodically retrieves real-time data on inventory, prices, expiration dates, and promotional activities from sales points (such as supermarkets and convenience stores) via an API that interfaces with the sales system, and stores this data in its database. The terminal has built-in camera and microphone modules, allowing users to upload data to the server by capturing facial expressions or recording voice messages. The server then calls sentiment analysis engine software (such as a deep learning-based sentiment recognition API or cloud services) to analyze and determine the user's current mood (e.g., happiness, stress).

[0233] When recipe recommendations are needed, the server automatically organizes existing user and sales point data, and combines this with the analyzed user sentiment state to generate prompts for the generative AI model, such as:

[0234] User profile:

[0235] I like Japanese food.

[0236] Egg allergy

[0237] I've been under a lot of pressure lately.

[0238] Point-of-sale inventory:

[0239] Fresh mackerel

[0240] Cabbage is plentiful

[0241] The tofu is almost gone.

[0242] Please generate a simple Japanese home-style recipe that includes stress-relieving ingredients. Exclude eggs and prioritize using readily available or soon-to-expire ingredients.

[0243] The prompt is input into a generative artificial intelligence model (such as a mainstream large-scale language model), which outputs the recipe. The server parses the recipe and displays it to the user via a mobile app. Simultaneously, the app automatically generates a shopping list and navigation information, displaying the specific location of each ingredient at the sales point, based on the recipe requirements and the sales location layout, to assist the user's shopping.

[0244] In addition, the server can prioritize recommending ingredients that are about to expire or have excess inventory based on information such as the expiration date and inventory of products at sales points in the database, and automatically generate corresponding recipes. The server can also recommend supplementary products (such as beverages and snacks) based on users' past purchase records, preferences, and sentiment data, achieving a more personalized and practical recommendation effect.

[0245] This system leverages the server's efficient data processing capabilities, the human-computer interaction advantages of smart terminals, and the powerful text and content generation capabilities of generative artificial intelligence models to dynamically, personally, and in real-time respond to user needs, enhance the shopping and dining experience, reduce food waste at sales points, and promote product sales.

[0246] The technical features of this invention are not limited to the specific hardware, software or data formats described above. Any system, means and method with the same function and basic implementation concept should be included within the scope of protection of this invention.

[0247] use Figure 14 The processing flow is explained.

[0248] Step 1:

[0249] Users open the application on their devices and enter a personal information entry interface, where they fill in user information such as family members, dietary preferences, budget, cooking experience, and health status. After the device checks the format of the input, it uploads the data to the server via an API interface.

[0250] Input: Basic information entered by the user on the terminal.

[0251] The terminal's data acquisition module processes the raw user input data into a structured format and sends it to the server.

[0252] Output: Structured user information data.

[0253] Step 2:

[0254] The server receives user information data from the terminal, parses and verifies the data, generates and updates user profiles, and stores them in the database.

[0255] Input: Structured user information data.

[0256] The server performs data parsing, verification, and archiving operations.

[0257] Output: User profiles stored in the database.

[0258] Step 3:

[0259] The server periodically retrieves point-of-sale information, including inventory, price, shelf life, and promotional items, through the point-of-sale system API. The retrieved information is then standardized and formatted before being saved to the database.

[0260] Input: Raw inventory information returned by the point of sale API.

[0261] The server parses the API response and cleans and structures data such as inventory and price.

[0262] Output: Standardized point-of-sale information within the database.

[0263] Step 4:

[0264] The terminal captures the user's facial expressions or voice through a camera or microphone, and the user can choose to take a photo or input voice to reflect their current emotions.

[0265] Input: Facial expressions or voice data collected by the user through the terminal.

[0266] The terminal uploads this emotional data to the server.

[0267] Output: Raw audio and video emotion data uploaded to the server.

[0268] Step 5:

[0269] The server calls emotion recognition software to analyze the received facial expressions or voice data, identify the user's current emotional state (such as stress, happiness, fatigue, etc.), and write the emotion tag into the user's profile.

[0270] Input: Raw audio and video emotion data.

[0271] The server executes emotion recognition algorithms and model discrimination to derive emotion labels.

[0272] Output: Emotional labels (e.g., "stressed") are stored in the user profile.

[0273] Step 6:

[0274] The server integrates current user profile data, point-of-sale information, and sentiment tags to automatically generate prompts for the generative AI model, and then inputs these prompts into the generative AI model.

[0275] Input: User profile, point-of-sale information, emotion tags.

[0276] The server integrates, combines, and processes information from multiple sources to output standardized prompts.

[0277] Output: The generated prompts are used as input to the AI ​​model.

[0278] Step 7:

[0279] The server inputs the prompts into a generative artificial intelligence model (such as a large language model), which then generates a personalized recipe, a list of required ingredients, and simple preparation steps.

[0280] Input: Prompt statement.

[0281] Generative artificial intelligence models process and calculate based on the content of sentences to generate complete recipe content.

[0282] Output: AI-generated recipe text and raw material data.

[0283] Step 8:

[0284] The server pushes the generated recipe and ingredient list to the terminal. Based on the recipe requirements and sales point layout, the terminal generates a shopping list and navigation path to display to the user.

[0285] Input: Recipe and raw material data pushed by the server.

[0286] The terminal calls local or cloud map services to match the location of shelves within the sales point, generating navigation instructions and shopping lists.

[0287] Output: Recipe details, shopping list, and navigation information displayed on the terminal.

[0288] Step 9:

[0289] The server analyzes point-of-sale inventory data, identifies products nearing their expiration date or with excess inventory, generates new prompts, calls an artificial intelligence model to output additional recipes that prioritize the consumption of these products, and pushes them to the terminal for add-to-cart recommendations.

[0290] Input: Point-of-sale inventory and shelf-life data.

[0291] The server data processing module filters out key products and reorganizes the prompts.

[0292] Output: Additional recipes with add-on suggestions are pushed to the terminal.

[0293] Step 10:

[0294] Based on the user's past purchase history, preferences, and emotional state, the server further recommends additional products such as beverages and snacks, generating a recommendation list that is pushed to the terminal, where users can choose to add them to their shopping list.

[0295] Input: User purchase history, preferences, and sentiment data.

[0296] The server analyzes the data and generates multi-level recommended candidates.

[0297] Output: An additional product recommendation list is displayed on the terminal, and users can add them to their shopping list with one click.

[0298] The specific processing unit 290 sends the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires sound representing user input regarding the result of the specific processing. The control unit 46A sends the sound data representing user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the sound data.

[0299] Data generation model 58 is a so-called generative AI (Artificial Intelligence). Examples of data generation models 58 include ChatGPT (registered trademark) (accessible via the internet (URL: https: / / openai.com / blog / chatgpt)). Data generation model 58 is obtained through deep learning on a neural network. Input to data generation model 58 are prompt words containing instructions, and inference data such as sound data representing sound, text data representing text, and image data representing images (e.g., still image data or animation data). Data generation model 58 infers from the input inference data based on the instructions represented by the prompt words and outputs the inference result in one or more data forms, such as sound data, text data, and image data. Data generation model 58 includes, for example, text generation AI, image generation AI, and multimodal generation AI. Here, inference refers to, for example, analysis, classification, prediction, and / or induction. The specific processing unit 290 performs the aforementioned specific processing while using data generation model 58. The data generation model 58 can also be a model finely tuned to output inference results from prompts that do not contain instructions. In this case, the data generation model 58 can output inference results based on prompts that do not contain instructions. The data processing apparatus 12, etc., includes various data generation models 58, including AI other than the generation AI. AI other than the generation AI can be, for example, linear regression, logistic regression, decision trees, random forests, support vector machines (SVM), k-means clustering, convolutional neural networks (CNN), recurrent neural networks (RNN), generative adversarial networks (GAN), or Naive Bayes, and can perform various processes, but is not limited to this example. Furthermore, the AI ​​can also be an AI agent. Furthermore, when the processing of the above-mentioned parts is performed by AI, the processing can be partially or entirely performed by AI, but is not limited to this example. Furthermore, the processing performed by the AI ​​including the generation AI can be replaced by processing in the rule base, and the processing in the rule base can also be replaced by processing performed by the AI ​​including the generation AI.

[0300] Furthermore, the processing of the aforementioned data processing system 10 is performed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the smart device 14, but it can also be performed by both the specific processing unit 290 of the data processing device 12 and the control unit 46A of the smart device 14. Additionally, the specific processing unit 290 of the data processing device 12 acquires or collects information required for processing from the smart device 14 or external devices, and the smart device 14 acquires or collects information required for processing from the data processing device 12 or external devices.

[0301] For example, the collection unit is implemented by the control unit 46A of the smart device 14 or the specific processing unit 290 of the data processing device 12. For example, the acquisition unit uses the camera 42 or communication I / F 44 of the smart device 14 to acquire step data, which is then processed by the specific processing unit 290 of the data processing device 12. For example, the analysis unit is implemented by the specific processing unit 290 of the data processing device 12, which analyzes the data from the collection unit and the acquisition unit. For example, the generation unit is implemented by the specific processing unit 290 of the data processing device 12, which uses a generation AI to generate a menu. For example, the serving unit is implemented by the output device 40 of the smart device 14 or the specific processing unit 290 of the data processing device 12, which provides the generated menu to the user. The correspondence between each unit and the device or control unit is not limited to the above examples and various changes can be made.

[0302] In the above embodiments, examples of specific processing by the data processing device 12 are given, but the technology disclosed herein is not limited to this, and specific processing may also be performed by the smart device 14.

[0303] Second Implementation Method

[0304] Figure 3 An example of the configuration of the data processing system 210 according to the second embodiment is shown.

[0305] like Figure 3 As shown, the data processing system 210 includes a data processing device 12 and smart glasses 214. A server can be cited as an example of the data processing device 12.

[0306] The data processing apparatus 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" as understood in this disclosure. The computer 22 includes a processor 28, RAM 30, and memory 32. The processor 28, RAM 30, and memory 32 are connected to a bus 34. Furthermore, the database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0307] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, RAM 48, and memory 50. The processor 46, RAM 48, and memory 50 are connected to a bus 52. Furthermore, the microphone 238, speaker 240, camera 42, and communication I / F 44 are also connected to the bus 52.

[0308] Microphone 238 receives instructions from user 20 by receiving sounds emitted by user 20. Microphone 238 captures sounds emitted by user 20 and converts the captured sounds into sound data, which is then output to processor 46. Speaker 240 outputs sound according to instructions from processor 46.

[0309] Camera 42 is a small digital camera equipped with an optical system such as a lens, aperture and shutter, and imaging elements such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, to capture images of the user 20's surroundings (e.g., the field of view defined by an angle equivalent to the field of vision of an average healthy person).

[0310] Communication I / F44 is connected to network 54. Communication I / F44 and 26 are responsible for the transmission and reception of various information between processor 46 and processor 28 via network 54. The transmission and reception of various information between processor 46 and processor 28 using communication I / F44 and 26 is performed in a secure state.

[0311] Figure 4 This illustrates an example of the main functions of the data processing device 12 and the smart glasses 214. For example... Figure 4 As shown, in the data processing device 12, specific processing is performed by the processor 28. The specific processing program 56 is stored in the memory 32.

[0312] The specific processing program 56 is an example of a "program" involved in the technology of this disclosure. The processor 28 reads the specific processing program 56 from the memory 32 and executes the read specific processing program 56 on the RAM 30. Specific processing is implemented by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0313] The memory 32 stores a data generation model 58 and an emotion-specific model 59. The data generation model 58 and the emotion-specific model 59 are used by the specific processing unit 290. The specific processing unit 290 can use the emotion-specific model 59 to infer the user's emotions and perform specific processing based on the user's emotions. In the emotion inference function (emotion-specific function) using the emotion-specific model 59, various inferences and predictions related to the user's emotions are performed, including inferences and predictions of the user's emotions, but this is not limited to this example. Furthermore, emotion inference and prediction may also include, for example, emotion analysis (parsing).

[0314] In the smart glasses 214, the processor 46 performs reception and output processing. The memory 50 stores the reception and output program 60. The processor 46 reads the reception and output program 60 from the memory 50 and executes the read reception and output program 60 on the RAM 48. The reception and output processing is implemented by the processor 46 operating as a control unit 46A according to the reception and output program 60 executed on the RAM 48. Furthermore, the smart glasses 214 has the same data generation model and emotion-specific model as the data generation model 58 and the emotion-specific model 59, and these models can also be used to perform the same processing as the specific processing unit 290.

[0315] Next, the specific processing of the specific processing unit 290 of the data processing device 12 will be described. Each part of the system described below is implemented by the data processing device 12 and the smart glasses 214. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0316] Example 1

[0317] The process is the same as that of the specific process described in Embodiment 1 in the first embodiment above, so the description is omitted.

[0318] Application Example 1

[0319] The process is the same as that in the specific processing described in Application Example 1 of the first embodiment above, so the description is omitted.

[0320] Example 2

[0321] The process is the same as that of the specific process in Embodiment 2 described in the first embodiment above, so the description is omitted.

[0322] Application Example 2

[0323] The process is the same as that in the specific processing described in Application Example 2 of the first embodiment above, so the description is omitted.

[0324] The specific processing unit 290 sends the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A outputs the result of the specific processing to the speaker 240. The microphone 238 acquires sound input representing the user's input regarding the result of the specific processing. The control unit 46A sends the sound data representing the user's input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the sound data.

[0325] Data generation model 58 is a so-called generative AI (Artificial Intelligence). Examples of data generation models 58 include ChatGPT (registered trademark) (accessible via the internet (URL: https: / / openai.com / blog / chatgpt)). Data generation model 58 is obtained through deep learning on a neural network. Input to data generation model 58 are prompt words containing instructions, and inference data such as sound data representing sound, text data representing text, and image data representing images (e.g., still image data or animation data). Data generation model 58 infers from the input inference data based on the instructions represented by the prompt words and outputs the inference result in one or more data forms, such as sound data, text data, and image data. Data generation model 58 includes, for example, text generation AI, image generation AI, and multimodal generation AI. Here, inference refers to, for example, analysis, classification, prediction, and / or induction. The specific processing unit 290 performs the aforementioned specific processing while using data generation model 58. The data generation model 58 can also be a model finely tuned to output inference results from prompts that do not contain instructions. In this case, the data generation model 58 can output inference results based on prompts that do not contain instructions. The data processing apparatus 12, etc., includes various data generation models 58, including AI other than the generation AI. AI other than the generation AI can be, for example, linear regression, logistic regression, decision trees, random forests, support vector machines (SVM), k-means clustering, convolutional neural networks (CNN), recurrent neural networks (RNN), generative adversarial networks (GAN), or Naive Bayes, and can perform various processes, but is not limited to this example. Furthermore, the AI ​​can also be an AI agent. Furthermore, when the processing of the above-mentioned parts is performed by AI, the processing can be partially or entirely performed by AI, but is not limited to this example. Furthermore, the processing performed by the AI ​​including the generation AI can be replaced by processing in the rule base, and the processing in the rule base can also be replaced by processing performed by the AI ​​including the generation AI.

[0326] Furthermore, the processing of the aforementioned data processing system 10 is performed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the smart glasses 214, but it can also be performed by both the specific processing unit 290 of the data processing device 12 and the control unit 46A of the smart glasses 214. Additionally, the specific processing unit 290 of the data processing device 12 acquires or collects information required for processing from the smart glasses 214 or external devices, and the smart glasses 214 acquires or collects information required for processing from the data processing device 12 or external devices.

[0327] For example, the collection unit is implemented by the control unit 46A of the smart glasses 214 or the specific processing unit 290 of the data processing device 12. For example, the acquisition unit uses the camera 42 or communication I / F 44 of the smart glasses 214 to acquire step data, which is then processed by the specific processing unit 290 of the data processing device 12. For example, the analysis unit is implemented by the specific processing unit 290 of the data processing device 12, which analyzes the data from the collection unit and the acquisition unit. For example, the generation unit is implemented by the specific processing unit 290 of the data processing device 12, which uses a generation AI to generate a menu. For example, the serving unit is implemented by the speaker 240 of the smart glasses 214 or the specific processing unit 290 of the data processing device 12, which provides the generated menu to the user. The correspondence between each unit and the device or control unit is not limited to the above examples and various changes can be made.

[0328] In the above embodiments, examples of specific processing by the data processing device 12 are given, but the technology disclosed herein is not limited to this, and specific processing may also be performed by the smart glasses 214.

[0329] Third Implementation Method

[0330] Figure 5 An example of the configuration of the data processing system 310 according to the third embodiment is shown.

[0331] like Figure 5 As shown, the data processing system 310 includes a data processing device 12 and a head-mounted terminal 314. A server can be cited as an example of the data processing device 12.

[0332] The data processing apparatus 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" as understood in this disclosure. The computer 22 includes a processor 28, RAM 30, and memory 32. The processor 28, RAM 30, and memory 32 are connected to a bus 34. Furthermore, the database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0333] The head-mounted terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and memory 50. The processor 46, RAM 48, and memory 50 are connected to a bus 52. Furthermore, the microphone 238, speaker 240, camera 42, display 343, and communication I / F 44 are also connected to the bus 52.

[0334] Microphone 238 receives instructions from user 20 by receiving sounds emitted by user 20. Microphone 238 captures sounds emitted by user 20 and converts the captured sounds into sound data, which is then output to processor 46. Speaker 240 outputs sound according to instructions from processor 46.

[0335] Camera 42 is a small digital camera equipped with an optical system such as a lens, aperture and shutter, and imaging elements such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, to capture images of the user 20's surroundings (e.g., the field of view defined by an angle equivalent to the field of vision of an average healthy person).

[0336] Communication I / F44 is connected to network 54. Communication I / F44 and 26 are responsible for the transmission and reception of various information between processor 46 and processor 28 via network 54. The transmission and reception of various information between processor 46 and processor 28 using communication I / F44 and 26 is performed in a secure state.

[0337] Figure 6 This illustrates an example of the main functions of the data processing device 12 and the head-mounted terminal 314. For example... Figure 6 As shown, in the data processing device 12, specific processing is performed by the processor 28. The specific processing program 56 is stored in the memory 32.

[0338] The specific processing program 56 is an example of a "program" involved in the technology of this disclosure. The processor 28 reads the specific processing program 56 from the memory 32 and executes the read specific processing program 56 on the RAM 30. Specific processing is implemented by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0339] The memory 32 stores the data generation model 58 and the emotion-specific model 59. The data generation model 58 and the emotion-specific model 59 are used by the specific processing unit 290.

[0340] In the head-mounted terminal 314, the processor 46 performs the acceptance / output processing. The memory 50 stores the acceptance / output program 60. The processor 46 reads the acceptance / output program 60 from the memory 50 and executes the read acceptance / output program 60 on the RAM 48. The acceptance / output processing is implemented by the processor 46 operating as a control unit 46A according to the acceptance / output program 60 executed on the RAM 48.

[0341] Next, the specific processing of the specific processing unit 290 of the data processing device 12 will be described. Each part of the system described below is implemented by the data processing device 12 and the head-mounted terminal 314. In the following description, the data processing device 12 will be referred to as the "server" and the head-mounted terminal 314 will be referred to as the "terminal".

[0342] Example 1

[0343] The process is the same as that of the specific process described in Embodiment 1 in the first embodiment above, so the description is omitted.

[0344] Application Example 1

[0345] The process is the same as that in the specific processing described in Application Example 1 of the first embodiment above, so the description is omitted.

[0346] Example 2

[0347] The process is the same as that of the specific process in Embodiment 2 described in the first embodiment above, so the description is omitted.

[0348] Application Example 2

[0349] The process is the same as that in the specific processing described in Application Example 2 of the first embodiment above, so the description is omitted.

[0350] The specific processing unit 290 sends the result of the specific processing to the head-mounted terminal 314. In the head-mounted terminal 314, the control unit 46A outputs the result of the specific processing to the speaker 240 and the display 343. The microphone 238 acquires sound input representing the user's input regarding the result of the specific processing. The control unit 46A sends the sound data representing the user's input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the sound data.

[0351] Data generation model 58 is a so-called generative AI (Artificial Intelligence). Examples of data generation models 58 include ChatGPT (registered trademark) (accessible via the internet (URL: https: / / openai.com / blog / chatgpt)). Data generation model 58 is obtained through deep learning on a neural network. Input to data generation model 58 are prompt words containing instructions, and inference data such as sound data representing sound, text data representing text, and image data representing images (e.g., still image data or animation data). Data generation model 58 infers the input inference data based on the instructions represented by the prompt words and outputs the inference result in one or more data forms such as sound data, text data, and image data. Data generation model 58 includes, for example, text generation AI, image generation AI, and multimodal generation AI. Here, inference refers to, for example, analysis, classification, prediction, and / or induction. The specific processing unit 290 performs the aforementioned specific processing while using data generation model 58. The data generation model 58 can also be a model finely tuned to output inference results from prompts that do not contain instructions. In this case, the data generation model 58 can output inference results based on prompts that do not contain instructions. The data processing apparatus 12, etc., includes various data generation models 58, including AI other than the generation AI. AI other than the generation AI can be, for example, linear regression, logistic regression, decision trees, random forests, support vector machines (SVM), k-means clustering, convolutional neural networks (CNN), recurrent neural networks (RNN), generative adversarial networks (GAN), or Naive Bayes, and can perform various processes, but is not limited to this example. Furthermore, the AI ​​can also be an AI agent. Furthermore, when the processing of the above-mentioned parts is performed by AI, the processing can be partially or entirely performed by AI, but is not limited to this example. Furthermore, the processing performed by the AI ​​including the generation AI can be replaced by processing in the rule base, and the processing in the rule base can also be replaced by processing performed by the AI ​​including the generation AI.

[0352] Furthermore, the processing of the aforementioned data processing system 10 is performed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the head-mounted terminal 314, but it can also be performed by both the specific processing unit 290 of the data processing device 12 and the control unit 46A of the head-mounted terminal 314. Additionally, the specific processing unit 290 of the data processing device 12 acquires or collects information required for processing from the head-mounted terminal 314 or external devices, and the head-mounted terminal 314 acquires or collects information required for processing from the data processing device 12 or external devices.

[0353] For example, the collection unit is implemented by the control unit 46A of the head-mounted terminal 314 or the specific processing unit 290 of the data processing device 12. For example, the acquisition unit uses the camera 42 or communication I / F 44 of the head-mounted terminal 314 to acquire step data, which is then processed by the specific processing unit 290 of the data processing device 12. For example, the analysis unit is implemented by the specific processing unit 290 of the data processing device 12 to analyze the data from the collection unit and the acquisition unit. For example, the generation unit is implemented by the specific processing unit 290 of the data processing device 12 to generate a menu using a generation AI. For example, the serving unit is implemented by the speaker 240 and display 343 of the head-mounted terminal 314 or the specific processing unit 290 of the data processing device 12 to provide the generated menu to the user. The correspondence between each unit and the device or control unit is not limited to the above examples and various changes can be made.

[0354] In the above embodiments, examples of specific processing by the data processing device 12 are given, but the technology disclosed herein is not limited to this, and specific processing may also be performed by the head-mounted terminal 314.

[0355] Fourth Implementation Method

[0356] Figure 7 An example of the configuration of the data processing system 410 according to the fourth embodiment is shown.

[0357] like Figure 7 As shown, the data processing system 410 includes a data processing device 12 and a robot 414. A server can be cited as an example of the data processing device 12.

[0358] The data processing apparatus 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" as understood in this disclosure. The computer 22 includes a processor 28, RAM 30, and memory 32. The processor 28, RAM 30, and memory 32 are connected to a bus 34. Furthermore, the database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0359] Robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and memory 50. The processor 46, RAM 48, and memory 50 are connected to a bus 52. Furthermore, the microphone 238, speaker 240, camera 42, controlled object 443, and communication I / F 44 are also connected to the bus 52.

[0360] Microphone 238 receives instructions from user 20 by receiving sounds emitted by user 20. Microphone 238 captures sounds emitted by user 20 and converts the captured sounds into sound data, which is then output to processor 46. Speaker 240 outputs sound according to instructions from processor 46.

[0361] Camera 42 is a small digital camera equipped with an optical system such as a lens, aperture and shutter, and imaging elements such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, to photograph the area around robot 414 (e.g., the field of view defined by a perspective equivalent to the field of vision of an average healthy person).

[0362] Communication I / F44 is connected to network 54. Communication I / F44 and 26 are responsible for the transmission and reception of various information between processor 46 and processor 28 via network 54. The transmission and reception of various information between processor 46 and processor 28 using communication I / F44 and 26 is performed in a secure state.

[0363] The controlled object 443 includes a display device, LEDs (light-emitting diodes) for the eyes, and motors for driving the arms, hands, and feet. The posture or movement of the robot 414 is controlled by controlling the motors in the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. In addition, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0364] Figure 8 This illustrates an example of the main functions of the data processing device 12 and the robot 414. For example... Figure 8 As shown, in the data processing device 12, specific processing is performed by the processor 28. The specific processing program 56 is stored in the memory 32.

[0365] The specific processing program 56 is an example of a "program" involved in the technology of this disclosure. The processor 28 reads the specific processing program 56 from the memory 32 and executes the read specific processing program 56 on the RAM 30. Specific processing is implemented by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0366] The memory 32 stores the data generation model 58 and the emotion-specific model 59. The data generation model 58 and the emotion-specific model 59 are used by the specific processing unit 290.

[0367] In robot 414, the processor 46 performs the acceptance and output processing. The memory 50 stores the acceptance and output program 60. The processor 46 reads the acceptance and output program 60 from the memory 50 and executes the read acceptance and output program 60 on RAM 48. The acceptance and output processing is implemented by the processor 46 acting as the control unit 46A according to the acceptance and output program 60 executed on RAM 48.

[0368] Next, the specific processing of the specific processing unit 290 of the data processing device 12 will be described. Each part of the system described below is implemented by the data processing device 12 and the robot 414. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 will be referred to as the "terminal".

[0369] Example 1

[0370] The process is the same as that of the specific process described in Embodiment 1 in the first embodiment above, so the description is omitted.

[0371] Application Example 1

[0372] The process is the same as that in the specific processing described in Application Example 1 of the first embodiment above, so the description is omitted.

[0373] Example 2

[0374] The process is the same as that of the specific process in Embodiment 2 described in the first embodiment above, so the description is omitted.

[0375] Application Example 2

[0376] The process is the same as that in the specific processing described in Application Example 2 of the first embodiment above, so the description is omitted.

[0377] The specific processing unit 290 sends the result of the specific processing to the robot 414. In the robot 414, the control unit 46A outputs the result of the specific processing to the speaker 240 and the controlled object 443. The microphone 238 acquires sound input representing the result of the specific processing. The control unit 46A sends the sound data representing the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the sound data.

[0378] Data generation model 58 is a so-called generative AI (Artificial Intelligence). Examples of data generation models 58 include ChatGPT (registered trademark) (accessible via the internet (URL: https: / / openai.com / blog / chatgpt)). Data generation model 58 is obtained through deep learning on a neural network. Input to data generation model 58 are prompt words containing instructions, and inference data such as sound data representing sound, text data representing text, and image data representing images (e.g., still image data or animation data). Data generation model 58 infers from the input inference data based on the instructions represented by the prompt words and outputs the inference result in one or more data forms, such as sound data, text data, and image data. Data generation model 58 includes, for example, text generation AI, image generation AI, and multimodal generation AI. Here, inference refers to, for example, analysis, classification, prediction, and / or induction. The specific processing unit 290 performs the aforementioned specific processing while using data generation model 58. The data generation model 58 can also be a model finely tuned to output inference results from prompts that do not contain instructions. In this case, the data generation model 58 can output inference results based on prompts that do not contain instructions. The data processing apparatus 12, etc., includes various data generation models 58, including AI other than the generation AI. AI other than the generation AI can be, for example, linear regression, logistic regression, decision trees, random forests, support vector machines (SVM), k-means clustering, convolutional neural networks (CNN), recurrent neural networks (RNN), generative adversarial networks (GAN), or Naive Bayes, and can perform various processes, but is not limited to this example. Furthermore, the AI ​​can also be an AI agent. Furthermore, when the processing of the above-mentioned parts is performed by AI, the processing can be partially or entirely performed by AI, but is not limited to this example. Furthermore, the processing performed by the AI ​​including the generation AI can be replaced by processing in the rule base, and the processing in the rule base can also be replaced by processing performed by the AI ​​including the generation AI.

[0379] Furthermore, the processing of the aforementioned data processing system 10 is performed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the robot 414, but it can also be performed by both the specific processing unit 290 of the data processing device 12 and the control unit 46A of the robot 414. Additionally, the specific processing unit 290 of the data processing device 12 acquires or collects information required for processing from the robot 414 or external devices, and the robot 414 acquires or collects information required for processing from the data processing device 12 or external devices.

[0380] For example, the collection unit is implemented by the control unit 46A of the robot 414 or the specific processing unit 290 of the data processing device 12. For example, the acquisition unit uses the camera 42 or communication I / F 44 of the robot 414 to acquire step data, which is then processed by the specific processing unit 290 of the data processing device 12. For example, the analysis unit is implemented by the specific processing unit 290 of the data processing device 12, which analyzes the data from the collection unit and the acquisition unit. For example, the generation unit is implemented by the specific processing unit 290 of the data processing device 12, which uses a generation AI to generate a menu. For example, the serving unit is implemented by the speaker 240 of the robot 414 and the control object 443 or the specific processing unit 290 of the data processing device 12, which provides the generated menu to the user. The correspondence between each unit and the device or control unit is not limited to the above examples and various changes can be made.

[0381] In the above embodiments, examples of specific processing by the data processing device 12 are given, but the technology disclosed herein is not limited to this, and specific processing may also be performed by the robot 414.

[0382] Furthermore, the emotion-specific model 59, acting as an emotion engine, can determine a user's emotion based on a specific mapping. Specifically, the emotion-specific model 59 can determine a user's emotion based on an emotion graph that serves as a specific mapping (see [reference]). Figure 9 The system determines the user's emotions. Furthermore, the emotion-specific model 59 can similarly determine the robot's emotions, and the specific processing unit 290 performs specific processing based on the robot's emotions.

[0383] Figure 9 This is a diagram representing an emotion map 400 that maps multiple emotions. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotion is. On the outer side of the concentric circles, emotions representing states or behaviors arising from mood are arranged. Emotions are concepts that include feelings and mental states. Emotions generated by reactions occurring in the brain are arranged roughly to the left of the concentric circles. Emotions derived from situational judgments are arranged roughly to the right of the concentric circles. Emotions generated by reactions occurring in the brain and derived from situational judgments are arranged roughly above and below the concentric circles. Furthermore, "pleasant" emotions are arranged above the concentric circles, and "unpleasant" emotions are arranged below them. Thus, in the emotion map 400, multiple emotions are mapped based on the structure that generates emotions, and emotions that are likely to occur simultaneously are mapped close to each other.

[0384] These emotions are distributed at the three o'clock position of the emotion map 400, typically fluctuating between peace and anxiety. In the right half of the emotion map 400, situational awareness dominates over internal sensation, thus resulting in an impression of calm.

[0385] The inner side of the emotion map 400 represents the inner state, while the outer side represents behavior. Therefore, the further outward you are from the emotion map 400, the more visible the emotion becomes (manifested in behavior).

[0386] Here, human emotions are based on various balances such as posture and blood sugar levels. When these balances deviate from an ideal state, it indicates an unpleasant state; when they approach the ideal state, it indicates a pleasant state. Emotions in robots, cars, motorcycles, etc., can also be created in the following way: based on various balances such as posture and remaining battery power, when these balances deviate from an ideal state, it indicates an unpleasant state; when they approach the ideal state, it indicates a pleasant state. Emotion maps can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a Brain Physiological Signal Analysis System for Voice Emotion Recognition and Emotion, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). In the left half of the emotion map, emotions belonging to the sensory-dominated region, called "response," are arranged. Furthermore, in the right half of the emotion map, emotions belonging to the situational cognition-dominated region, called "situation," are arranged.

[0387] In the emotion map, two types of emotions that promote learning are defined. One is a negative emotion on the situational side, in the middle or peripheral region of "repentance" or "reflection." This occurs when the robot experiences negative emotions such as "I don't want to experience this feeling again" or "I don't want to be blamed again." The other is a positive emotion on the response side, near the "desire" region. This occurs when there are positive feelings such as "wanting more" or "wanting to know more."

[0388] The emotion-specific model 59 inputs user input into a pre-trained neural network to obtain emotion values ​​representing each emotion shown in the emotion map 400, thereby determining the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network... Figure 10 As shown in the sentiment graph 900, it was trained in a way that sentiments that are configured close to each other have similar values. Figure 10 The text shows examples of emotions such as "peace of mind", "stability", and "reassurance" that have similar emotion values.

[0389] The above description focuses on the functions of the data processing device 12, but the system of this disclosure is not necessarily installed on a server. The system of this disclosure can also be installed as a general information processing system. This disclosure can also be installed, for example, as a software program running on a personal computer, an application running on a smartphone, etc. The method of this disclosure can also be provided to users in the form of SaaS (Software as a Service).

[0390] In the above embodiments, an example of a specific process being performed by a single computer 22 is given. However, the technology disclosed herein is not limited to this, and the specific process can also be distributed among multiple computers, including computer 22. For example, the data generation model 58 can be located on an external device of the data processing apparatus 12, where data is generated based on the input data.

[0391] In the above embodiments, examples of storing a specific processing program 56 in the memory 32 have been described, but the technology disclosed herein is not limited thereto. For example, the specific processing program 56 may also be stored in a portable computer-readable non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed into the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0392] Alternatively, a specific processing program 56 may be pre-stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 according to the requirements of the data processing device 12.

[0393] In addition, it is not necessary to store all the specific processing program 56 in the storage device such as the server connected to the data processing device 12 via the network 54 or in the memory 32; a portion of the specific processing program 56 may be stored in advance.

[0394] As hardware resources for performing specific processes, various processors, as shown below, can be used. For example, a CPU can be listed as a processor, which functions as a general-purpose processor that performs specific processes by executing software, i.e., a program. Furthermore, processors can be listed as special-purpose circuits such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application-Specific Integrated Circuits), which are processors with circuitry specifically designed to perform specific processes. Each processor has built-in or connected memory, and each processor executes specific processes using that memory.

[0395] The hardware resources for performing a specific process can consist of one of these various processors, or a combination of two or more processors of the same or different types (e.g., a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resources for performing a specific process can be a single processor.

[0396] As an example of a single processor, there are two approaches: First, a processor is composed of a combination of one or more CPUs and software, which functions as a hardware resource to perform a specific process; second, as represented by a SoC (System-on-a-chip), a processor is used to implement the functionality of the entire system, which includes multiple hardware resources for performing a specific process, using a single IC (Integrated Circuit) chip. In this way, the specific process is implemented by using one or more of the aforementioned processors as hardware resources.

[0397] Furthermore, the hardware architecture of these various processors, more specifically, can utilize circuits that combine semiconductor elements and other circuit components. Moreover, the specific process described above is just one example. Therefore, without departing from the main point, unnecessary steps can certainly be deleted, new steps added, or the processing order changed.

[0398] The descriptions and illustrations above are detailed explanations of a portion of the technology disclosed herein, and are merely one example of the technology disclosed herein. For example, the above descriptions of the structure, function, effect, and results are just one example of the structure, function, effect, and results of a portion of the technology disclosed herein. Therefore, without departing from the spirit of the technology disclosed herein, unnecessary parts may be deleted, new elements added, or replacements may be made to the descriptions and illustrations above. Furthermore, to avoid confusion and facilitate understanding of a portion of the technology disclosed herein, explanations of common technical knowledge that do not require special explanation under the premise of being able to implement the technology disclosed herein have been omitted from the descriptions and illustrations above.

[0399] All documents, patent applications and technical specifications set forth in this specification are incorporated herein by reference to the same extent that each document, patent application and technical specification is specifically and individually described therein and referenced by reference.

[0400] In addition, the following notes are provided in response to the above explanation.

[0401] Example 1

[0402] (Note 1)

[0403] An information processing system includes: an information processing device for acquiring user attribute information; an information acquisition device for periodically acquiring supply location attribute information; a computing device for automatically generating cooking instruction information using a generative artificial intelligence model based on the user attribute information and the supply location attribute information; an information providing device for notifying an information processing terminal of the generated cooking instruction information; a guidance device for displaying item information and its location guidance on the information processing terminal based on the cooking instruction information; an information management device for storing and updating user attribute information and supply location attribute information in an information processing server; a data parsing device for parsing the periodically acquired supply location attribute information and processing the item inventory status, price information, shelf life information, and sales priority attribute information of the supply location; and an input generation device for generating prompt statements based on user attribute information and supply location attribute information and dynamically constituting the input data for the generative artificial intelligence model.

[0404] (Note 2)

[0405] The information processing system according to Appendix 1 further includes: a processing device that, based on the shelf life information and inventory information in the supply location attribute information, identifies items nearing their expiration date or surplus inventory items, prioritizes the use of the identified items for regenerating cooking instruction information, and provides a prompt at the information processing terminal.

[0406] (Note 3)

[0407] The information processing system according to Appendix 1 further includes: a processing device that generates a recommendation message for the auxiliary items based on the user's purchase history information and attribute information, specific auxiliary item information, and provides the recommendation message at the information processing terminal.

[0408] Application Example 1

[0409] (Note 1)

[0410] An information processing system includes: a device for inputting individual information; a device for acquiring information about a providing location; a device for generating prompt statements based on the individual information and the providing location information using an information generation device, and inputting the prompt statements into a generative artificial intelligence model to obtain optimal proposal information; a device for displaying the proposal information obtained by the generative artificial intelligence model to the individual; a device for guiding the individual on the placement location or acquisition path of items included in the proposal information; a device for acquiring the individual's state information using an emotion recognition device, and adjusting the prompt statements input into the generative artificial intelligence model based on the state information; and a device for displaying the optimized proposal information to the individual again based on the state information.

[0411] (Note 2)

[0412] The information processing system according to Appendix 1 also includes means for prioritizing items based on validity period or inventory quantity and generating proposal information containing the selected items.

[0413] (Note 3)

[0414] The information processing system according to Note 1 further includes means for identifying and recommending specific associated supplementary items to an individual based on previously acquired records or attribute information.

[0415] Example 2

[0416] (Note 1)

[0417] An information processing system includes: a device for inputting user attribute data; a device for acquiring trade information including location inventory information, price information, and expiration date information; a device for inputting predetermined prompt statements into a generative artificial intelligence model based on the user attribute data and the trade information to generate recipe instruction information; a device for displaying the generated recipe instruction information to the user via a terminal; a device for guiding the identification of required consumer goods and shopping assistance information based on the recipe instruction information; a device for acquiring user facial expression data or voice data to obtain emotional state information; a device for parsing the emotional state information and adjusting the recipe instruction information according to the user's emotional state; a device for recommending additional consumer goods to the user based on the user's purchase history and preference information; and a device for acquiring information on new or key products, generating recipe instruction information containing the new or key products, and displaying it to the user.

[0418] (Note 2)

[0419] The information processing system according to Appendix 1 includes means for identifying consumer goods with nearing expiration dates from trade information and for preferentially generating recipe instruction information containing the identified consumer goods.

[0420] (Note 3)

[0421] The information processing system according to Appendix 1 includes a device for recommending beverages, food, or hobbies to users as supplementary consumer goods.

[0422] Application Example 2

[0423] (Note 1)

[0424] An information processing system includes: an information acquisition device for inputting user information; an information acquisition device for acquiring point-of-sale information; an information processing device for generating prompt statements for a generative artificial intelligence model based on the user information and the point-of-sale information, and generating cooking plan information using the prompt statements; an information output device for presenting the generated cooking plan information to the user; an information providing device for providing instructions on required raw material information based on the cooking plan information; an emotion acquisition device for acquiring the user's emotional state; and an information processing device for adjusting the cooking plan information based on the acquired emotional state.

[0425] (Note 2)

[0426] The information processing system according to Appendix 1 further includes: an information processing device for identifying items with an approaching expiration date based on point-of-sale information containing expiration date information, and preferentially using the identified items to generate cooking recipe information.

[0427] (Note 3)

[0428] The information processing system according to Appendix 1 further includes: an information processing device for recommending additional products based on the user's purchase history, preference information, and sentiment information.

Claims

1. An information processing system, characterized in that, include: Device for inputting user information; Devices used to acquire store information; A device for generating recipes based on the user information and the store information; A device used to display the generated recipes to the user; and A device used to guide the required ingredients.

2. The information processing system according to claim 1, characterized in that, Also includes: A device for identifying products nearing their expiration date and prioritizing the use of these products to generate recipes.

3. The information processing system according to claim 1, characterized in that, Also includes: A device for recommending add-on items, including beverages and candy.

Images