system

Abstract

To provide a system that allows users to enjoy a personalized shopping experience based on their preferences and purchasing history, and also allows users to obtain sales and coupon information in real time. [Solution] A system including a means for a user to check in to a facility, a means for a terminal to acquire location information and send it to a server, a means for the server to acquire the user's profile information and purchase history, a means for the server to analyze the user's hobbies and preferences using AI and recommend products and stores, a means for the server to generate and send an action plan customized for the user, a means for the terminal to notify the user of the action plan and display it, a means for the terminal to track the user's action data and send it to the server, and a means for the server to analyze the new action data and optimize the plan.

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] While personalized product recommendations are common in online shopping, it is extremely difficult for users to find products and stores that interest them in physical shopping malls. This is especially true in urban shopping malls with numerous stores, where it is difficult for users to efficiently search for products that suit them. Opportunities to obtain real-time sales and coupon information are also limited. This makes the user's shopping experience less engaging and efficient. Furthermore, shopping malls lack effective ways to communicate sales and events at each store to users. [Means for solving the problem]

[0005] The present invention provides a system including: a means for a user to check in to a facility; a means for a terminal to acquire location information and transmit it to a server; a means for the server to acquire the user's profile information and purchase history; a means for the server to analyze the user's hobbies and preferences using AI and recommend products and stores; a means for the server to generate and transmit a customized action plan for the user; a means for the terminal to notify and display the action plan to the user; a means for the terminal to track the user's behavioral data and transmit it to the server; and a means for the server to analyze the new behavioral data and optimize the plan. This allows users to enjoy a personalized shopping experience based on their preferences and purchase history, and to obtain sale and coupon information in real time. It also enables stores in shopping malls to effectively deliver promotional information to users.

[0006] "Users" are ordinary consumers who use the system to search for and purchase products within the shopping mall.

[0007] A "facility" is a commercial facility that houses multiple stores, such as a shopping mall or department store.

[0008] "Checking in" is the act of a user confirming their location through the system and sending that information to the server to register their visit.

[0009] A "terminal" is a mobile terminal device that a user can carry around, such as a smartphone or tablet.

[0010] "Location information" refers to data about a user's current geographic location obtained via GPS or Wi-Fi.

[0011] A "server" is a computer system that stores user data and performs key processing such as AI analysis.

[0012] "Profile information" is data such as a user's personal preferences and purchasing history that has been collected in advance.

[0013] "Purchase history" is a record of products and services purchased by a user in the past.

[0014] "AI" stands for Artificial Intelligence, a technology that analyzes data and generates personalized product recommendations and action plans.

[0015] "Behavioral data" is a record including the route and actions taken by the user within the shopping mall.

[0016] An "action plan" is a plan of recommended stores and products to visit and activities that AI generates based on the user's hobbies and preferences.

[0017] "Tracking" is the act of recording and following a user's movements and actions in real time through a device.

[0018] "Sales Information" is the latest information on discounts and special offers offered by each store.

[0019] "Coupon information" refers to information about discount coupons or bonus codes that can be used to purchase specific products.

[0020] "Notifications" are alerts or messages that a device sends to a user to inform them of new information or updates.

[0021] "Optimization" is the process of recalculating and updating the user's best action plan based on new data collected. [Brief explanation of the drawings]

[0022] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3]FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0023] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0024] First, the terms used in the following description will be explained.

[0025] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0026] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0027] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0028] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0029] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0030] [First embodiment]

[0031] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0032] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0033] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0034] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0035] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0036] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0037] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0038] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0039] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0040] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0041] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0042] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0043] To implement the present invention, a user-owned terminal, a server, and an infrastructure for connecting them via communications are required. The processing flow of the entire system will be specifically described below.

[0044] Overall system processing flow

[0045] 1. A user checks in to a shopping mall

[0046] When a user arrives at the shopping mall, they launch the dedicated application on their device and open the check-in screen. When they tap the check-in button, the device obtains the user's location information. This location information is obtained with high accuracy using GPS and Wi-Fi.

[0047] The device sends the acquired location information to a server, which confirms that the user is located within a specific shopping mall.

[0048] 2. Get the user's profile information

[0049] The server identifies the user ID based on the received check-in information and retrieves the user's profile information and purchase history from the database, including data on past purchases and favorite stores.

[0050] 3. AI-based data analysis and product recommendations

[0051] The AI ​​module installed on the server combines the acquired user profile information with current location information to analyze the user's hobbies and preferences, taking into account purchasing history and favorite trends.

[0052] Based on the analysis results, the AI ​​will recommend the best products and stores for the user, including items and stores that the user may be interested in.

[0053] 4. Generate a customized action plan

[0054] The server then generates a personalized action plan based on the AI ​​analysis results, which includes recommended products and store visit routes.

[0055] The server obtains sales and coupon information provided by each store in the shopping mall and incorporates this information into an action plan.

[0056] 5. User Notification and Display

[0057] The server transmits the generated action plan to the terminal.

[0058] The device notifies the user of the received action plan and displays the details within the application. For example, it may send a notification to the user saying, "We've created a recommended shopping route for you."

[0059] 6. Real-time feedback and optimization

[0060] The device tracks the user's actions in real time, in particular recording the stores the user actually visits and the route they take.

[0061] The device sends the collected behavioral data to the server.

[0062] The server analyzes the new behavioral data it receives, and the AI ​​dynamically optimizes the plan. For example, if the user stops at an unexpected store, it will use that information to make new recommendations or adjust the plan.

[0063] Specific examples

[0064] For example, when a user checks in to a shopping mall, the system will recommend fashion brand stores that the user is likely to be particularly interested in based on the user's past purchasing history. It also provides information about limited-time sales, allowing the user to efficiently search for products that interest them. Furthermore, if the user is also interested in a modeling event, they will also be notified of information about that event. In this way, customized action plans maximize the user's convenience and enjoyment.

[0065] The processing flow will be explained below.

[0066] Step 1:

[0067] When the user arrives at the shopping mall, they launch the dedicated application on their device, open the check-in screen, and tap the check-in button.

[0068] Step 2:

[0069] The device uses GPS sensors and Wi-Fi information to obtain the user's location, which is determined with a high degree of accuracy.

[0070] Step 3:

[0071] The device sends the acquired location information to the server, which confirms that the user is located within a specific shopping mall.

[0072] Step 4:

[0073] The server identifies the user ID based on the check-in information it receives, and retrieves the user's profile information and purchase history from the database.

[0074] Step 5:

[0075] An AI module installed on the server integrates the acquired user profile information with the user's current location information, and analyzes the user's hobbies and preferences.

[0076] Step 6:

[0077] Based on the results of the AI ​​analysis, the system recommends the most suitable products and stores for the user, including items and stores that are likely to interest the user.

[0078] Step 7:

[0079] The server generates a personalized action plan based on the AI ​​analysis results, which includes recommended products and store visit routes.

[0080] Step 8:

[0081] The server acquires sale information and coupon information provided by each store in the shopping mall and incorporates this information into an action plan.

[0082] Step 9:

[0083] The server transmits the generated action plan to the terminal.

[0084] Step 10:

[0085] The device notifies the user of the action plan it has received, such as sending a notification that a "recommended shopping route has been created."

[0086] Step 11:

[0087] When the user taps the notification, the details of the action plan will be displayed on the device, and the user can check the recommended stores, products, and sales information.

[0088] Step 12:

[0089] The device tracks the user's actions in real time, including recording the stores they visit and the route they take.

[0090] Step 13:

[0091] The device transmits the collected behavioral data to a server.

[0092] Step 14:

[0093] The server analyzes the new behavioral data it receives, and the AI ​​dynamically optimizes the plan. For example, if the user stops at an unexpected store, it will use that information to make new recommendations or adjust the plan.

[0094] Example 1

[0095] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0096] In recent years, there has been a demand for systems that utilize users' location information and purchase history to provide personalized, optimized shopping plans to improve the shopping experience in facilities. However, existing systems are unable to fully utilize users' preferences and real-time behavioral data, making it difficult to recommend optimal products and stores or provide appropriate shopping plans. Furthermore, due to the lack of dynamic feedback and optimization, these systems are unable to respond immediately to changes in user behavior.

[0097] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0098] In this invention, the server includes a means for a user to check in to a facility, a means for a terminal to acquire location information and send it to the server, a means for the server to acquire user profile information and purchase history, a means for analyzing the user's hobbies and preferences using AI and recommending products and stores, a means for generating and sending an action plan customized for the user, a means for notifying the user of the action plan and displaying it, a means for tracking the user's behavioral data and sending it to the server, and a means for analyzing new behavioral data and optimizing the plan. This makes it possible to recommend products and stores based on the user's hobbies and preferences, and to provide an optimal action plan that reflects real-time behavioral data.

[0099] A "user" is a person who uses the system to receive services or products within a facility.

[0100] A "facility" is a location where a particular service is provided, such as a shopping mall or public building where check-in occurs.

[0101] A "terminal" is a computing device carried by a user, such as a smartphone or tablet.

[0102] "Location information" refers to information that indicates the current location of a user or device, obtained using GPS or Wi-Fi.

[0103] "Server" means a remote computing system that receives User information and processes and analyzes the data.

[0104] "Profile information" refers to data that compiles attribute information and past behavioral history about a user.

[0105] "Purchase history" is a record of products purchased by a user in the past and a history of stores visited.

[0106] "AI" refers to an algorithm that uses artificial intelligence technology to analyze data and understand users' tastes and preferences.

[0107] "Hobbies and preferences" refers to information indicating the types of products and services that a user likes.

[0108] A "customized action plan" is a recommended plan of activities and destinations generated based on a user's individual needs and past behavior.

[0109] "Behavioral data" refers to real-time behavioral information such as a user's current location, travel route, and stores visited.

[0110] "Optimization" refers to the process of optimizing action plans and recommendations based on collected data.

[0111] "Sale information" refers to information about discounts and special offers offered by stores and facilities.

[0112] "Coupon information" refers to information about discount coupons and special coupons that can be used by users.

[0113] To implement the present invention, a user-owned terminal, a server, and an infrastructure for connecting them via communications are required. The specific operation of the system will be described below.

[0114] First, when a user arrives at a shopping mall or other facility, they launch a dedicated application on their smartphone, tablet, or other device and open the check-in screen. When they tap the check-in button, the device uses its GPS sensor and Wi-Fi module to obtain location information. This location information is converted into JSON format and sent over the Internet to a server. The server analyzes the information and records it in a database. This initial process confirms that the user is within a specific facility.

[0115] The server then identifies the user based on their check-in information and retrieves their profile information and purchase history from a relational database, which executes SQL queries to pull the required data. This information is then passed to the AI ​​module.

[0116] The AI ​​module in the server integrates the acquired profile information and purchase history to analyze the user's hobbies and preferences. The AI ​​module evaluates the category and price of the products the user has purchased in the past, as well as the frequency of visits, and recommends the most suitable products and stores for the user. The recommendation results are stored as a data structure.

[0117] Next, the server generates a customized action plan for the user based on the analysis results from the AI. This action plan includes the recommended order in which to visit stores and information on the user's movement within the store. The server also obtains the latest sales and coupon information provided by each store in the shopping mall and incorporates it into the action plan. This information is obtained using an API request. The generated action plan is formatted in JSON format and sent to the device.

[0118] The device analyzes the received action plan and notifies the user. Once the push notification is sent, the user can check the detailed action plan in a dedicated application. The plan includes store location information, the order in which to visit, sale information, and more.

[0119] Additionally, the device tracks the user's current location and travel route in real time and periodically sends the collected behavioral data to the server. This data is securely transmitted using HTTPS. The server then inputs the new behavioral data it receives back into the AI ​​module to dynamically optimize the plan. For example, if the user stops at an unexpected store, that information can be used to make new recommendations or adjust the plan. The new optimized plan is then sent back to the device and the user is notified.

[0120] Specific examples

[0121] For example, when a user checks in to a shopping mall, the system will recommend fashion brand stores that the user is likely to be interested in based on the user's past purchasing history. It also provides information about limited-time sales, allowing the user to efficiently search for products that interest them. Furthermore, if the user is also interested in a modeling event, they will be notified of information about that event. In this way, customized action plans maximize the user's convenience and enjoyment.

[0122] Example prompts for generative AI models

[0123] 1. "Please explain the specific steps a user takes to check in to a shopping mall."

[0124] 2. "Please use AI to analyze data based on user profile information and purchase history."

[0125] 3. "Explain how to dynamically optimize shopping plans based on real-time behavioral data."

[0126] By using these prompt sentences, it is expected that the generative AI model will understand the overall system processing and provide appropriate answers.

[0127] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0128] Step 1:

[0129] After arriving at the shopping mall, the user launches a dedicated application on their smartphone and opens the check-in screen. When they tap the "Check-in" button, the device uses the GPS sensor and Wi-Fi module to obtain their current location information. This location information is converted into JSON format and sent to the server. The input is the user's operation, and the output is location information data in JSON format. Specifically, the device calls an internal API to obtain location information and sends that data to the server.

[0130] Step 2:

[0131] The server executes a database query to identify the user ID based on the received location data. The server retrieves the user's profile information and purchase history from the database. The input is location data and user ID, and the output is profile information and purchase history data. Specifically, the server uses an SQL query to extract the required information from the database.

[0132] Step 3:

[0133] The AI ​​module installed on the server analyzes the user's hobbies and preferences using the acquired profile information and purchase history. The AI ​​module evaluates past purchase history and favorite trends and calculates the best products and stores for the user. The input is profile information and purchase history data, and the output is a list of recommended products. Specifically, the AI ​​algorithm analyzes this data and generates the analysis results.

[0134] Step 4:

[0135] The server generates an action plan customized for the user based on the analysis results from the AI ​​module. This plan includes the recommended order in which to visit stores and information on movement within the store. The server also obtains the latest sales and coupon information provided by stores in the shopping mall via API requests and incorporates this into the action plan. The input is the analysis results and store information, and the output is the action plan. Specifically, the server runs the plan generation algorithm, collects the necessary store information, and completes the action plan.

[0136] Step 5:

[0137] The server sends the generated action plan in JSON format to the device. The device analyzes the received action plan and sends a push notification saying, "We've created a recommended shopping route for you." The user can check the detailed action plan in a dedicated application. The input is the action plan data, and the output is a push notification and display information for the user. Specifically, the device analyzes the received data and notifies the user of the information.

[0138] Step 6:

[0139] The device tracks the user's current location and travel route in real time and periodically sends this behavioral data to the server. For example, it sends the data securely using HTTPS. The server inputs the new behavioral data it receives into the AI ​​module and dynamically optimizes the plan. If the user stops at an unexpected store, that information is used to make new recommendations and adjust the plan. The input is the new behavioral data, and the output is an optimized plan. Specifically, the AI ​​recalculates the plan based on the real-time data and sends the updated plan to the device.

[0140] Through these steps, the system leverages users' real-time behavioral data to provide a personalized shopping experience.

[0141] (Application example 1)

[0142] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0143] Modern shopping malls have many stores and products, making it difficult for users to shop efficiently and comfortably. It takes time and effort, especially, for users to find products and stores that interest them. Furthermore, if sales and coupon information is not provided appropriately, users may lose motivation to purchase. Furthermore, if plans are not optimized in real time according to user behavior, the user experience may be impaired.

[0144] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0145] In this invention, the server includes means for generating prompts recommended by AI based on the user's purchase history and profile information, means for notifying and optimizing an action plan including recommended stores and promotion information in real time, means for the user to check in to a facility, means for the terminal to acquire location information and send it to the server, means for the server to acquire the user's profile information and purchase history, means for the server to analyze the user's hobbies and preferences using AI and recommend products and stores, means for the server to generate and send an action plan customized for the user, means for the terminal to notify and display the action plan to the user, means for the terminal to track the user's behavioral data and send it to the server, and means for the server to analyze new behavioral data and optimize the plan. This allows users to efficiently find products and stores of their interest, and by applying sale information and coupon information, purchasing motivation is increased, and the user experience is optimized through real-time feedback.

[0146] "Checking in" refers to the act of notifying a server of a user's location information via a dedicated application when the user arrives at a facility.

[0147] A "terminal" is a device such as a smartphone or tablet operated by a user, and is a device for obtaining location information, sending notifications, displaying action plans, and the like.

[0148] "Location information" refers to geographical location data obtained using GPS or Wi-Fi, and is information used to identify a user's current location.

[0149] A "server" is a computer system that manages user profile information and purchase history, and recommends products and stores to users.

[0150] "Profile information" refers to integrated data such as a user's attributes, past purchasing history, interests and preferences, and generally refers to information about an individual user.

[0151] "Purchase history" is a record of products and services purchased by a user in the past, and is data used to analyze the user's purchasing trends.

[0152] "AI (artificial intelligence)" is a general term for algorithms and technologies that use user profile information and purchasing history to analyze a user's hobbies and preferences and recommend the most suitable products and stores.

[0153] "Hobbies and preferences" refers to the interests and concerns a user has toward specific products and services, and is an indicator for predicting user behavior and preferences.

[0154] An "action plan" is a plan generated by the server that includes a route to visit, recommended stores, product information, and sale information for a user.

[0155] "Notification" refers to a message or alert sent from the server to the user's device, and is a means of conveying action plans and recommended information to the user.

[0156] "Tracking" is the process of following a user's behavior in real time and recording the data obtained.

[0157] "Plan optimization" is the process of analyzing new user behavior data and updating the current behavior plan to an optimal form.

[0158] "Recommendation" is the act of AI selecting products and stores that a user might be interested in based on the user's profile information and purchasing history.

[0159] A "prompt" is text data that AI uses to give instructions and recommendations to users to show them the best course of action or choice.

[0160] "Real-time" means processing data instantly in response to changes in user behavior and the environment, and providing appropriate information immediately.

[0161] To implement this invention, a user-owned terminal, a server, and an infrastructure for connecting these via communications are required. The processing of the entire system will be described below.

[0162] System configuration

[0163] The system includes a terminal application for users to check in, a server that receives and processes data, and a tracking function for acquiring user behavior data. The terminal uses a GPS module and Wi-Fi adapter to obtain highly accurate location information and send it to the server. The server has a database containing user profile information and purchase history, an AI module, and a notification system.

[0164] Program Implementation

[0165] 1. User checks in

[0166] When a user arrives at a shopping mall or other facility, they launch a dedicated application on their smartphone and check in. When they tap the check-in button, the device uses GPS and Wi-Fi to obtain their current location information and sends it to a server.

[0167] 2. The server processes the data

[0168] The server identifies the user ID based on the received location information. It also retrieves the user's profile information and purchase history from the database. Based on this information, the AI ​​module analyzes the user's hobbies and preferences.

[0169] 3. Recommendations by AI module

[0170] The AI ​​module uses the user's profile information and purchase history to recommend products and stores that the user may be interested in. This is done using AI frameworks such as TENSORFLOW (registered trademark) and PyTorch.

[0171] 4. Generate a customized action plan

[0172] Based on the analysis results of the AI ​​module, the server generates an optimal action plan for the user, which includes recommended store visit routes, sale information, coupon information, and more.

[0173] 5. User Notification and Interface

[0174] The terminal notifies the user of the action plan received from the server and displays it in the application. The action plan displayed includes a message such as "Here is a recommended route to the stores for you."

[0175] 6. Real-time feedback

[0176] The device tracks the user's actual behavior in real time and sends the data to the server, which then re-analyzes the data based on the new behavior data and dynamically optimizes the action plan.

[0177] The specific hardware and software used includes:

[0178] Hardware: Smartphone with built-in GPS sensor.

[0179] Software: Dedicated application, server is Django (backend framework), Python, SQLite (database), AI module is TensorFlow or PyTorch.

[0180] Specific examples

[0181] For example, when a user checks in to a shopping mall, AI analyzes their past purchase history and finds that they particularly like a certain fashion brand. Based on this, the user is notified of current sales at the fashion brand and the route to the store. Specific examples of prompts include the following:

[0182] "User profile: {Name: User A, Age: 30s, Gender: Female, Interests: Fashion, Purchase history: Fashion brand products.} Based on this data, please provide the optimal shopping plan for this user."

[0183] This allows users to enjoy shopping efficiently and improves the quality of their experience.

[0184] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0185] Step 1:

[0186] When a user arrives at a facility such as a shopping mall, they launch a dedicated application on their smartphone and tap the check-in button. The device uses a GPS module and Wi-Fi adapter to obtain current location information and sends it to a server. The input is the user's operation upon arriving at the facility, and the output is the transmission of location information to the server.

[0187] Step 2:

[0188] The server identifies the user ID based on the received location information. The server then retrieves the user's profile information and purchase history from the database. The input is location information, and the output is the extracted user ID, profile information, and purchase history.

[0189] Step 3:

[0190] The AI ​​module installed on the server integrates the acquired user profile information and purchase history. Based on this, the AI ​​module analyzes the user's hobbies and preferences. The input is the integrated profile information and purchase history, and the output is the analysis results of the user's hobbies and preferences. TensorFlow and PyTorch are used for the AI ​​module.

[0191] Step 4:

[0192] The server recommends the best products and stores for the user based on the analysis results of the AI ​​module. These recommendations include items and stores that the user is likely to be interested in. The input is the analysis results of the AI ​​module, and the output is a list of recommended products and stores.

[0193] Step 5:

[0194] The server generates a customized action plan based on the information on recommended stores and products. This plan includes the recommended store visit route, sale information, coupon information, etc. The input is a list of recommended stores and products, and the output is the customized action plan.

[0195] Step 6:

[0196] The server sends the generated action plan to the user's device. The device notifies the user of the received action plan and displays it in the application. The input is the action plan, and the output is notification and display to the user. The displayed action plan includes a message such as "Here is a recommended route to the stores for you."

[0197] Step 7:

[0198] The terminal tracks the user's actual behavior in real time and transmits the data to the server. The input is the user's behavior data, and the output is the transmission of the behavior data to the server.

[0199] Step 8:

[0200] The server then performs a second analysis based on the new behavioral data and dynamically optimizes the behavior plan. The input is the new behavioral data, and the output is the optimized behavior plan. This allows for real-time optimization of the plan according to the user's behavior.

[0201] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0202] To implement the present invention, a user-owned terminal, a server, an emotion engine, and an infrastructure for connecting these via communications are required. The processing flow of the entire system will be specifically explained below.

[0203] Overall system processing flow

[0204] 1. A user checks in to a shopping mall

[0205] When a user arrives at the shopping mall, they launch the dedicated application on their device and open the check-in screen. When they tap the check-in button, the device obtains the user's location information. This location information is obtained with high accuracy using GPS and Wi-Fi.

[0206] The device sends the acquired location information to a server, which confirms that the user is located within a specific shopping mall.

[0207] 2. Obtaining user profile information and emotion data

[0208] The server identifies the user ID based on the received check-in information and retrieves the user's profile information and purchase history from the database.

[0209] The user's facial expressions and tone of voice are captured by the device's camera and microphone and sent to the emotion engine.

[0210] The emotion engine analyzes the user's facial expression data and tone of voice to determine their current emotional state.

[0211] 3. Data analysis and product recommendations using AI and emotion engines

[0212] The AI ​​module installed on the server integrates the acquired user profile information, current location information, and emotional data to analyze the user's hobbies and preferences.

[0213] Based on the analysis, the AI ​​and emotion engine will recommend the best products and stores for the user, including items and stores that are likely to interest the user.

[0214] 4. Generate a customized action plan

[0215] The server uses the AI ​​and emotion engine analysis results to generate a personalized action plan for the user, including recommended products and store routes.

[0216] The server obtains sales and coupon information provided by each store in the shopping mall and incorporates this information into an action plan.

[0217] 5. User Notification and Display

[0218] The server transmits the generated action plan to the terminal.

[0219] The device notifies the user of the received action plan and displays the details within the application. For example, it may send a notification to the user saying, "We've created a recommended shopping route for you."

[0220] 6. Real-time feedback and optimization

[0221] The device tracks the user's actions in real time, in particular recording the stores the user actually visits and the route they take.

[0222] The device sends the collected behavioral data to the server.

[0223] The server analyzes the new behavioral data and updated emotional data it receives, and the AI ​​dynamically optimizes the plan based on this. For example, if the user stops at an unexpected store, it will use that information to make new recommendations or adjust the plan.

[0224] Specific examples

[0225] For example, when a user checks in to a shopping mall, the system will recommend fashion brand stores that the user is likely to be interested in based on the user's past purchasing history and current emotional state. It also provides information about products on limited-time sales, allowing users to efficiently search for products that interest them. Furthermore, the system will notify users of events that may interest them (e.g., modeling events). In this way, customized action plans maximize user convenience and enjoyment. By utilizing feedback from the emotion engine, if the user is excited about a particular product, it is also possible to suggest other products in that genre or related events in real time.

[0226] The processing flow will be explained below.

[0227] Step 1:

[0228] When the user arrives at the shopping mall, they launch the dedicated application on their device, open the check-in screen, and tap the check-in button.

[0229] Step 2:

[0230] The device uses GPS sensors and Wi-Fi information to obtain the user's location, which is determined with a high degree of accuracy.

[0231] Step 3:

[0232] The device sends the acquired location information to the server, which confirms that the user is located within a specific shopping mall.

[0233] Step 4:

[0234] The server identifies the user ID based on the check-in information it receives, and retrieves the user's profile information and purchase history from the database.

[0235] Step 5:

[0236] The device captures the user's facial expression data with a camera and their tone of voice with a microphone, and sends the acquired data to the emotion engine.

[0237] Step 6:

[0238] The emotion engine analyzes the user's facial expression data and tone of voice to determine their current emotional state, which can include happiness, excitement, focus, etc.

[0239] Step 7:

[0240] The AI ​​module on the server integrates the acquired user profile information, current location information, and emotional data, and analyzes the user's hobbies and preferences.

[0241] Step 8:

[0242] Based on the results of analysis by AI and an emotion engine, the system recommends the most suitable products and stores for users, including items and stores that are likely to interest them.

[0243] Step 9:

[0244] The server generates a personalized action plan based on the analysis results of the AI ​​and emotion engine, including recommended products and store visit routes.

[0245] Step 10:

[0246] The server incorporates sale information and coupon information provided by each store in the shopping mall into the action plan.

[0247] Step 11:

[0248] The server transmits the generated action plan to the terminal.

[0249] Step 12:

[0250] The device notifies the user of the action plan it has received, such as sending a notification that a "recommended shopping route has been created."

[0251] Step 13:

[0252] When the user taps the notification, the details of the action plan will be displayed on the device, and the user can check the recommended stores, products, and sales information.

[0253] Step 14:

[0254] The device tracks the user's actions in real time, including recording the stores they visit and the route they take.

[0255] Step 15:

[0256] The device transmits the collected behavioral data to a server.

[0257] Step 16:

[0258] The server analyzes the new behavioral data and updated emotional data received. The AI ​​dynamically optimizes the plan based on this data. If the user stops at an unexpected store, the AI ​​will use this information to make new recommendations or adjust the plan.

[0259] As a specific example, if the emotion engine determines that a user is feeling excited in front of a specific store in a shopping mall after checking in, it will recommend products and information about other stores that fit that state of excitement in real time. For example, new products from a fashion brand or related accessory stores will be recommended. Also, if the user is feeling relaxed, information about cafes and rest areas will be provided. This allows users to enjoy a shopping experience optimized for their interests and emotions.

[0260] Example 2

[0261] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0262] Conventional shopping support systems typically recommend products and stores based on a user's location information and purchase history. However, these systems are unable to fully consider a user's momentary emotions or specific hobbies and preferences, making it difficult to provide optimal recommendations and action plans. As a result, users' purchasing experiences are limited, reducing the effectiveness of the system.

[0263] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0264] In this invention, the server includes means for acquiring user profile information and purchase history, means for acquiring the user's facial expression data and voice data and transmitting them to the emotion analysis device, means for the emotion analysis device to analyze the user's facial expression data and voice data to determine the user's emotional state, and means for analyzing the user's hobbies, preferences, and emotional state using AI and recommending products and stores. This makes it possible to provide more accurate product and store recommendations and action plans that take the user's emotional state into consideration.

[0265] "User" means a person who uses the system.

[0266] "Facilities" refer to places that users visit, such as shopping malls and commercial facilities.

[0267] "Device" refers to an electronic device such as a smartphone or tablet owned by a user.

[0268] "Location information" means data about a user's current location obtained using GPS, Wi-Fi, etc.

[0269] "Server" refers to a computer system that stores, processes, and distributes data over a network.

[0270] "Profile information" refers to data about a user's personal information and their interests and preferences.

[0271] "Purchase history" means a record of products and services purchased by a user in the past.

[0272] "Facial expression data" refers to digital data relating to the user's facial expressions captured by the device's camera.

[0273] "Voice data" refers to digital data relating to the user's voice, including the tone and content of the user's voice.

[0274] An "emotion analysis device" refers to a system that analyzes facial expression data and voice data to determine a user's emotional state.

[0275] "AI" refers to the technology that uses artificial intelligence to analyze data and understand and predict users' hobbies, preferences, and emotional states.

[0276] "Hobbies and preferences" refers to items and categories in which a user has interests or preferences.

[0277] An "action plan" refers to a plan that includes recommended routes for users to move around the facility efficiently and comfortably, as well as stores and products of interest.

[0278] "Tracking" means tracking a user's actual behavior and movement path in real time.

[0279] "Optimization" refers to dynamically adjusting and improving existing action plans based on new user behavior data.

[0280] "Sale Information" means information about discounts and special offers offered by stores within the shopping mall.

[0281] "Coupon information" refers to information about discount coupons and special offers that users can use.

[0282] To implement the present invention, the following elements are required: a terminal owned by a user, a server, an emotion analysis device, and infrastructure for connecting these via communications. A detailed description of the system is provided below.

[0283] Hardware and Software Configuration

[0284] Device: A smartphone or tablet is used. These devices are equipped with a GPS sensor, camera, microphone, etc. A dedicated application is also installed and is used by the user to check in and collect emotional data.

[0285] Server: A computer system that stores, processes, and distributes data over a network. It uses a database (SQL Server, MySQL (registered trademark), etc.) to manage user profile information and purchase history. It is equipped with an AI module (TensorFlow, PyTorch, etc.) to integrate and analyze data.

[0286] Emotion analysis device: A system for analyzing facial expression data and voice data. It uses a facial expression analysis algorithm (such as OpenCV) and a voice emotion analysis module (such as Google® Cloud Speech-to-Text).

[0287] Communication infrastructure: Connect devices, servers, and emotion analysis devices using the internet or Wi-Fi.

[0288] Details of data processing and data calculation

[0289] When a user arrives at a shopping mall, they launch the dedicated app and display the check-in screen. When the user taps the check-in button, the device uses GPS and Wi-Fi to obtain location information and sends that information to the server. The server identifies the user ID based on this location information and retrieves profile information and purchase history from the database.

[0290] The device then uses a camera and microphone to capture the user's facial expression and voice data, which are then sent to an emotion analyzer. The emotion analyzer analyzes the user's emotional state in real time and sends the results back to the server. The server then uses an AI module to integrate the profile information, location information, and emotion data to analyze the user's interests and preferences.

[0291] As a result, the server recommends the most suitable products and stores for the user and generates a customized action plan based on this. This action plan includes a recommended route and a list of recommended products. The server also obtains sales and coupon information provided by each store in the shopping mall and integrates it into the action plan.

[0292] The generated action plan is sent to the device and notified to the user. The device receives it and displays it to the user. As the user walks the recommended route, the device tracks their actions and collects new action data. This data is sent to the server, and the AI ​​module dynamically optimizes the plan.

[0293] Specific examples

[0294] For example, when a user checks in to a shopping mall, the system will recommend fashion brand stores that the user is likely to be interested in based on the user's past purchasing history and current emotional state. It will also provide information on products on limited-time sales. This allows users to efficiently search for products that interest them. In addition, the system will notify users of events that may interest them (e.g., modeling events). Using feedback from an emotion analyzer, if the user becomes excited in front of a particular product, it will be possible to suggest other products in that genre or related events in real time.

[0295] Prompt Sentence Examples

[0296] "When a user checks in to a shopping mall, the emotion analyzer analyzes the user's emotional state based on data acquired from the device's camera and microphone. Based on the results, recommend the most suitable products and stores for the user."

[0297] In this way, in order to implement the invention, by appropriately using the above-mentioned hardware and software and coordinating the flow of the entire system, it is possible to provide users with an optimal purchasing experience.

[0298] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0299] Step 1:

[0300] A user checks in to a facility

[0301] When a user arrives at the shopping mall, they launch the dedicated application and open the check-in screen. When they tap the check-in button, the device acquires the user's location information.

[0302] Input: User action (tapping the check-in button)

[0303] Output: Obtain location information (latitude, longitude data)

[0304] Specific operation: The device detects the user's current location using the GPS sensor and Wi-Fi and sends the location information to the server.

[0305] Step 2:

[0306] The device acquires location information and sends it to the server.

[0307] The device uses the acquired location information to confirm that the user is within a specific facility, and sends this location information in JSON format to the server.

[0308] Input: Location information (latitude, longitude)

[0309] Output: Send location information to the server

[0310] Specific operation: The device uses the location information API to format the acquired location data into JSON format and send it to the server to determine the user's current location.

[0311] Step 3:

[0312] The server retrieves the user's profile information and purchase history.

[0313] The server identifies the user ID based on the received location information and retrieves profile information and purchase history from the database.

[0314] Input: Location information, user ID

[0315] Output: Profile information, purchase history

[0316] What happens: The server executes a database query to retrieve the user's profile information and purchase history data, which is used in the subsequent analysis process.

[0317] Step 4:

[0318] The device acquires the user's facial expression and voice data and sends it to the emotion analysis device.

[0319] The user's facial expressions and tone of voice are collected from a camera and microphone, and this data is sent to an emotion analysis device.

[0320] Input: facial expression data, voice data

[0321] Output: Send data to emotion analyzer

[0322] Specific operation: The device captures the user's facial expressions using the camera function and records audio using the microphone. This data is then sent to the emotion analysis device in real time.

[0323] Step 5:

[0324] The emotion analyzer analyzes the user's facial expression and voice data to determine their emotional state.

[0325] The emotion analyzer uses facial expression analysis algorithms and voice emotion analysis modules to analyze the user's emotional state.

[0326] Input: facial expression data, voice data

[0327] Output: Emotional state discrimination result

[0328] Specific operation: The emotion analysis device uses tools such as OpenCV and Google Cloud Speech-to-Text to analyze the transmitted facial expression and voice data to determine the user's current emotional state.

[0329] Step 6:

[0330] The server uses AI to analyze the user's tastes and emotional state and recommend products and stores.

[0331] The AI ​​module installed on the server integrates user profile information, purchase history, and emotional state data to analyze hobbies and preferences.

[0332] Input: Profile information, purchase history, emotional state data

[0333] Output: Product and store recommendation results

[0334] Specific operation: The server's AI module uses TensorFlow and PyTorch to analyze user data and generate a list of the most suitable products and stores.

[0335] Step 7:

[0336] The server generates and sends a customized action plan to the user.

[0337] Based on the recommendations from the AI, the server generates an optimal action plan for the user and sends it to the device.

[0338] Input: Product and store recommendation results

[0339] Output: A customized action plan

[0340] Specific operation: The action plan includes the order of stores to visit, the route, and a list of recommended products. The server also obtains sales and coupon information from the shopping mall and incorporates this information into the action plan.

[0341] Step 8:

[0342] The device notifies and displays the action plan to the user.

[0343] The terminal notifies the user of the action plan received from the server and displays the details within the application.

[0344] Input: Customized Action Plan

[0345] Output: User notification and display

[0346] What it does: The device uses in-app notifications to notify the user of the action plan and display a message such as "We've created a recommended shopping route for you."

[0347] Step 9:

[0348] The device tracks user behavior data and sends it to the server.

[0349] The device tracks users' real-time behavior and collects data on specific stores and routes.

[0350] Input: User behavior data

[0351] Output: Sending tracking data to the server

[0352] Specific operation: The device uses GPS and Bluetooth beacons to record the user's movements and periodically transmits them to the server.

[0353] Step 10:

[0354] The server analyzes new behavioral data and optimizes the plan

[0355] The server dynamically optimizes the behavior plan based on the new behavior data received.

[0356] Input: New behavioral data

[0357] Output: Optimized action plan

[0358] Specific behavior: The server's AI takes into account new behavioral data and emotional state, adjusting and updating its behavior plan in real time based on unplanned behaviors and changing interests.

[0359] (Application example 2)

[0360] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0361] In today's shopping experience, users lack the information and guidance they need to make efficient and satisfying purchases. In particular, it is difficult to provide personalized product and store recommendations based on a user's preferences and current emotional state in brick-and-mortar stores. There is also a need for action plans that effectively incorporate real-time sales and coupon information. It is necessary to solve these issues and provide users with an optimal shopping experience.

[0362] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0363] In this invention, the server includes means for acquiring facial expression and voice data from the device's camera and microphone and sending it to an emotion engine, means for the emotion engine to analyze the facial expression and voice data and determine the user's emotional state, and means for the server to use AI to integrate and analyze the user's hobbies, preferences, and emotional data and recommend products and stores. This makes it possible to recommend appropriate products and stores based on the user's hobbies, preferences, and emotional state, and to provide a more personalized action plan by tracking the user's behavioral data.

[0364] A "user" is a consumer who visits a facility or physical store and is the person who receives an action plan or product recommendations.

[0365] "Facility" refers to a collection of commercial facilities and physical stores that users visit, such as shopping malls and department stores.

[0366] "Checking in" is the act of a user arriving at a facility and identifying their location using a smartphone or other device.

[0367] A "terminal" is a mobile device such as a smartphone or tablet that is owned by the user, and is a device that acquires location information, collects emotional data, notifies and displays action plans, etc.

[0368] "Location information" is data that indicates a user's current location and movement patterns, determined using GPS sensors and Wi-Fi information.

[0369] The "server" is a central management system that manages user profile information, purchase history, and emotional data, and analyzes and processes them using AI.

[0370] "Profile information" is user data that includes personalized information such as the user's age, gender, preferences, and past purchasing history.

[0371] "Purchase history" is a record of products and services purchased by a user in the past.

[0372] "AI" is a system that uses artificial intelligence technology to analyze users' preferences and emotions and recommend the most suitable products and stores.

[0373] The "Emotion Engine" is a system that uses facial expression recognition and voice analysis technologies to analyze and determine the user's emotional state.

[0374] "Hobbies and preferences" are attributes preferred by individual users, such as areas of interest and concern, favorite things and activities, etc.

[0375] "Recommending products and stores" means presenting products that may interest the user and stores that they should visit based on data analyzed by AI and an emotion engine.

[0376] An "action plan" is a schedule that includes a list of stores and products to visit and a travel route, and is generated by the server to enable the user to enjoy shopping efficiently.

[0377] "Notification" refers to a push notification or alert that the device sends to inform the user of the generated action plan or recommended information.

[0378] "Tracking" refers to a device recording a user's behavioral data in real time and sending it to a server.

[0379] "Sale information" is product information about discounts and special prices offered by stores.

[0380] "Coupon information" is information about discount coupons and special offers offered by stores.

[0381] "Optimization" is the process by which AI dynamically adjusts and improves its action plan based on the user's new behavioral data and updated emotional data.

[0382] To implement this invention, a server, a user terminal, an emotion engine, and a communication infrastructure are required. Below, an outline of the program processing of the entire system will be explained in detail.

[0383] First, the user arrives at the facility and checks in by launching a dedicated application on their device. The user's device uses a GPS sensor and Wi-Fi information to obtain highly accurate location information. This location information is sent to the server, which confirms that the user is within a specific facility. After check-in, the server retrieves the user's profile information and purchase history from a database based on the user ID.

[0384] The device then uses a camera and microphone to capture the user's facial expressions and voice. This data is sent to the emotion engine, which uses facial expression recognition and voice analysis technologies to determine the user's current emotional state. The emotion engine utilizes facial expression recognition APIs (e.g., Google Face API) and voice analysis APIs (e.g., Google Speech-to-Text).

[0385] Once the user's emotional state is determined, the server integrates and analyzes the profile information, purchase history, location information, and emotional data. This analysis uses artificial intelligence (AI) technology, such as machine learning models using TensorFlow or PyTorch. The server then recommends products and stores based on the user's preferences and emotional state.

[0386] The server then generates a customized itinerary, which includes information such as the stores and products to visit, sales and coupons, and the best route to travel. The itinerary is then sent to the user's device, which notifies the user and displays it within the application. Notifications are sent using Firebase Cloud Messaging (FCM) and Apple Push Notification Service (APNS).

[0387] When a user begins to take action, the device tracks the user's movements and destinations in real time and sends that data to the server. The server analyzes the newly collected behavioral data and updated emotional data, and the AI ​​optimizes the action plan in real time. This allows the device to quickly respond to any new stores or products the user shows interest in.

[0388] For example, when a user checks in to a fashion brand store within the facility, the system will identify the brand that excites them based on their past purchase history and facial recognition, and notify them of the store's sale information in real time. Also, if the user shows an extended interest in a particular product, it will be possible to present related products and events in that genre.

[0389] To accomplish this process, the following example prompts can be used:

[0390] "Create an application that guides users when they check in to a physical store. Based on their past purchase history and current emotional data, it needs to recommend the most suitable stores and products to users. This application will acquire the user's location using the smartphone's GPS and Wi-Fi, and collect emotional data using facial recognition and speech analysis APIs. Finally, it will push a generated action plan to the smartphone. The technologies used should include the following: GPS API, Wi-Fi sensor, Google Face API, Google Speech-to-Text, AWS (registered trademark), TensorFlow, and Firebase Cloud Messaging."

[0391] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0392] Step 1:

[0393] The user arrives at the facility and checks in by launching a dedicated application on the terminal.

[0394] Input: Location information of user and device arriving at facility

[0395] Specific operation: The user opens the smartphone application and taps the check-in button to obtain location information. The device uses the GPS sensor and Wi-Fi information to obtain the precise location.

[0396] Output: Obtained user location information

[0397] Step 2:

[0398] The location information acquired by the device is sent to the server.

[0399] Input: User's location

[0400] Specific operation: The device creates a communication request to send the acquired location information packet to the server and sends it to the server API endpoint.

[0401] Output: User location information received by the server

[0402] Step 3:

[0403] The server retrieves profile information and purchase history from a database based on the user ID.

[0404] Input: User's location and user ID

[0405] Specific operation: The server executes a database query using the location information and user ID as keys to obtain the user's profile information and purchase history.

[0406] Output: User profile information and purchase history

[0407] Step 4:

[0408] The device uses a camera and microphone to capture facial and voice data and sends it to the emotion engine.

[0409] Input: Real-time facial and voice data of the user

[0410] Specific operation: The device camera takes a picture of the user's face and records their voice with the microphone. A request is created to send this data to the emotion engine, and then it is sent.

[0411] Output: Facial and vocal data sent to the emotion engine

[0412] Step 5:

[0413] An emotion engine analyzes facial and voice data to determine the user's emotional state.

[0414] Input: facial expression and voice data

[0415] How it works: The emotion engine uses facial expression recognition APIs and voice analysis APIs to analyze data and determine the user's emotional state (excited, happy, calm, etc.).

[0416] Output: Determined user's emotional state

[0417] Step 6:

[0418] The server integrates and analyzes profile information, purchase history, location information, and emotional data to recommend products and stores.

[0419] Input: Profile information, purchase history, location information, emotional data

[0420] Specific operation: The server uses a generative AI model to analyze the user's hobbies and preferences based on input data and generate a list of optimal products and stores.

[0421] Output: A list of recommended products and stores

[0422] Step 7:

[0423] The server generates and transmits a customized action plan.

[0424] Input: Recommended products and store lists, sale information, coupon information

[0425] Specific operation: The server uses an action plan generation algorithm to generate an action plan that includes the order of stores and products to visit, travel routes, and sale and coupon information. The generated action plan is then sent to the user's device.

[0426] Output: A customized action plan

[0427] Step 8:

[0428] The terminal notifies the user of the generated action plan and displays it within the application.

[0429] Input: Generated action plan

[0430] Specific operation: The device will use a push notification service (FCM or APNS) to send a notification of the action plan to the user. The application will then interactively display a detailed action plan.

[0431] Output: A plan of action communicated to the user and displayed

[0432] Step 9:

[0433] The device tracks the user's behavior in real time and transmits the data to a server.

[0434] Input: Real-time user behavior data

[0435] Specific operation: The device records the user's location and visited destinations in real time, and creates and sends a request to send that data to the server.

[0436] Output: User behavior data sent to the server

[0437] Step 10:

[0438] The server analyzes the new behavioral data and updated emotional data to optimize the behavior plan.

[0439] Input: New behavioral data and updated emotion data

[0440] How it works: Based on the data received in real time, the server uses a generative AI model to dynamically adjust and optimize the action plan, recommending new products and stores, changing the route, etc. as needed.

[0441] Output: Optimized action plan

[0442] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0443] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0444] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0445] [Second embodiment]

[0446] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0447] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0448] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0449] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0450] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0451] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0452] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0453] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0454] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0455] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0456] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0457] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0458] To implement the present invention, a user-owned terminal, a server, and an infrastructure for connecting them via communications are required. The processing flow of the entire system will be specifically described below.

[0459] Overall system processing flow

[0460] 1. A user checks in to a shopping mall

[0461] When a user arrives at the shopping mall, they launch the dedicated application on their device and open the check-in screen. When they tap the check-in button, the device obtains the user's location information. This location information is obtained with high accuracy using GPS and Wi-Fi.

[0462] The device sends the acquired location information to a server, which confirms that the user is located within a specific shopping mall.

[0463] 2. Get the user's profile information

[0464] The server identifies the user ID based on the received check-in information and retrieves the user's profile information and purchase history from the database, including data on past purchases and favorite stores.

[0465] 3. AI-based data analysis and product recommendations

[0466] The AI ​​module installed on the server combines the acquired user profile information with current location information to analyze the user's hobbies and preferences, taking into account purchasing history and favorite trends.

[0467] Based on the analysis results, the AI ​​will recommend the best products and stores for the user, including items and stores that the user may be interested in.

[0468] 4. Generate a customized action plan

[0469] The server then generates a personalized action plan based on the AI ​​analysis results, which includes recommended products and store visit routes.

[0470] The server obtains sales and coupon information provided by each store in the shopping mall and incorporates this information into an action plan.

[0471] 5. User Notification and Display

[0472] The server transmits the generated action plan to the terminal.

[0473] The device notifies the user of the received action plan and displays the details within the application. For example, it may send a notification to the user saying, "We've created a recommended shopping route for you."

[0474] 6. Real-time feedback and optimization

[0475] The device tracks the user's actions in real time, in particular recording the stores the user actually visits and the route they take.

[0476] The device sends the collected behavioral data to the server.

[0477] The server analyzes the new behavioral data it receives, and the AI ​​dynamically optimizes the plan. For example, if the user stops at an unexpected store, it will use that information to make new recommendations or adjust the plan.

[0478] Specific examples

[0479] For example, when a user checks in to a shopping mall, the system will recommend fashion brand stores that the user is likely to be particularly interested in based on the user's past purchasing history. It also provides information about limited-time sales, allowing the user to efficiently search for products that interest them. Furthermore, if the user is also interested in a modeling event, they will also be notified of information about that event. In this way, customized action plans maximize the user's convenience and enjoyment.

[0480] The processing flow will be explained below.

[0481] Step 1:

[0482] When the user arrives at the shopping mall, they launch the dedicated application on their device, open the check-in screen, and tap the check-in button.

[0483] Step 2:

[0484] The device uses GPS sensors and Wi-Fi information to obtain the user's location, which is determined with a high degree of accuracy.

[0485] Step 3:

[0486] The device sends the acquired location information to the server, which confirms that the user is located within a specific shopping mall.

[0487] Step 4:

[0488] The server identifies the user ID based on the check-in information it receives, and retrieves the user's profile information and purchase history from the database.

[0489] Step 5:

[0490] An AI module installed on the server integrates the acquired user profile information with the user's current location information, and analyzes the user's hobbies and preferences.

[0491] Step 6:

[0492] Based on the results of the AI ​​analysis, the system recommends the most suitable products and stores for the user, including items and stores that are likely to interest the user.

[0493] Step 7:

[0494] The server generates a personalized action plan based on the AI ​​analysis results, which includes recommended products and store visit routes.

[0495] Step 8:

[0496] The server acquires sale information and coupon information provided by each store in the shopping mall and incorporates this information into an action plan.

[0497] Step 9:

[0498] The server transmits the generated action plan to the terminal.

[0499] Step 10:

[0500] The device notifies the user of the action plan it has received, such as sending a notification that a "recommended shopping route has been created."

[0501] Step 11:

[0502] When the user taps the notification, the details of the action plan will be displayed on the device, and the user can check the recommended stores, products, and sales information.

[0503] Step 12:

[0504] The device tracks the user's actions in real time, including recording the stores they visit and the route they take.

[0505] Step 13:

[0506] The device transmits the collected behavioral data to a server.

[0507] Step 14:

[0508] The server analyzes the new behavioral data it receives, and the AI ​​dynamically optimizes the plan. For example, if the user stops at an unexpected store, it will use that information to make new recommendations or adjust the plan.

[0509] Example 1

[0510] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0511] In recent years, there has been a demand for systems that utilize users' location information and purchase history to provide personalized, optimized shopping plans to improve the shopping experience in facilities. However, existing systems are unable to fully utilize users' preferences and real-time behavioral data, making it difficult to recommend optimal products and stores or provide appropriate shopping plans. Furthermore, due to the lack of dynamic feedback and optimization, these systems are unable to respond immediately to changes in user behavior.

[0512] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0513] In this invention, the server includes a means for a user to check in to a facility, a means for a terminal to acquire location information and send it to the server, a means for the server to acquire user profile information and purchase history, a means for analyzing the user's hobbies and preferences using AI and recommending products and stores, a means for generating and sending an action plan customized for the user, a means for notifying the user of the action plan and displaying it, a means for tracking the user's behavioral data and sending it to the server, and a means for analyzing new behavioral data and optimizing the plan. This makes it possible to recommend products and stores based on the user's hobbies and preferences, and to provide an optimal action plan that reflects real-time behavioral data.

[0514] A "user" is a person who uses the system to receive services or products within a facility.

[0515] A "facility" is a location where a particular service is provided, such as a shopping mall or public building where check-in occurs.

[0516] A "terminal" is a computing device carried by a user, such as a smartphone or tablet.

[0517] "Location information" refers to information that indicates the current location of a user or device, obtained using GPS or Wi-Fi.

[0518] "Server" means a remote computing system that receives User information and processes and analyzes the data.

[0519] "Profile information" refers to data that compiles attribute information and past behavioral history about a user.

[0520] "Purchase history" is a record of products purchased by a user in the past and a history of stores visited.

[0521] "AI" refers to an algorithm that uses artificial intelligence technology to analyze data and understand users' tastes and preferences.

[0522] "Hobbies and preferences" refers to information indicating the types of products and services that a user likes.

[0523] A "customized action plan" is a recommended plan of activities and destinations generated based on a user's individual needs and past behavior.

[0524] "Behavioral data" refers to real-time behavioral information such as a user's current location, travel route, and stores visited.

[0525] "Optimization" refers to the process of optimizing action plans and recommendations based on collected data.

[0526] "Sale information" refers to information about discounts and special offers offered by stores and facilities.

[0527] "Coupon information" refers to information about discount coupons and special coupons that can be used by users.

[0528] To implement the present invention, a user-owned terminal, a server, and an infrastructure for connecting them via communications are required. The specific operation of the system will be described below.

[0529] First, when a user arrives at a shopping mall or other facility, they launch a dedicated application on their smartphone, tablet, or other device and open the check-in screen. When they tap the check-in button, the device uses its GPS sensor and Wi-Fi module to obtain location information. This location information is converted into JSON format and sent over the Internet to a server. The server analyzes the information and records it in a database. This initial process confirms that the user is within a specific facility.

[0530] The server then identifies the user based on their check-in information and retrieves their profile information and purchase history from a relational database, which executes SQL queries to pull the required data. This information is then passed to the AI ​​module.

[0531] The AI ​​module in the server integrates the acquired profile information and purchase history to analyze the user's hobbies and preferences. The AI ​​module evaluates the category and price of the products the user has purchased in the past, as well as the frequency of visits, and recommends the most suitable products and stores for the user. The recommendation results are stored as a data structure.

[0532] Next, the server generates a customized action plan for the user based on the analysis results from the AI. This action plan includes the recommended order in which to visit stores and information on the user's movement within the store. The server also obtains the latest sales and coupon information provided by each store in the shopping mall and incorporates it into the action plan. This information is obtained using an API request. The generated action plan is formatted in JSON format and sent to the device.

[0533] The device analyzes the received action plan and notifies the user. Once the push notification is sent, the user can check the detailed action plan in a dedicated application. The plan includes store location information, the order in which to visit, sale information, and more.

[0534] Additionally, the device tracks the user's current location and travel route in real time and periodically sends the collected behavioral data to the server. This data is securely transmitted using HTTPS. The server then inputs the new behavioral data it receives back into the AI ​​module to dynamically optimize the plan. For example, if the user stops at an unexpected store, that information can be used to make new recommendations or adjust the plan. The new optimized plan is then sent back to the device and the user is notified.

[0535] Specific examples

[0536] For example, when a user checks in to a shopping mall, the system will recommend fashion brand stores that the user is likely to be interested in based on the user's past purchasing history. It also provides information about limited-time sales, allowing the user to efficiently search for products that interest them. Furthermore, if the user is also interested in a modeling event, they will be notified of information about that event. In this way, customized action plans maximize the user's convenience and enjoyment.

[0537] Example prompts for generative AI models

[0538] 1. "Please explain the specific steps a user takes to check in to a shopping mall."

[0539] 2. "Please use AI to analyze data based on user profile information and purchase history."

[0540] 3. "Explain how to dynamically optimize shopping plans based on real-time behavioral data."

[0541] By using these prompt sentences, it is expected that the generative AI model will understand the overall system processing and provide appropriate answers.

[0542] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0543] Step 1:

[0544] After arriving at the shopping mall, the user launches a dedicated application on their smartphone and opens the check-in screen. When they tap the "Check-in" button, the device uses the GPS sensor and Wi-Fi module to obtain their current location information. This location information is converted into JSON format and sent to the server. The input is the user's operation, and the output is location information data in JSON format. Specifically, the device calls an internal API to obtain location information and sends that data to the server.

[0545] Step 2:

[0546] The server executes a database query to identify the user ID based on the received location data. The server retrieves the user's profile information and purchase history from the database. The input is location data and user ID, and the output is profile information and purchase history data. Specifically, the server uses an SQL query to extract the required information from the database.

[0547] Step 3:

[0548] The AI ​​module installed on the server analyzes the user's hobbies and preferences using the acquired profile information and purchase history. The AI ​​module evaluates past purchase history and favorite trends and calculates the best products and stores for the user. The input is profile information and purchase history data, and the output is a list of recommended products. Specifically, the AI ​​algorithm analyzes this data and generates the analysis results.

[0549] Step 4:

[0550] The server generates an action plan customized for the user based on the analysis results from the AI ​​module. This plan includes the recommended order in which to visit stores and information on movement within the store. The server also obtains the latest sales and coupon information provided by stores in the shopping mall via API requests and incorporates this into the action plan. The input is the analysis results and store information, and the output is the action plan. Specifically, the server runs the plan generation algorithm, collects the necessary store information, and completes the action plan.

[0551] Step 5:

[0552] The server sends the generated action plan in JSON format to the device. The device analyzes the received action plan and sends a push notification saying, "We've created a recommended shopping route for you." The user can check the detailed action plan in a dedicated application. The input is the action plan data, and the output is a push notification and display information for the user. Specifically, the device analyzes the received data and notifies the user of the information.

[0553] Step 6:

[0554] The device tracks the user's current location and travel route in real time and periodically sends this behavioral data to the server. For example, it sends the data securely using HTTPS. The server inputs the new behavioral data it receives into the AI ​​module and dynamically optimizes the plan. If the user stops at an unexpected store, that information is used to make new recommendations and adjust the plan. The input is the new behavioral data, and the output is an optimized plan. Specifically, the AI ​​recalculates the plan based on the real-time data and sends the updated plan to the device.

[0555] Through these steps, the system leverages users' real-time behavioral data to provide a personalized shopping experience.

[0556] (Application example 1)

[0557] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0558] Modern shopping malls have many stores and products, making it difficult for users to shop efficiently and comfortably. It takes time and effort, especially, for users to find products and stores that interest them. Furthermore, if sales and coupon information is not provided appropriately, users may lose motivation to purchase. Furthermore, if plans are not optimized in real time according to user behavior, the user experience may be impaired.

[0559] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0560] In this invention, the server includes means for generating prompts recommended by AI based on the user's purchase history and profile information, means for notifying and optimizing an action plan including recommended stores and promotion information in real time, means for the user to check in to a facility, means for the terminal to acquire location information and send it to the server, means for the server to acquire the user's profile information and purchase history, means for the server to analyze the user's hobbies and preferences using AI and recommend products and stores, means for the server to generate and send an action plan customized for the user, means for the terminal to notify and display the action plan to the user, means for the terminal to track the user's behavioral data and send it to the server, and means for the server to analyze new behavioral data and optimize the plan. This allows users to efficiently find products and stores of their interest, and by applying sale information and coupon information, purchasing motivation is increased, and the user experience is optimized through real-time feedback.

[0561] "Checking in" refers to the act of notifying a server of a user's location information via a dedicated application when the user arrives at a facility.

[0562] A "terminal" is a device such as a smartphone or tablet operated by a user, and is a device for obtaining location information, sending notifications, displaying action plans, and the like.

[0563] "Location information" refers to geographical location data obtained using GPS or Wi-Fi, and is information used to identify a user's current location.

[0564] A "server" is a computer system that manages user profile information and purchase history, and recommends products and stores to users.

[0565] "Profile information" refers to integrated data such as a user's attributes, past purchasing history, interests and preferences, and generally refers to information about an individual user.

[0566] "Purchase history" is a record of products and services purchased by a user in the past, and is data used to analyze the user's purchasing trends.

[0567] "AI (artificial intelligence)" is a general term for algorithms and technologies that use user profile information and purchasing history to analyze a user's hobbies and preferences and recommend the most suitable products and stores.

[0568] "Hobbies and preferences" refers to the interests and concerns a user has toward specific products and services, and is an indicator for predicting user behavior and preferences.

[0569] An "action plan" is a plan generated by the server that includes a route to visit, recommended stores, product information, and sale information for a user.

[0570] "Notification" refers to a message or alert sent from the server to the user's device, and is a means of conveying action plans and recommended information to the user.

[0571] "Tracking" is the process of following a user's behavior in real time and recording the data obtained.

[0572] "Plan optimization" is the process of analyzing new user behavior data and updating the current behavior plan to an optimal form.

[0573] "Recommendation" is the act of AI selecting products and stores that a user might be interested in based on the user's profile information and purchasing history.

[0574] A "prompt" is text data that AI uses to give instructions and recommendations to users to show them the best course of action or choice.

[0575] "Real-time" means processing data instantly in response to changes in user behavior and the environment, and providing appropriate information immediately.

[0576] To implement this invention, a user-owned terminal, a server, and an infrastructure for connecting these via communications are required. The processing of the entire system will be described below.

[0577] System configuration

[0578] The system includes a terminal application for users to check in, a server that receives and processes data, and a tracking function for acquiring user behavior data. The terminal uses a GPS module and Wi-Fi adapter to obtain highly accurate location information and send it to the server. The server has a database containing user profile information and purchase history, an AI module, and a notification system.

[0579] Program Implementation

[0580] 1. User checks in

[0581] When a user arrives at a shopping mall or other facility, they launch a dedicated application on their smartphone and check in. When they tap the check-in button, the device uses GPS and Wi-Fi to obtain their current location information and sends it to a server.

[0582] 2. The server processes the data

[0583] The server identifies the user ID based on the received location information. It also retrieves the user's profile information and purchase history from the database. Based on this information, the AI ​​module analyzes the user's hobbies and preferences.

[0584] 3. Recommendations by AI module

[0585] The AI ​​module uses a user's profile information and purchase history to recommend products and stores that the user might be interested in. It does this using AI frameworks such as TensorFlow and PyTorch.

[0586] 4. Generate a customized action plan

[0587] Based on the analysis results of the AI ​​module, the server generates an optimal action plan for the user, which includes recommended store visit routes, sale information, coupon information, and more.

[0588] 5. User Notification and Interface

[0589] The terminal notifies the user of the action plan received from the server and displays it in the application. The action plan displayed includes a message such as "Here is a recommended route to the stores for you."

[0590] 6. Real-time feedback

[0591] The device tracks the user's actual behavior in real time and sends the data to the server, which then re-analyzes the data based on the new behavior data and dynamically optimizes the action plan.

[0592] The specific hardware and software used includes:

[0593] Hardware: Smartphone with built-in GPS sensor.

[0594] Software: Dedicated application, server is Django (backend framework), Python, SQLite (database), AI module is TensorFlow or PyTorch.

[0595] Specific examples

[0596] For example, when a user checks in to a shopping mall, AI analyzes their past purchase history and finds that they particularly like a certain fashion brand. Based on this, the user is notified of current sales at the fashion brand and the route to the store. Specific examples of prompts include the following:

[0597] "User profile: {Name: User A, Age: 30s, Gender: Female, Interests: Fashion, Purchase history: Fashion brand products.} Based on this data, please provide the optimal shopping plan for this user."

[0598] This allows users to enjoy shopping efficiently and improves the quality of their experience.

[0599] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0600] Step 1:

[0601] When a user arrives at a facility such as a shopping mall, they launch a dedicated application on their smartphone and tap the check-in button. The device uses a GPS module and Wi-Fi adapter to obtain current location information and sends it to a server. The input is the user's operation upon arriving at the facility, and the output is the transmission of location information to the server.

[0602] Step 2:

[0603] The server identifies the user ID based on the received location information. The server then retrieves the user's profile information and purchase history from the database. The input is location information, and the output is the extracted user ID, profile information, and purchase history.

[0604] Step 3:

[0605] The AI ​​module installed on the server integrates the acquired user profile information and purchase history. Based on this, the AI ​​module analyzes the user's hobbies and preferences. The input is the integrated profile information and purchase history, and the output is the analysis results of the user's hobbies and preferences. TensorFlow and PyTorch are used for the AI ​​module.

[0606] Step 4:

[0607] The server recommends the best products and stores for the user based on the analysis results of the AI ​​module. These recommendations include items and stores that the user is likely to be interested in. The input is the analysis results of the AI ​​module, and the output is a list of recommended products and stores.

[0608] Step 5:

[0609] The server generates a customized action plan based on the information on recommended stores and products. This plan includes the recommended store visit route, sale information, coupon information, etc. The input is a list of recommended stores and products, and the output is the customized action plan.

[0610] Step 6:

[0611] The server sends the generated action plan to the user's device. The device notifies the user of the received action plan and displays it in the application. The input is the action plan, and the output is notification and display to the user. The displayed action plan includes a message such as "Here is a recommended route to the stores for you."

[0612] Step 7:

[0613] The terminal tracks the user's actual behavior in real time and transmits the data to the server. The input is the user's behavior data, and the output is the transmission of the behavior data to the server.

[0614] Step 8:

[0615] The server then performs a second analysis based on the new behavioral data and dynamically optimizes the behavior plan. The input is the new behavioral data, and the output is the optimized behavior plan. This allows for real-time optimization of the plan according to the user's behavior.

[0616] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0617] To implement the present invention, a user-owned terminal, a server, an emotion engine, and an infrastructure for connecting these via communications are required. The processing flow of the entire system will be specifically explained below.

[0618] Overall system processing flow

[0619] 1. A user checks in to a shopping mall

[0620] When a user arrives at the shopping mall, they launch the dedicated application on their device and open the check-in screen. When they tap the check-in button, the device obtains the user's location information. This location information is obtained with high accuracy using GPS and Wi-Fi.

[0621] The device sends the acquired location information to a server, which confirms that the user is located within a specific shopping mall.

[0622] 2. Obtaining user profile information and emotion data

[0623] The server identifies the user ID based on the received check-in information and retrieves the user's profile information and purchase history from the database.

[0624] The user's facial expressions and tone of voice are captured by the device's camera and microphone and sent to the emotion engine.

[0625] The emotion engine analyzes the user's facial expression data and tone of voice to determine their current emotional state.

[0626] 3. Data analysis and product recommendations using AI and emotion engines

[0627] The AI ​​module installed on the server integrates the acquired user profile information, current location information, and emotional data to analyze the user's hobbies and preferences.

[0628] Based on the analysis, the AI ​​and emotion engine will recommend the best products and stores for the user, including items and stores that are likely to interest the user.

[0629] 4. Generate a customized action plan

[0630] The server uses the AI ​​and emotion engine analysis results to generate a personalized action plan for the user, including recommended products and store routes.

[0631] The server obtains sales and coupon information provided by each store in the shopping mall and incorporates this information into an action plan.

[0632] 5. User Notification and Display

[0633] The server transmits the generated action plan to the terminal.

[0634] The device notifies the user of the received action plan and displays the details within the application. For example, it may send a notification to the user saying, "We've created a recommended shopping route for you."

[0635] 6. Real-time feedback and optimization

[0636] The device tracks the user's actions in real time, in particular recording the stores the user actually visits and the route they take.

[0637] The device sends the collected behavioral data to the server.

[0638] The server analyzes the new behavioral data and updated emotional data it receives, and the AI ​​dynamically optimizes the plan based on this. For example, if the user stops at an unexpected store, it will use that information to make new recommendations or adjust the plan.

[0639] Specific examples

[0640] For example, when a user checks in to a shopping mall, the system will recommend fashion brand stores that the user is likely to be interested in based on the user's past purchasing history and current emotional state. It also provides information about products on limited-time sales, allowing users to efficiently search for products that interest them. Furthermore, the system will notify users of events that may interest them (e.g., modeling events). In this way, customized action plans maximize user convenience and enjoyment. By utilizing feedback from the emotion engine, if the user is excited about a particular product, it is also possible to suggest other products in that genre or related events in real time.

[0641] The processing flow will be explained below.

[0642] Step 1:

[0643] When the user arrives at the shopping mall, they launch the dedicated application on their device, open the check-in screen, and tap the check-in button.

[0644] Step 2:

[0645] The device uses GPS sensors and Wi-Fi information to obtain the user's location, which is determined with a high degree of accuracy.

[0646] Step 3:

[0647] The device sends the acquired location information to the server, which confirms that the user is located within a specific shopping mall.

[0648] Step 4:

[0649] The server identifies the user ID based on the check-in information it receives, and retrieves the user's profile information and purchase history from the database.

[0650] Step 5:

[0651] The device captures the user's facial expression data with a camera and their tone of voice with a microphone, and sends the acquired data to the emotion engine.

[0652] Step 6:

[0653] The emotion engine analyzes the user's facial expression data and tone of voice to determine their current emotional state, which can include happiness, excitement, focus, etc.

[0654] Step 7:

[0655] The AI ​​module on the server integrates the acquired user profile information, current location information, and emotional data, and analyzes the user's hobbies and preferences.

[0656] Step 8:

[0657] Based on the results of analysis by AI and an emotion engine, the system recommends the most suitable products and stores for users, including items and stores that are likely to interest them.

[0658] Step 9:

[0659] The server generates a personalized action plan based on the analysis results of the AI ​​and emotion engine, including recommended products and store visit routes.

[0660] Step 10:

[0661] The server incorporates sale information and coupon information provided by each store in the shopping mall into the action plan.

[0662] Step 11:

[0663] The server transmits the generated action plan to the terminal.

[0664] Step 12:

[0665] The device notifies the user of the action plan it has received, such as sending a notification that a "recommended shopping route has been created."

[0666] Step 13:

[0667] When the user taps the notification, the details of the action plan will be displayed on the device, and the user can check the recommended stores, products, and sales information.

[0668] Step 14:

[0669] The device tracks the user's actions in real time, including recording the stores they visit and the route they take.

[0670] Step 15:

[0671] The device transmits the collected behavioral data to a server.

[0672] Step 16:

[0673] The server analyzes the new behavioral data and updated emotional data received. The AI ​​dynamically optimizes the plan based on this data. If the user stops at an unexpected store, the AI ​​will use this information to make new recommendations or adjust the plan.

[0674] As a specific example, if the emotion engine determines that a user is feeling excited in front of a specific store in a shopping mall after checking in, it will recommend products and information about other stores that fit that state of excitement in real time. For example, new products from a fashion brand or related accessory stores will be recommended. Also, if the user is feeling relaxed, information about cafes and rest areas will be provided. This allows users to enjoy a shopping experience optimized for their interests and emotions.

[0675] Example 2

[0676] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0677] Conventional shopping support systems typically recommend products and stores based on a user's location information and purchase history. However, these systems are unable to fully consider a user's momentary emotions or specific hobbies and preferences, making it difficult to provide optimal recommendations and action plans. As a result, users' purchasing experiences are limited, reducing the effectiveness of the system.

[0678] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0679] In this invention, the server includes means for acquiring user profile information and purchase history, means for acquiring the user's facial expression data and voice data and transmitting them to the emotion analysis device, means for the emotion analysis device to analyze the user's facial expression data and voice data to determine the user's emotional state, and means for analyzing the user's hobbies, preferences, and emotional state using AI and recommending products and stores. This makes it possible to provide more accurate product and store recommendations and action plans that take the user's emotional state into consideration.

[0680] "User" means a person who uses the system.

[0681] "Facilities" refer to places that users visit, such as shopping malls and commercial facilities.

[0682] "Device" refers to an electronic device such as a smartphone or tablet owned by a user.

[0683] "Location information" means data about a user's current location obtained using GPS, Wi-Fi, etc.

[0684] "Server" refers to a computer system that stores, processes, and distributes data over a network.

[0685] "Profile information" refers to data about a user's personal information and their interests and preferences.

[0686] "Purchase history" means a record of products and services purchased by a user in the past.

[0687] "Facial expression data" refers to digital data relating to the user's facial expressions captured by the device's camera.

[0688] "Voice data" refers to digital data relating to the user's voice, including the tone and content of the user's voice.

[0689] An "emotion analysis device" refers to a system that analyzes facial expression data and voice data to determine a user's emotional state.

[0690] "AI" refers to the technology that uses artificial intelligence to analyze data and understand and predict users' hobbies, preferences, and emotional states.

[0691] "Hobbies and preferences" refers to items and categories in which a user has interests or preferences.

[0692] An "action plan" refers to a plan that includes recommended routes for users to move around the facility efficiently and comfortably, as well as stores and products of interest.

[0693] "Tracking" means tracking a user's actual behavior and movement path in real time.

[0694] "Optimization" refers to dynamically adjusting and improving existing action plans based on new user behavior data.

[0695] "Sale Information" means information about discounts and special offers offered by stores within the shopping mall.

[0696] "Coupon information" refers to information about discount coupons and special offers that users can use.

[0697] To implement the present invention, the following elements are required: a terminal owned by a user, a server, an emotion analysis device, and infrastructure for connecting these via communications. A detailed description of the system is provided below.

[0698] Hardware and Software Configuration

[0699] Device: A smartphone or tablet is used. These devices are equipped with a GPS sensor, camera, microphone, etc. A dedicated application is also installed and is used by the user to check in and collect emotional data.

[0700] Server: A computer system that stores, processes, and distributes data over a network. It uses a database (SQL Server, MySQL, etc.) to manage user profile information and purchase history. It is equipped with an AI module (TensorFlow, PyTorch, etc.) and integrates and analyzes data.

[0701] Emotion analysis device: A system for analyzing facial expression data and voice data. It uses a facial expression analysis algorithm (such as OpenCV) or a voice emotion analysis module (such as Google Cloud Speech-to-Text).

[0702] Communication infrastructure: Connect devices, servers, and emotion analysis devices using the internet or Wi-Fi.

[0703] Details of data processing and data calculation

[0704] When a user arrives at a shopping mall, they launch the dedicated app and display the check-in screen. When the user taps the check-in button, the device uses GPS and Wi-Fi to obtain location information and sends that information to the server. The server identifies the user ID based on this location information and retrieves profile information and purchase history from the database.

[0705] The device then uses a camera and microphone to capture the user's facial expression and voice data, which are then sent to an emotion analyzer. The emotion analyzer analyzes the user's emotional state in real time and sends the results back to the server. The server then uses an AI module to integrate the profile information, location information, and emotion data to analyze the user's interests and preferences.

[0706] As a result, the server recommends the most suitable products and stores for the user and generates a customized action plan based on this. This action plan includes a recommended route and a list of recommended products. The server also obtains sales and coupon information provided by each store in the shopping mall and integrates it into the action plan.

[0707] The generated action plan is sent to the device and notified to the user. The device receives it and displays it to the user. As the user walks the recommended route, the device tracks their actions and collects new action data. This data is sent to the server, and the AI ​​module dynamically optimizes the plan.

[0708] Specific examples

[0709] For example, when a user checks in to a shopping mall, the system will recommend fashion brand stores that the user is likely to be interested in based on the user's past purchasing history and current emotional state. It will also provide information on products on limited-time sales. This allows users to efficiently search for products that interest them. In addition, the system will notify users of events that may interest them (e.g., modeling events). Using feedback from an emotion analyzer, if the user becomes excited in front of a particular product, it will be possible to suggest other products in that genre or related events in real time.

[0710] Prompt Sentence Examples

[0711] "When a user checks in to a shopping mall, the emotion analyzer analyzes the user's emotional state based on data acquired from the device's camera and microphone. Based on the results, recommend the most suitable products and stores for the user."

[0712] In this way, in order to implement the invention, by appropriately using the above-mentioned hardware and software and coordinating the flow of the entire system, it is possible to provide users with an optimal purchasing experience.

[0713] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0714] Step 1:

[0715] A user checks in to a facility

[0716] When a user arrives at the shopping mall, they launch the dedicated application and open the check-in screen. When they tap the check-in button, the device acquires the user's location information.

[0717] Input: User action (tapping the check-in button)

[0718] Output: Obtain location information (latitude, longitude data)

[0719] Specific operation: The device detects the user's current location using the GPS sensor and Wi-Fi and sends the location information to the server.

[0720] Step 2:

[0721] The device acquires location information and sends it to the server.

[0722] The device uses the acquired location information to confirm that the user is within a specific facility, and sends this location information in JSON format to the server.

[0723] Input: Location information (latitude, longitude)

[0724] Output: Send location information to the server

[0725] Specific operation: The device uses the location information API to format the acquired location data into JSON format and send it to the server to determine the user's current location.

[0726] Step 3:

[0727] The server retrieves the user's profile information and purchase history.

[0728] The server identifies the user ID based on the received location information and retrieves profile information and purchase history from the database.

[0729] Input: Location information, user ID

[0730] Output: Profile information, purchase history

[0731] What happens: The server executes a database query to retrieve the user's profile information and purchase history data, which is used in the subsequent analysis process.

[0732] Step 4:

[0733] The device acquires the user's facial expression and voice data and sends it to the emotion analysis device.

[0734] The user's facial expressions and tone of voice are collected from a camera and microphone, and this data is sent to an emotion analysis device.

[0735] Input: facial expression data, voice data

[0736] Output: Send data to emotion analyzer

[0737] Specific operation: The device captures the user's facial expressions using the camera function and records audio using the microphone. This data is then sent to the emotion analysis device in real time.

[0738] Step 5:

[0739] The emotion analyzer analyzes the user's facial expression and voice data to determine their emotional state.

[0740] The emotion analyzer uses facial expression analysis algorithms and voice emotion analysis modules to analyze the user's emotional state.

[0741] Input: facial expression data, voice data

[0742] Output: Emotional state discrimination result

[0743] Specific operation: The emotion analysis device uses tools such as OpenCV and Google Cloud Speech-to-Text to analyze the transmitted facial expression and voice data to determine the user's current emotional state.

[0744] Step 6:

[0745] The server uses AI to analyze the user's tastes and emotional state and recommend products and stores.

[0746] The AI ​​module installed on the server integrates user profile information, purchase history, and emotional state data to analyze hobbies and preferences.

[0747] Input: Profile information, purchase history, emotional state data

[0748] Output: Product and store recommendation results

[0749] Specific operation: The server's AI module uses TensorFlow and PyTorch to analyze user data and generate a list of the most suitable products and stores.

[0750] Step 7:

[0751] The server generates and sends a customized action plan to the user.

[0752] Based on the recommendations from the AI, the server generates an optimal action plan for the user and sends it to the device.

[0753] Input: Product and store recommendation results

[0754] Output: A customized action plan

[0755] Specific operation: The action plan includes the order of stores to visit, the route, and a list of recommended products. The server also obtains sales and coupon information from the shopping mall and incorporates this information into the action plan.

[0756] Step 8:

[0757] The device notifies and displays the action plan to the user.

[0758] The terminal notifies the user of the action plan received from the server and displays the details within the application.

[0759] Input: Customized Action Plan

[0760] Output: User notification and display

[0761] What it does: The device uses in-app notifications to notify the user of the action plan and display a message such as "We've created a recommended shopping route for you."

[0762] Step 9:

[0763] The device tracks user behavior data and sends it to the server.

[0764] The device tracks users' real-time behavior and collects data on specific stores and routes.

[0765] Input: User behavior data

[0766] Output: Sending tracking data to the server

[0767] Specific operation: The device uses GPS and Bluetooth beacons to record the user's movements and periodically transmits them to the server.

[0768] Step 10:

[0769] The server analyzes new behavioral data and optimizes the plan

[0770] The server dynamically optimizes the behavior plan based on the new behavior data received.

[0771] Input: New behavioral data

[0772] Output: Optimized action plan

[0773] Specific behavior: The server's AI takes into account new behavioral data and emotional state, adjusting and updating its behavior plan in real time based on unplanned behaviors and changing interests.

[0774] (Application example 2)

[0775] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0776] In today's shopping experience, users lack the information and guidance they need to make efficient and satisfying purchases. In particular, it is difficult to provide personalized product and store recommendations based on a user's preferences and current emotional state in brick-and-mortar stores. There is also a need for action plans that effectively incorporate real-time sales and coupon information. It is necessary to solve these issues and provide users with an optimal shopping experience.

[0777] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0778] In this invention, the server includes means for acquiring facial expression and voice data from the device's camera and microphone and sending it to an emotion engine, means for the emotion engine to analyze the facial expression and voice data and determine the user's emotional state, and means for the server to use AI to integrate and analyze the user's hobbies, preferences, and emotional data and recommend products and stores. This makes it possible to recommend appropriate products and stores based on the user's hobbies, preferences, and emotional state, and to provide a more personalized action plan by tracking the user's behavioral data.

[0779] A "user" is a consumer who visits a facility or physical store and is the person who receives an action plan or product recommendations.

[0780] "Facility" refers to a collection of commercial facilities and physical stores that users visit, such as shopping malls and department stores.

[0781] "Checking in" is the act of a user arriving at a facility and identifying their location using a smartphone or other device.

[0782] A "terminal" is a mobile device such as a smartphone or tablet that is owned by the user, and is a device that acquires location information, collects emotional data, notifies and displays action plans, etc.

[0783] "Location information" is data that indicates a user's current location and movement patterns, determined using GPS sensors and Wi-Fi information.

[0784] The "server" is a central management system that manages user profile information, purchase history, and emotional data, and analyzes and processes them using AI.

[0785] "Profile information" is user data that includes personalized information such as the user's age, gender, preferences, and past purchasing history.

[0786] "Purchase history" is a record of products and services purchased by a user in the past.

[0787] "AI" is a system that uses artificial intelligence technology to analyze users' preferences and emotions and recommend the most suitable products and stores.

[0788] The "Emotion Engine" is a system that uses facial expression recognition and voice analysis technologies to analyze and determine the user's emotional state.

[0789] "Hobbies and preferences" are attributes preferred by individual users, such as areas of interest and concern, favorite things and activities, etc.

[0790] "Recommending products and stores" means presenting products that may interest the user and stores that they should visit based on data analyzed by AI and an emotion engine.

[0791] An "action plan" is a schedule that includes a list of stores and products to visit and a travel route, and is generated by the server to enable the user to enjoy shopping efficiently.

[0792] "Notification" refers to a push notification or alert that the device sends to inform the user of the generated action plan or recommended information.

[0793] "Tracking" refers to a device recording a user's behavioral data in real time and sending it to a server.

[0794] "Sale information" is product information about discounts and special prices offered by stores.

[0795] "Coupon information" is information about discount coupons and special offers offered by stores.

[0796] "Optimization" is the process by which AI dynamically adjusts and improves its action plan based on the user's new behavioral data and updated emotional data.

[0797] To implement this invention, a server, a user terminal, an emotion engine, and a communication infrastructure are required. Below, an outline of the program processing of the entire system will be explained in detail.

[0798] First, the user arrives at the facility and checks in by launching a dedicated application on their device. The user's device uses a GPS sensor and Wi-Fi information to obtain highly accurate location information. This location information is sent to the server, which confirms that the user is within a specific facility. After check-in, the server retrieves the user's profile information and purchase history from a database based on the user ID.

[0799] The device then uses a camera and microphone to capture the user's facial expressions and voice. This data is sent to the emotion engine, which uses facial expression recognition and voice analysis technologies to determine the user's current emotional state. The emotion engine utilizes facial expression recognition APIs (e.g., Google Face API) and voice analysis APIs (e.g., Google Speech-to-Text).

[0800] Once the user's emotional state is determined, the server integrates and analyzes the profile information, purchase history, location information, and emotional data. This analysis uses artificial intelligence (AI) technology, such as machine learning models using TensorFlow or PyTorch. The server then recommends products and stores based on the user's preferences and emotional state.

[0801] The server then generates a customized itinerary, which includes information such as the stores and products to visit, sales and coupons, and the best route to travel. The itinerary is then sent to the user's device, which notifies the user and displays it within the application. Notifications are sent using Firebase Cloud Messaging (FCM) and Apple Push Notification Service (APNS).

[0802] When a user begins to take action, the device tracks the user's movements and destinations in real time and sends that data to the server. The server analyzes the newly collected behavioral data and updated emotional data, and the AI ​​optimizes the action plan in real time. This allows the device to quickly respond to any new stores or products the user shows interest in.

[0803] For example, when a user checks in to a fashion brand store within the facility, the system will identify the brand that excites them based on their past purchase history and facial recognition, and notify them of the store's sale information in real time. Also, if the user shows an extended interest in a particular product, it will be possible to present related products and events in that genre.

[0804] To accomplish this process, the following example prompts can be used:

[0805] "Create an application that guides users when they check in to a physical store. Based on their past purchase history and current emotional data, it needs to recommend the most suitable stores and products to users. The application will use the smartphone's GPS and Wi-Fi to obtain the user's location, and facial recognition and speech analysis APIs to collect emotional data. Finally, it will push a generated action plan to the smartphone. Technologies used should include the following: GPS API, Wi-Fi sensor, Google Face API, Google Speech-to-Text, AWS, TensorFlow, and Firebase Cloud Messaging."

[0806] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0807] Step 1:

[0808] The user arrives at the facility and checks in by launching a dedicated application on the terminal.

[0809] Input: Location information of user and device arriving at facility

[0810] Specific operation: The user opens the smartphone application and taps the check-in button to obtain location information. The device uses the GPS sensor and Wi-Fi information to obtain the precise location.

[0811] Output: Obtained user location information

[0812] Step 2:

[0813] The location information acquired by the device is sent to the server.

[0814] Input: User's location

[0815] Specific operation: The device creates a communication request to send the acquired location information packet to the server and sends it to the server API endpoint.

[0816] Output: User location information received by the server

[0817] Step 3:

[0818] The server retrieves profile information and purchase history from a database based on the user ID.

[0819] Input: User's location and user ID

[0820] Specific operation: The server executes a database query using the location information and user ID as keys to obtain the user's profile information and purchase history.

[0821] Output: User profile information and purchase history

[0822] Step 4:

[0823] The device uses a camera and microphone to capture facial and voice data and sends it to the emotion engine.

[0824] Input: Real-time facial and voice data of the user

[0825] Specific operation: The device camera takes a picture of the user's face and records their voice with the microphone. A request is created to send this data to the emotion engine, and then it is sent.

[0826] Output: Facial and vocal data sent to the emotion engine

[0827] Step 5:

[0828] An emotion engine analyzes facial and voice data to determine the user's emotional state.

[0829] Input: facial expression and voice data

[0830] How it works: The emotion engine uses facial expression recognition APIs and voice analysis APIs to analyze data and determine the user's emotional state (excited, happy, calm, etc.).

[0831] Output: Determined user's emotional state

[0832] Step 6:

[0833] The server integrates and analyzes profile information, purchase history, location information, and emotional data to recommend products and stores.

[0834] Input: Profile information, purchase history, location information, emotional data

[0835] Specific operation: The server uses a generative AI model to analyze the user's hobbies and preferences based on input data and generate a list of optimal products and stores.

[0836] Output: A list of recommended products and stores

[0837] Step 7:

[0838] The server generates and transmits a customized action plan.

[0839] Input: Recommended products and store lists, sale information, coupon information

[0840] Specific operation: The server uses an action plan generation algorithm to generate an action plan that includes the order of stores and products to visit, travel routes, and sale and coupon information. The generated action plan is then sent to the user's device.

[0841] Output: A customized action plan

[0842] Step 8:

[0843] The terminal notifies the user of the generated action plan and displays it within the application.

[0844] Input: Generated action plan

[0845] Specific operation: The device will use a push notification service (FCM or APNS) to send a notification of the action plan to the user. The application will then interactively display a detailed action plan.

[0846] Output: A plan of action communicated to the user and displayed

[0847] Step 9:

[0848] The device tracks the user's behavior in real time and transmits the data to a server.

[0849] Input: Real-time user behavior data

[0850] Specific operation: The device records the user's location and visited destinations in real time, and creates and sends a request to send that data to the server.

[0851] Output: User behavior data sent to the server

[0852] Step 10:

[0853] The server analyzes the new behavioral data and updated emotional data to optimize the behavior plan.

[0854] Input: New behavioral data and updated emotion data

[0855] How it works: Based on the data received in real time, the server uses a generative AI model to dynamically adjust and optimize the action plan, recommending new products and stores, changing the route, etc. as needed.

[0856] Output: Optimized action plan

[0857] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0858] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0859] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0860] [Third embodiment]

[0861] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0862] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0863] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0864] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0865] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0866] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0867] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0868] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0869] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0870] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0871] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0872] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0873] To implement the present invention, a user-owned terminal, a server, and an infrastructure for connecting them via communications are required. The processing flow of the entire system will be specifically described below.

[0874] Overall system processing flow

[0875] 1. A user checks in to a shopping mall

[0876] When a user arrives at the shopping mall, they launch the dedicated application on their device and open the check-in screen. When they tap the check-in button, the device obtains the user's location information. This location information is obtained with high accuracy using GPS and Wi-Fi.

[0877] The device sends the acquired location information to a server, which confirms that the user is located within a specific shopping mall.

[0878] 2. Get the user's profile information

[0879] The server identifies the user ID based on the received check-in information and retrieves the user's profile information and purchase history from the database, including data on past purchases and favorite stores.

[0880] 3. AI-based data analysis and product recommendations

[0881] The AI ​​module installed on the server combines the acquired user profile information with current location information to analyze the user's hobbies and preferences, taking into account purchasing history and favorite trends.

[0882] Based on the analysis results, the AI ​​will recommend the best products and stores for the user, including items and stores that the user may be interested in.

[0883] 4. Generate a customized action plan

[0884] The server then generates a personalized action plan based on the AI ​​analysis results, which includes recommended products and store visit routes.

[0885] The server obtains sales and coupon information provided by each store in the shopping mall and incorporates this information into an action plan.

[0886] 5. User Notification and Display

[0887] The server transmits the generated action plan to the terminal.

[0888] The device notifies the user of the received action plan and displays the details within the application. For example, it may send a notification to the user saying, "We've created a recommended shopping route for you."

[0889] 6. Real-time feedback and optimization

[0890] The device tracks the user's actions in real time, in particular recording the stores the user actually visits and the route they take.

[0891] The device sends the collected behavioral data to the server.

[0892] The server analyzes the new behavioral data it receives, and the AI ​​dynamically optimizes the plan. For example, if the user stops at an unexpected store, it will use that information to make new recommendations or adjust the plan.

[0893] Specific examples

[0894] For example, when a user checks in to a shopping mall, the system will recommend fashion brand stores that the user is likely to be particularly interested in based on the user's past purchasing history. It also provides information about limited-time sales, allowing the user to efficiently search for products that interest them. Furthermore, if the user is also interested in a modeling event, they will also be notified of information about that event. In this way, customized action plans maximize the user's convenience and enjoyment.

[0895] The processing flow will be explained below.

[0896] Step 1:

[0897] When the user arrives at the shopping mall, they launch the dedicated application on their device, open the check-in screen, and tap the check-in button.

[0898] Step 2:

[0899] The device uses GPS sensors and Wi-Fi information to obtain the user's location, which is determined with a high degree of accuracy.

[0900] Step 3:

[0901] The device sends the acquired location information to the server, which confirms that the user is located within a specific shopping mall.

[0902] Step 4:

[0903] The server identifies the user ID based on the check-in information it receives, and retrieves the user's profile information and purchase history from the database.

[0904] Step 5:

[0905] An AI module installed on the server integrates the acquired user profile information with the user's current location information, and analyzes the user's hobbies and preferences.

[0906] Step 6:

[0907] Based on the results of the AI ​​analysis, the system recommends the most suitable products and stores for the user, including items and stores that are likely to interest the user.

[0908] Step 7:

[0909] The server generates a personalized action plan based on the AI ​​analysis results, which includes recommended products and store visit routes.

[0910] Step 8:

[0911] The server acquires sale information and coupon information provided by each store in the shopping mall and incorporates this information into an action plan.

[0912] Step 9:

[0913] The server transmits the generated action plan to the terminal.

[0914] Step 10:

[0915] The device notifies the user of the action plan it has received, such as sending a notification that a "recommended shopping route has been created."

[0916] Step 11:

[0917] When the user taps the notification, the details of the action plan will be displayed on the device, and the user can check the recommended stores, products, and sales information.

[0918] Step 12:

[0919] The device tracks the user's actions in real time, including recording the stores they visit and the route they take.

[0920] Step 13:

[0921] The device transmits the collected behavioral data to a server.

[0922] Step 14:

[0923] The server analyzes the new behavioral data it receives, and the AI ​​dynamically optimizes the plan. For example, if the user stops at an unexpected store, it will use that information to make new recommendations or adjust the plan.

[0924] Example 1

[0925] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0926] In recent years, there has been a demand for systems that utilize users' location information and purchase history to provide personalized, optimized shopping plans to improve the shopping experience in facilities. However, existing systems are unable to fully utilize users' preferences and real-time behavioral data, making it difficult to recommend optimal products and stores or provide appropriate shopping plans. Furthermore, due to the lack of dynamic feedback and optimization, these systems are unable to respond immediately to changes in user behavior.

[0927] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0928] In this invention, the server includes a means for a user to check in to a facility, a means for a terminal to acquire location information and send it to the server, a means for the server to acquire user profile information and purchase history, a means for analyzing the user's hobbies and preferences using AI and recommending products and stores, a means for generating and sending an action plan customized for the user, a means for notifying the user of the action plan and displaying it, a means for tracking the user's behavioral data and sending it to the server, and a means for analyzing new behavioral data and optimizing the plan. This makes it possible to recommend products and stores based on the user's hobbies and preferences, and to provide an optimal action plan that reflects real-time behavioral data.

[0929] A "user" is a person who uses the system to receive services or products within a facility.

[0930] A "facility" is a location where a particular service is provided, such as a shopping mall or public building where check-in occurs.

[0931] A "terminal" is a computing device carried by a user, such as a smartphone or tablet.

[0932] "Location information" refers to information that indicates the current location of a user or device, obtained using GPS or Wi-Fi.

[0933] "Server" means a remote computing system that receives User information and processes and analyzes the data.

[0934] "Profile information" refers to data that compiles attribute information and past behavioral history about a user.

[0935] "Purchase history" is a record of products purchased by a user in the past and a history of stores visited.

[0936] "AI" refers to an algorithm that uses artificial intelligence technology to analyze data and understand users' tastes and preferences.

[0937] "Hobbies and preferences" refers to information indicating the types of products and services that a user likes.

[0938] A "customized action plan" is a recommended plan of activities and destinations generated based on a user's individual needs and past behavior.

[0939] "Behavioral data" refers to real-time behavioral information such as a user's current location, travel route, and stores visited.

[0940] "Optimization" refers to the process of optimizing action plans and recommendations based on collected data.

[0941] "Sale information" refers to information about discounts and special offers offered by stores and facilities.

[0942] "Coupon information" refers to information about discount coupons and special coupons that can be used by users.

[0943] To implement the present invention, a user-owned terminal, a server, and an infrastructure for connecting them via communications are required. The specific operation of the system will be described below.

[0944] First, when a user arrives at a shopping mall or other facility, they launch a dedicated application on their smartphone, tablet, or other device and open the check-in screen. When they tap the check-in button, the device uses its GPS sensor and Wi-Fi module to obtain location information. This location information is converted into JSON format and sent over the Internet to a server. The server analyzes the information and records it in a database. This initial process confirms that the user is within a specific facility.

[0945] The server then identifies the user based on their check-in information and retrieves their profile information and purchase history from a relational database, which executes SQL queries to pull the required data. This information is then passed to the AI ​​module.

[0946] The AI ​​module in the server integrates the acquired profile information and purchase history to analyze the user's hobbies and preferences. The AI ​​module evaluates the category and price of the products the user has purchased in the past, as well as the frequency of visits, and recommends the most suitable products and stores for the user. The recommendation results are stored as a data structure.

[0947] Next, the server generates a customized action plan for the user based on the analysis results from the AI. This action plan includes the recommended order in which to visit stores and information on the user's movement within the store. The server also obtains the latest sales and coupon information provided by each store in the shopping mall and incorporates it into the action plan. This information is obtained using an API request. The generated action plan is formatted in JSON format and sent to the device.

[0948] The device analyzes the received action plan and notifies the user. Once the push notification is sent, the user can check the detailed action plan in a dedicated application. The plan includes store location information, the order in which to visit, sale information, and more.

[0949] Additionally, the device tracks the user's current location and travel route in real time and periodically sends the collected behavioral data to the server. This data is securely transmitted using HTTPS. The server then inputs the new behavioral data it receives back into the AI ​​module to dynamically optimize the plan. For example, if the user stops at an unexpected store, that information can be used to make new recommendations or adjust the plan. The new optimized plan is then sent back to the device and the user is notified.

[0950] Specific examples

[0951] For example, when a user checks in to a shopping mall, the system will recommend fashion brand stores that the user is likely to be interested in based on the user's past purchasing history. It also provides information about limited-time sales, allowing the user to efficiently search for products that interest them. Furthermore, if the user is also interested in a modeling event, they will be notified of information about that event. In this way, customized action plans maximize the user's convenience and enjoyment.

[0952] Example prompts for generative AI models

[0953] 1. "Please explain the specific steps a user takes to check in to a shopping mall."

[0954] 2. "Please use AI to analyze data based on user profile information and purchase history."

[0955] 3. "Explain how to dynamically optimize shopping plans based on real-time behavioral data."

[0956] By using these prompt sentences, it is expected that the generative AI model will understand the overall system processing and provide appropriate answers.

[0957] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0958] Step 1:

[0959] After arriving at the shopping mall, the user launches a dedicated application on their smartphone and opens the check-in screen. When they tap the "Check-in" button, the device uses the GPS sensor and Wi-Fi module to obtain their current location information. This location information is converted into JSON format and sent to the server. The input is the user's operation, and the output is location information data in JSON format. Specifically, the device calls an internal API to obtain location information and sends that data to the server.

[0960] Step 2:

[0961] The server executes a database query to identify the user ID based on the received location data. The server retrieves the user's profile information and purchase history from the database. The input is location data and user ID, and the output is profile information and purchase history data. Specifically, the server uses an SQL query to extract the required information from the database.

[0962] Step 3:

[0963] The AI ​​module installed on the server analyzes the user's hobbies and preferences using the acquired profile information and purchase history. The AI ​​module evaluates past purchase history and favorite trends and calculates the best products and stores for the user. The input is profile information and purchase history data, and the output is a list of recommended products. Specifically, the AI ​​algorithm analyzes this data and generates the analysis results.

[0964] Step 4:

[0965] The server generates an action plan customized for the user based on the analysis results from the AI ​​module. This plan includes the recommended order in which to visit stores and information on movement within the store. The server also obtains the latest sales and coupon information provided by stores in the shopping mall via API requests and incorporates this into the action plan. The input is the analysis results and store information, and the output is the action plan. Specifically, the server runs the plan generation algorithm, collects the necessary store information, and completes the action plan.

[0966] Step 5:

[0967] The server sends the generated action plan in JSON format to the device. The device analyzes the received action plan and sends a push notification saying, "We've created a recommended shopping route for you." The user can check the detailed action plan in a dedicated application. The input is the action plan data, and the output is a push notification and display information for the user. Specifically, the device analyzes the received data and notifies the user of the information.

[0968] Step 6:

[0969] The device tracks the user's current location and travel route in real time and periodically sends this behavioral data to the server. For example, it sends the data securely using HTTPS. The server inputs the new behavioral data it receives into the AI ​​module and dynamically optimizes the plan. If the user stops at an unexpected store, that information is used to make new recommendations and adjust the plan. The input is the new behavioral data, and the output is an optimized plan. Specifically, the AI ​​recalculates the plan based on the real-time data and sends the updated plan to the device.

[0970] Through these steps, the system leverages users' real-time behavioral data to provide a personalized shopping experience.

[0971] (Application example 1)

[0972] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0973] Modern shopping malls have many stores and products, making it difficult for users to shop efficiently and comfortably. It takes time and effort, especially, for users to find products and stores that interest them. Furthermore, if sales and coupon information is not provided appropriately, users may lose motivation to purchase. Furthermore, if plans are not optimized in real time according to user behavior, the user experience may be impaired.

[0974] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0975] In this invention, the server includes means for generating prompts recommended by AI based on the user's purchase history and profile information, means for notifying and optimizing an action plan including recommended stores and promotion information in real time, means for the user to check in to a facility, means for the terminal to acquire location information and send it to the server, means for the server to acquire the user's profile information and purchase history, means for the server to analyze the user's hobbies and preferences using AI and recommend products and stores, means for the server to generate and send an action plan customized for the user, means for the terminal to notify and display the action plan to the user, means for the terminal to track the user's behavioral data and send it to the server, and means for the server to analyze new behavioral data and optimize the plan. This allows users to efficiently find products and stores of their interest, and by applying sale information and coupon information, purchasing motivation is increased, and the user experience is optimized through real-time feedback.

[0976] "Checking in" refers to the act of notifying a server of a user's location information via a dedicated application when the user arrives at a facility.

[0977] A "terminal" is a device such as a smartphone or tablet operated by a user, and is a device for obtaining location information, sending notifications, displaying action plans, and the like.

[0978] "Location information" refers to geographical location data obtained using GPS or Wi-Fi, and is information used to identify a user's current location.

[0979] A "server" is a computer system that manages user profile information and purchase history, and recommends products and stores to users.

[0980] "Profile information" refers to integrated data such as a user's attributes, past purchasing history, interests and preferences, and generally refers to information about an individual user.

[0981] "Purchase history" is a record of products and services purchased by a user in the past, and is data used to analyze the user's purchasing trends.

[0982] "AI (artificial intelligence)" is a general term for algorithms and technologies that use user profile information and purchasing history to analyze a user's hobbies and preferences and recommend the most suitable products and stores.

[0983] "Hobbies and preferences" refers to the interests and concerns a user has toward specific products and services, and is an indicator for predicting user behavior and preferences.

[0984] An "action plan" is a plan generated by the server that includes a route to visit, recommended stores, product information, and sale information for a user.

[0985] "Notification" refers to a message or alert sent from the server to the user's device, and is a means of conveying action plans and recommended information to the user.

[0986] "Tracking" is the process of following a user's behavior in real time and recording the data obtained.

[0987] "Plan optimization" is the process of analyzing new user behavior data and updating the current behavior plan to an optimal form.

[0988] "Recommendation" is the act of AI selecting products and stores that a user might be interested in based on the user's profile information and purchasing history.

[0989] A "prompt" is text data that AI uses to give instructions and recommendations to users to show them the best course of action or choice.

[0990] "Real-time" means processing data instantly in response to changes in user behavior and the environment, and providing appropriate information immediately.

[0991] To implement this invention, a user-owned terminal, a server, and an infrastructure for connecting these via communications are required. The processing of the entire system will be described below.

[0992] System configuration

[0993] The system includes a terminal application for users to check in, a server that receives and processes data, and a tracking function for acquiring user behavior data. The terminal uses a GPS module and Wi-Fi adapter to obtain highly accurate location information and send it to the server. The server has a database containing user profile information and purchase history, an AI module, and a notification system.

[0994] Program Implementation

[0995] 1. User checks in

[0996] When a user arrives at a shopping mall or other facility, they launch a dedicated application on their smartphone and check in. When they tap the check-in button, the device uses GPS and Wi-Fi to obtain their current location information and sends it to a server.

[0997] 2. The server processes the data

[0998] The server identifies the user ID based on the received location information. It also retrieves the user's profile information and purchase history from the database. Based on this information, the AI ​​module analyzes the user's hobbies and preferences.

[0999] 3. Recommendations by AI module

[1000] The AI ​​module uses a user's profile information and purchase history to recommend products and stores that the user might be interested in. It does this using AI frameworks such as TensorFlow and PyTorch.

[1001] 4. Generate a customized action plan

[1002] Based on the analysis results of the AI ​​module, the server generates an optimal action plan for the user, which includes recommended store visit routes, sale information, coupon information, and more.

[1003] 5. User Notification and Interface

[1004] The terminal notifies the user of the action plan received from the server and displays it in the application. The action plan displayed includes a message such as "Here is a recommended route to the stores for you."

[1005] 6. Real-time feedback

[1006] The device tracks the user's actual behavior in real time and sends the data to the server, which then re-analyzes the data based on the new behavior data and dynamically optimizes the action plan.

[1007] The specific hardware and software used includes:

[1008] Hardware: Smartphone with built-in GPS sensor.

[1009] Software: Dedicated application, server is Django (backend framework), Python, SQLite (database), AI module is TensorFlow or PyTorch.

[1010] Specific examples

[1011] For example, when a user checks in to a shopping mall, AI analyzes their past purchase history and finds that they particularly like a certain fashion brand. Based on this, the user is notified of current sales at the fashion brand and the route to the store. Specific examples of prompts include the following:

[1012] "User profile: {Name: User A, Age: 30s, Gender: Female, Interests: Fashion, Purchase history: Fashion brand products.} Based on this data, please provide the optimal shopping plan for this user."

[1013] This allows users to enjoy shopping efficiently and improves the quality of their experience.

[1014] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1015] Step 1:

[1016] When a user arrives at a facility such as a shopping mall, they launch a dedicated application on their smartphone and tap the check-in button. The device uses a GPS module and Wi-Fi adapter to obtain current location information and sends it to a server. The input is the user's operation upon arriving at the facility, and the output is the transmission of location information to the server.

[1017] Step 2:

[1018] The server identifies the user ID based on the received location information. The server then retrieves the user's profile information and purchase history from the database. The input is location information, and the output is the extracted user ID, profile information, and purchase history.

[1019] Step 3:

[1020] The AI ​​module installed on the server integrates the acquired user profile information and purchase history. Based on this, the AI ​​module analyzes the user's hobbies and preferences. The input is the integrated profile information and purchase history, and the output is the analysis results of the user's hobbies and preferences. TensorFlow and PyTorch are used for the AI ​​module.

[1021] Step 4:

[1022] The server recommends the best products and stores for the user based on the analysis results of the AI ​​module. These recommendations include items and stores that the user is likely to be interested in. The input is the analysis results of the AI ​​module, and the output is a list of recommended products and stores.

[1023] Step 5:

[1024] The server generates a customized action plan based on the information on recommended stores and products. This plan includes the recommended store visit route, sale information, coupon information, etc. The input is a list of recommended stores and products, and the output is the customized action plan.

[1025] Step 6:

[1026] The server sends the generated action plan to the user's device. The device notifies the user of the received action plan and displays it in the application. The input is the action plan, and the output is notification and display to the user. The displayed action plan includes a message such as "Here is a recommended route to the stores for you."

[1027] Step 7:

[1028] The terminal tracks the user's actual behavior in real time and transmits the data to the server. The input is the user's behavior data, and the output is the transmission of the behavior data to the server.

[1029] Step 8:

[1030] The server then performs a second analysis based on the new behavioral data and dynamically optimizes the behavior plan. The input is the new behavioral data, and the output is the optimized behavior plan. This allows for real-time optimization of the plan according to the user's behavior.

[1031] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1032] To implement the present invention, a user-owned terminal, a server, an emotion engine, and an infrastructure for connecting these via communications are required. The processing flow of the entire system will be specifically explained below.

[1033] Overall system processing flow

[1034] 1. A user checks in to a shopping mall

[1035] When a user arrives at the shopping mall, they launch the dedicated application on their device and open the check-in screen. When they tap the check-in button, the device obtains the user's location information. This location information is obtained with high accuracy using GPS and Wi-Fi.

[1036] The device sends the acquired location information to a server, which confirms that the user is located within a specific shopping mall.

[1037] 2. Obtaining user profile information and emotion data

[1038] The server identifies the user ID based on the received check-in information and retrieves the user's profile information and purchase history from the database.

[1039] The user's facial expressions and tone of voice are captured by the device's camera and microphone and sent to the emotion engine.

[1040] The emotion engine analyzes the user's facial expression data and tone of voice to determine their current emotional state.

[1041] 3. Data analysis and product recommendations using AI and emotion engines

[1042] The AI ​​module installed on the server integrates the acquired user profile information, current location information, and emotional data to analyze the user's hobbies and preferences.

[1043] Based on the analysis, the AI ​​and emotion engine will recommend the best products and stores for the user, including items and stores that are likely to interest the user.

[1044] 4. Generate a customized action plan

[1045] The server uses the AI ​​and emotion engine analysis results to generate a personalized action plan for the user, including recommended products and store routes.

[1046] The server obtains sales and coupon information provided by each store in the shopping mall and incorporates this information into an action plan.

[1047] 5. User Notification and Display

[1048] The server transmits the generated action plan to the terminal.

[1049] The device notifies the user of the received action plan and displays the details within the application. For example, it may send a notification to the user saying, "We've created a recommended shopping route for you."

[1050] 6. Real-time feedback and optimization

[1051] The device tracks the user's actions in real time, in particular recording the stores the user actually visits and the route they take.

[1052] The device sends the collected behavioral data to the server.

[1053] The server analyzes the new behavioral data and updated emotional data it receives, and the AI ​​dynamically optimizes the plan based on this. For example, if the user stops at an unexpected store, it will use that information to make new recommendations or adjust the plan.

[1054] Specific examples

[1055] For example, when a user checks in to a shopping mall, the system will recommend fashion brand stores that the user is likely to be interested in based on the user's past purchasing history and current emotional state. It also provides information about products on limited-time sales, allowing users to efficiently search for products that interest them. Furthermore, the system will notify users of events that may interest them (e.g., modeling events). In this way, customized action plans maximize user convenience and enjoyment. By utilizing feedback from the emotion engine, if the user is excited about a particular product, it is also possible to suggest other products in that genre or related events in real time.

[1056] The processing flow will be explained below.

[1057] Step 1:

[1058] When the user arrives at the shopping mall, they launch the dedicated application on their device, open the check-in screen, and tap the check-in button.

[1059] Step 2:

[1060] The device uses GPS sensors and Wi-Fi information to obtain the user's location, which is determined with a high degree of accuracy.

[1061] Step 3:

[1062] The device sends the acquired location information to the server, which confirms that the user is located within a specific shopping mall.

[1063] Step 4:

[1064] The server identifies the user ID based on the check-in information it receives, and retrieves the user's profile information and purchase history from the database.

[1065] Step 5:

[1066] The device captures the user's facial expression data with a camera and their tone of voice with a microphone, and sends the acquired data to the emotion engine.

[1067] Step 6:

[1068] The emotion engine analyzes the user's facial expression data and tone of voice to determine their current emotional state, which can include happiness, excitement, focus, etc.

[1069] Step 7:

[1070] The AI ​​module on the server integrates the acquired user profile information, current location information, and emotional data, and analyzes the user's hobbies and preferences.

[1071] Step 8:

[1072] Based on the results of analysis by AI and an emotion engine, the system recommends the most suitable products and stores for users, including items and stores that are likely to interest them.

[1073] Step 9:

[1074] The server generates a personalized action plan based on the analysis results of the AI ​​and emotion engine, including recommended products and store visit routes.

[1075] Step 10:

[1076] The server incorporates sale information and coupon information provided by each store in the shopping mall into the action plan.

[1077] Step 11:

[1078] The server transmits the generated action plan to the terminal.

[1079] Step 12:

[1080] The device notifies the user of the action plan it has received, such as sending a notification that a "recommended shopping route has been created."

[1081] Step 13:

[1082] When the user taps the notification, the details of the action plan will be displayed on the device, and the user can check the recommended stores, products, and sales information.

[1083] Step 14:

[1084] The device tracks the user's actions in real time, including recording the stores they visit and the route they take.

[1085] Step 15:

[1086] The device transmits the collected behavioral data to a server.

[1087] Step 16:

[1088] The server analyzes the new behavioral data and updated emotional data received. The AI ​​dynamically optimizes the plan based on this data. If the user stops at an unexpected store, the AI ​​will use this information to make new recommendations or adjust the plan.

[1089] As a specific example, if the emotion engine determines that a user is feeling excited in front of a specific store in a shopping mall after checking in, it will recommend products and information about other stores that fit that state of excitement in real time. For example, new products from a fashion brand or related accessory stores will be recommended. Also, if the user is feeling relaxed, information about cafes and rest areas will be provided. This allows users to enjoy a shopping experience optimized for their interests and emotions.

[1090] Example 2

[1091] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1092] Conventional shopping support systems typically recommend products and stores based on a user's location information and purchase history. However, these systems are unable to fully consider a user's momentary emotions or specific hobbies and preferences, making it difficult to provide optimal recommendations and action plans. As a result, users' purchasing experiences are limited, reducing the effectiveness of the system.

[1093] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1094] In this invention, the server includes means for acquiring user profile information and purchase history, means for acquiring the user's facial expression data and voice data and transmitting them to the emotion analysis device, means for the emotion analysis device to analyze the user's facial expression data and voice data to determine the user's emotional state, and means for analyzing the user's hobbies, preferences, and emotional state using AI and recommending products and stores. This makes it possible to provide more accurate product and store recommendations and action plans that take the user's emotional state into consideration.

[1095] "User" means a person who uses the system.

[1096] "Facilities" refer to places that users visit, such as shopping malls and commercial facilities.

[1097] "Device" refers to an electronic device such as a smartphone or tablet owned by a user.

[1098] "Location information" means data about a user's current location obtained using GPS, Wi-Fi, etc.

[1099] "Server" refers to a computer system that stores, processes, and distributes data over a network.

[1100] "Profile information" refers to data about a user's personal information and their interests and preferences.

[1101] "Purchase history" means a record of products and services purchased by a user in the past.

[1102] "Facial expression data" refers to digital data relating to the user's facial expressions captured by the device's camera.

[1103] "Voice data" refers to digital data relating to the user's voice, including the tone and content of the user's voice.

[1104] An "emotion analysis device" refers to a system that analyzes facial expression data and voice data to determine a user's emotional state.

[1105] "AI" refers to the technology that uses artificial intelligence to analyze data and understand and predict users' hobbies, preferences, and emotional states.

[1106] "Hobbies and preferences" refers to items and categories in which a user has interests or preferences.

[1107] An "action plan" refers to a plan that includes recommended routes for users to move around the facility efficiently and comfortably, as well as stores and products of interest.

[1108] "Tracking" means tracking a user's actual behavior and movement path in real time.

[1109] "Optimization" refers to dynamically adjusting and improving existing action plans based on new user behavior data.

[1110] "Sale Information" means information about discounts and special offers offered by stores within the shopping mall.

[1111] "Coupon information" refers to information about discount coupons and special offers that users can use.

[1112] To implement the present invention, the following elements are required: a terminal owned by a user, a server, an emotion analysis device, and infrastructure for connecting these via communications. A detailed description of the system is provided below.

[1113] Hardware and Software Configuration

[1114] Device: A smartphone or tablet is used. These devices are equipped with a GPS sensor, camera, microphone, etc. A dedicated application is also installed and is used by the user to check in and collect emotional data.

[1115] Server: A computer system that stores, processes, and distributes data over a network. It uses a database (SQL Server, MySQL, etc.) to manage user profile information and purchase history. It is equipped with an AI module (TensorFlow, PyTorch, etc.) and integrates and analyzes data.

[1116] Emotion analysis device: A system for analyzing facial expression data and voice data. It uses a facial expression analysis algorithm (such as OpenCV) or a voice emotion analysis module (such as Google Cloud Speech-to-Text).

[1117] Communication infrastructure: Connect devices, servers, and emotion analysis devices using the internet or Wi-Fi.

[1118] Details of data processing and data calculation

[1119] When a user arrives at a shopping mall, they launch the dedicated app and display the check-in screen. When the user taps the check-in button, the device uses GPS and Wi-Fi to obtain location information and sends that information to the server. The server identifies the user ID based on this location information and retrieves profile information and purchase history from the database.

[1120] The device then uses a camera and microphone to capture the user's facial expression and voice data, which are then sent to an emotion analyzer. The emotion analyzer analyzes the user's emotional state in real time and sends the results back to the server. The server then uses an AI module to integrate the profile information, location information, and emotion data to analyze the user's interests and preferences.

[1121] As a result, the server recommends the most suitable products and stores for the user and generates a customized action plan based on this. This action plan includes a recommended route and a list of recommended products. The server also obtains sales and coupon information provided by each store in the shopping mall and integrates it into the action plan.

[1122] The generated action plan is sent to the device and notified to the user. The device receives it and displays it to the user. As the user walks the recommended route, the device tracks their actions and collects new action data. This data is sent to the server, and the AI ​​module dynamically optimizes the plan.

[1123] Specific examples

[1124] For example, when a user checks in to a shopping mall, the system will recommend fashion brand stores that the user is likely to be interested in based on the user's past purchasing history and current emotional state. It will also provide information on products on limited-time sales. This allows users to efficiently search for products that interest them. In addition, the system will notify users of events that may interest them (e.g., modeling events). Using feedback from an emotion analyzer, if the user becomes excited in front of a particular product, it will be possible to suggest other products in that genre or related events in real time.

[1125] Prompt Sentence Examples

[1126] "When a user checks in to a shopping mall, the emotion analyzer analyzes the user's emotional state based on data acquired from the device's camera and microphone. Based on the results, recommend the most suitable products and stores for the user."

[1127] In this way, in order to implement the invention, by appropriately using the above-mentioned hardware and software and coordinating the flow of the entire system, it is possible to provide users with an optimal purchasing experience.

[1128] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1129] Step 1:

[1130] A user checks in to a facility

[1131] When a user arrives at the shopping mall, they launch the dedicated application and open the check-in screen. When they tap the check-in button, the device acquires the user's location information.

[1132] Input: User action (tapping the check-in button)

[1133] Output: Obtain location information (latitude, longitude data)

[1134] Specific operation: The device detects the user's current location using the GPS sensor and Wi-Fi and sends the location information to the server.

[1135] Step 2:

[1136] The device acquires location information and sends it to the server.

[1137] The device uses the acquired location information to confirm that the user is within a specific facility, and sends this location information in JSON format to the server.

[1138] Input: Location information (latitude, longitude)

[1139] Output: Send location information to the server

[1140] Specific operation: The device uses the location information API to format the acquired location data into JSON format and send it to the server to determine the user's current location.

[1141] Step 3:

[1142] The server retrieves the user's profile information and purchase history.

[1143] The server identifies the user ID based on the received location information and retrieves profile information and purchase history from the database.

[1144] Input: Location information, user ID

[1145] Output: Profile information, purchase history

[1146] What happens: The server executes a database query to retrieve the user's profile information and purchase history data, which is used in the subsequent analysis process.

[1147] Step 4:

[1148] The device acquires the user's facial expression and voice data and sends it to the emotion analysis device.

[1149] The user's facial expressions and tone of voice are collected from a camera and microphone, and this data is sent to an emotion analysis device.

[1150] Input: facial expression data, voice data

[1151] Output: Send data to emotion analyzer

[1152] Specific operation: The device captures the user's facial expressions using the camera function and records audio using the microphone. This data is then sent to the emotion analysis device in real time.

[1153] Step 5:

[1154] The emotion analyzer analyzes the user's facial expression and voice data to determine their emotional state.

[1155] The emotion analyzer uses facial expression analysis algorithms and voice emotion analysis modules to analyze the user's emotional state.

[1156] Input: facial expression data, voice data

[1157] Output: Emotional state discrimination result

[1158] Specific operation: The emotion analysis device uses tools such as OpenCV and Google Cloud Speech-to-Text to analyze the transmitted facial expression and voice data to determine the user's current emotional state.

[1159] Step 6:

[1160] The server uses AI to analyze the user's tastes and emotional state and recommend products and stores.

[1161] The AI ​​module installed on the server integrates user profile information, purchase history, and emotional state data to analyze hobbies and preferences.

[1162] Input: Profile information, purchase history, emotional state data

[1163] Output: Product and store recommendation results

[1164] Specific operation: The server's AI module uses TensorFlow and PyTorch to analyze user data and generate a list of the most suitable products and stores.

[1165] Step 7:

[1166] The server generates and sends a customized action plan to the user.

[1167] Based on the recommendations from the AI, the server generates an optimal action plan for the user and sends it to the device.

[1168] Input: Product and store recommendation results

[1169] Output: A customized action plan

[1170] Specific operation: The action plan includes the order of stores to visit, the route, and a list of recommended products. The server also obtains sales and coupon information from the shopping mall and incorporates this information into the action plan.

[1171] Step 8:

[1172] The device notifies and displays the action plan to the user.

[1173] The terminal notifies the user of the action plan received from the server and displays the details within the application.

[1174] Input: Customized Action Plan

[1175] Output: User notification and display

[1176] What it does: The device uses in-app notifications to notify the user of the action plan and display a message such as "We've created a recommended shopping route for you."

[1177] Step 9:

[1178] The device tracks user behavior data and sends it to the server.

[1179] The device tracks users' real-time behavior and collects data on specific stores and routes.

[1180] Input: User behavior data

[1181] Output: Sending tracking data to the server

[1182] Specific operation: The device uses GPS and Bluetooth beacons to record the user's movements and periodically transmits them to the server.

[1183] Step 10:

[1184] The server analyzes new behavioral data and optimizes the plan

[1185] The server dynamically optimizes the behavior plan based on the new behavior data received.

[1186] Input: New behavioral data

[1187] Output: Optimized action plan

[1188] Specific behavior: The server's AI takes into account new behavioral data and emotional state, adjusting and updating its behavior plan in real time based on unplanned behaviors and changing interests.

[1189] (Application example 2)

[1190] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1191] In today's shopping experience, users lack the information and guidance they need to make efficient and satisfying purchases. In particular, it is difficult to provide personalized product and store recommendations based on a user's preferences and current emotional state in brick-and-mortar stores. There is also a need for action plans that effectively incorporate real-time sales and coupon information. It is necessary to solve these issues and provide users with an optimal shopping experience.

[1192] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1193] In this invention, the server includes means for acquiring facial expression and voice data from the device's camera and microphone and sending it to an emotion engine, means for the emotion engine to analyze the facial expression and voice data and determine the user's emotional state, and means for the server to use AI to integrate and analyze the user's hobbies, preferences, and emotional data and recommend products and stores. This makes it possible to recommend appropriate products and stores based on the user's hobbies, preferences, and emotional state, and to provide a more personalized action plan by tracking the user's behavioral data.

[1194] A "user" is a consumer who visits a facility or physical store and is the person who receives an action plan or product recommendations.

[1195] "Facility" refers to a collection of commercial facilities and physical stores that users visit, such as shopping malls and department stores.

[1196] "Checking in" is the act of a user arriving at a facility and identifying their location using a smartphone or other device.

[1197] A "terminal" is a mobile device such as a smartphone or tablet that is owned by the user, and is a device that acquires location information, collects emotional data, notifies and displays action plans, etc.

[1198] "Location information" is data that indicates a user's current location and movement patterns, determined using GPS sensors and Wi-Fi information.

[1199] The "server" is a central management system that manages user profile information, purchase history, and emotional data, and analyzes and processes them using AI.

[1200] "Profile information" is user data that includes personalized information such as the user's age, gender, preferences, and past purchasing history.

[1201] "Purchase history" is a record of products and services purchased by a user in the past.

[1202] "AI" is a system that uses artificial intelligence technology to analyze users' preferences and emotions and recommend the most suitable products and stores.

[1203] The "Emotion Engine" is a system that uses facial expression recognition and voice analysis technologies to analyze and determine the user's emotional state.

[1204] "Hobbies and preferences" are attributes preferred by individual users, such as areas of interest and concern, favorite things and activities, etc.

[1205] "Recommending products and stores" means presenting products that may interest the user and stores that they should visit based on data analyzed by AI and an emotion engine.

[1206] An "action plan" is a schedule that includes a list of stores and products to visit and a travel route, and is generated by the server to enable the user to enjoy shopping efficiently.

[1207] "Notification" refers to a push notification or alert that the device sends to inform the user of the generated action plan or recommended information.

[1208] "Tracking" refers to a device recording a user's behavioral data in real time and sending it to a server.

[1209] "Sale information" is product information about discounts and special prices offered by stores.

[1210] "Coupon information" is information about discount coupons and special offers offered by stores.

[1211] "Optimization" is the process by which AI dynamically adjusts and improves its action plan based on the user's new behavioral data and updated emotional data.

[1212] To implement this invention, a server, a user terminal, an emotion engine, and a communication infrastructure are required. Below, an outline of the program processing of the entire system will be explained in detail.

[1213] First, the user arrives at the facility and checks in by launching a dedicated application on their device. The user's device uses a GPS sensor and Wi-Fi information to obtain highly accurate location information. This location information is sent to the server, which confirms that the user is within a specific facility. After check-in, the server retrieves the user's profile information and purchase history from a database based on the user ID.

[1214] The device then uses a camera and microphone to capture the user's facial expressions and voice. This data is sent to the emotion engine, which uses facial expression recognition and voice analysis technologies to determine the user's current emotional state. The emotion engine utilizes facial expression recognition APIs (e.g., Google Face API) and voice analysis APIs (e.g., Google Speech-to-Text).

[1215] Once the user's emotional state is determined, the server integrates and analyzes the profile information, purchase history, location information, and emotional data. This analysis uses artificial intelligence (AI) technology, such as machine learning models using TensorFlow or PyTorch. The server then recommends products and stores based on the user's preferences and emotional state.

[1216] The server then generates a customized itinerary, which includes information such as the stores and products to visit, sales and coupons, and the best route to travel. The itinerary is then sent to the user's device, which notifies the user and displays it within the application. Notifications are sent using Firebase Cloud Messaging (FCM) and Apple Push Notification Service (APNS).

[1217] When a user begins to take action, the device tracks the user's movements and destinations in real time and sends that data to the server. The server analyzes the newly collected behavioral data and updated emotional data, and the AI ​​optimizes the action plan in real time. This allows the device to quickly respond to any new stores or products the user shows interest in.

[1218] For example, when a user checks in to a fashion brand store within the facility, the system will identify the brand that excites them based on their past purchase history and facial recognition, and notify them of the store's sale information in real time. Also, if the user shows an extended interest in a particular product, it will be possible to present related products and events in that genre.

[1219] To accomplish this process, the following example prompts can be used:

[1220] "Create an application that guides users when they check in to a physical store. Based on their past purchase history and current emotional data, it needs to recommend the most suitable stores and products to users. The application will use the smartphone's GPS and Wi-Fi to obtain the user's location, and facial recognition and speech analysis APIs to collect emotional data. Finally, it will push a generated action plan to the smartphone. Technologies used should include the following: GPS API, Wi-Fi sensor, Google Face API, Google Speech-to-Text, AWS, TensorFlow, and Firebase Cloud Messaging."

[1221] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1222] Step 1:

[1223] The user arrives at the facility and checks in by launching a dedicated application on the terminal.

[1224] Input: Location information of user and device arriving at facility

[1225] Specific operation: The user opens the smartphone application and taps the check-in button to obtain location information. The device uses the GPS sensor and Wi-Fi information to obtain the precise location.

[1226] Output: Obtained user location information

[1227] Step 2:

[1228] The location information acquired by the device is sent to the server.

[1229] Input: User's location

[1230] Specific operation: The device creates a communication request to send the acquired location information packet to the server and sends it to the server API endpoint.

[1231] Output: User location information received by the server

[1232] Step 3:

[1233] The server retrieves profile information and purchase history from a database based on the user ID.

[1234] Input: User's location and user ID

[1235] Specific operation: The server executes a database query using the location information and user ID as keys to obtain the user's profile information and purchase history.

[1236] Output: User profile information and purchase history

[1237] Step 4:

[1238] The device uses a camera and microphone to capture facial and voice data and sends it to the emotion engine.

[1239] Input: Real-time facial and voice data of the user

[1240] Specific operation: The device camera takes a picture of the user's face and records their voice with the microphone. A request is created to send this data to the emotion engine, and then it is sent.

[1241] Output: Facial and vocal data sent to the emotion engine

[1242] Step 5:

[1243] An emotion engine analyzes facial and voice data to determine the user's emotional state.

[1244] Input: facial expression and voice data

[1245] How it works: The emotion engine uses facial expression recognition APIs and voice analysis APIs to analyze data and determine the user's emotional state (excited, happy, calm, etc.).

[1246] Output: Determined user's emotional state

[1247] Step 6:

[1248] The server integrates and analyzes profile information, purchase history, location information, and emotional data to recommend products and stores.

[1249] Input: Profile information, purchase history, location information, emotional data

[1250] Specific operation: The server uses a generative AI model to analyze the user's hobbies and preferences based on input data and generate a list of optimal products and stores.

[1251] Output: A list of recommended products and stores

[1252] Step 7:

[1253] The server generates and transmits a customized action plan.

[1254] Input: Recommended products and store lists, sale information, coupon information

[1255] Specific operation: The server uses an action plan generation algorithm to generate an action plan that includes the order of stores and products to visit, travel routes, and sale and coupon information. The generated action plan is then sent to the user's device.

[1256] Output: A customized action plan

[1257] Step 8:

[1258] The terminal notifies the user of the generated action plan and displays it within the application.

[1259] Input: Generated action plan

[1260] Specific operation: The device will use a push notification service (FCM or APNS) to send a notification of the action plan to the user. The application will then interactively display a detailed action plan.

[1261] Output: A plan of action communicated to the user and displayed

[1262] Step 9:

[1263] The device tracks the user's behavior in real time and transmits the data to a server.

[1264] Input: Real-time user behavior data

[1265] Specific operation: The device records the user's location and visited destinations in real time, and creates and sends a request to send that data to the server.

[1266] Output: User behavior data sent to the server

[1267] Step 10:

[1268] The server analyzes the new behavioral data and updated emotional data to optimize the behavior plan.

[1269] Input: New behavioral data and updated emotion data

[1270] How it works: Based on the data received in real time, the server uses a generative AI model to dynamically adjust and optimize the action plan, recommending new products and stores, changing the route, etc. as needed.

[1271] Output: Optimized action plan

[1272] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1273] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1274] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1275] [Fourth embodiment]

[1276] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1277] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1278] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1279] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1280] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1281] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1282] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1283] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1284] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1285] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1286] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1287] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1288] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1289] To implement the present invention, a user-owned terminal, a server, and an infrastructure for connecting them via communications are required. The processing flow of the entire system will be specifically described below.

[1290] Overall system processing flow

[1291] 1. A user checks in to a shopping mall

[1292] When a user arrives at the shopping mall, they launch the dedicated application on their device and open the check-in screen. When they tap the check-in button, the device obtains the user's location information. This location information is obtained with high accuracy using GPS and Wi-Fi.

[1293] The device sends the acquired location information to a server, which confirms that the user is located within a specific shopping mall.

[1294] 2. Get the user's profile information

[1295] The server identifies the user ID based on the received check-in information and retrieves the user's profile information and purchase history from the database, including data on past purchases and favorite stores.

[1296] 3. AI-based data analysis and product recommendations

[1297] The AI ​​module installed on the server combines the acquired user profile information with current location information to analyze the user's hobbies and preferences, taking into account purchasing history and favorite trends.

[1298] Based on the analysis results, the AI ​​will recommend the best products and stores for the user, including items and stores that the user may be interested in.

[1299] 4. Generate a customized action plan

[1300] The server then generates a personalized action plan based on the AI ​​analysis results, which includes recommended products and store visit routes.

[1301] The server obtains sales and coupon information provided by each store in the shopping mall and incorporates this information into an action plan.

[1302] 5. User Notification and Display

[1303] The server transmits the generated action plan to the terminal.

[1304] The device notifies the user of the received action plan and displays the details within the application. For example, it may send a notification to the user saying, "We've created a recommended shopping route for you."

[1305] 6. Real-time feedback and optimization

[1306] The device tracks the user's actions in real time, in particular recording the stores the user actually visits and the route they take.

[1307] The device sends the collected behavioral data to the server.

[1308] The server analyzes the new behavioral data it receives, and the AI ​​dynamically optimizes the plan. For example, if the user stops at an unexpected store, it will use that information to make new recommendations or adjust the plan.

[1309] Specific examples

[1310] For example, when a user checks in to a shopping mall, the system will recommend fashion brand stores that the user is likely to be particularly interested in based on the user's past purchasing history. It also provides information about limited-time sales, allowing the user to efficiently search for products that interest them. Furthermore, if the user is also interested in a modeling event, they will also be notified of information about that event. In this way, customized action plans maximize the user's convenience and enjoyment.

[1311] The processing flow will be explained below.

[1312] Step 1:

[1313] When the user arrives at the shopping mall, they launch the dedicated application on their device, open the check-in screen, and tap the check-in button.

[1314] Step 2:

[1315] The device uses GPS sensors and Wi-Fi information to obtain the user's location, which is determined with a high degree of accuracy.

[1316] Step 3:

[1317] The device sends the acquired location information to the server, which confirms that the user is located within a specific shopping mall.

[1318] Step 4:

[1319] The server identifies the user ID based on the check-in information it receives, and retrieves the user's profile information and purchase history from the database.

[1320] Step 5:

[1321] An AI module installed on the server integrates the acquired user profile information with the user's current location information, and analyzes the user's hobbies and preferences.

[1322] Step 6:

[1323] Based on the results of the AI ​​analysis, the system recommends the most suitable products and stores for the user, including items and stores that are likely to interest the user.

[1324] Step 7:

[1325] The server generates a personalized action plan based on the AI ​​analysis results, which includes recommended products and store visit routes.

[1326] Step 8:

[1327] The server acquires sale information and coupon information provided by each store in the shopping mall and incorporates this information into an action plan.

[1328] Step 9:

[1329] The server transmits the generated action plan to the terminal.

[1330] Step 10:

[1331] The device notifies the user of the action plan it has received, such as sending a notification that a "recommended shopping route has been created."

[1332] Step 11:

[1333] When the user taps the notification, the details of the action plan will be displayed on the device, and the user can check the recommended stores, products, and sales information.

[1334] Step 12:

[1335] The device tracks the user's actions in real time, including recording the stores they visit and the route they take.

[1336] Step 13:

[1337] The device transmits the collected behavioral data to a server.

[1338] Step 14:

[1339] The server analyzes the new behavioral data it receives, and the AI ​​dynamically optimizes the plan. For example, if the user stops at an unexpected store, it will use that information to make new recommendations or adjust the plan.

[1340] Example 1

[1341] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1342] In recent years, there has been a demand for systems that utilize users' location information and purchase history to provide personalized, optimized shopping plans to improve the shopping experience in facilities. However, existing systems are unable to fully utilize users' preferences and real-time behavioral data, making it difficult to recommend optimal products and stores or provide appropriate shopping plans. Furthermore, due to the lack of dynamic feedback and optimization, these systems are unable to respond immediately to changes in user behavior.

[1343] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1344] In this invention, the server includes a means for a user to check in to a facility, a means for a terminal to acquire location information and send it to the server, a means for the server to acquire user profile information and purchase history, a means for analyzing the user's hobbies and preferences using AI and recommending products and stores, a means for generating and sending an action plan customized for the user, a means for notifying the user of the action plan and displaying it, a means for tracking the user's behavioral data and sending it to the server, and a means for analyzing new behavioral data and optimizing the plan. This makes it possible to recommend products and stores based on the user's hobbies and preferences, and to provide an optimal action plan that reflects real-time behavioral data.

[1345] A "user" is a person who uses the system to receive services or products within a facility.

[1346] A "facility" is a location where a particular service is provided, such as a shopping mall or public building where check-in occurs.

[1347] A "terminal" is a computing device carried by a user, such as a smartphone or tablet.

[1348] "Location information" refers to information that indicates the current location of a user or device, obtained using GPS or Wi-Fi.

[1349] "Server" means a remote computing system that receives User information and processes and analyzes the data.

[1350] "Profile information" refers to data that compiles attribute information and past behavioral history about a user.

[1351] "Purchase history" is a record of products purchased by a user in the past and a history of stores visited.

[1352] "AI" refers to an algorithm that uses artificial intelligence technology to analyze data and understand users' tastes and preferences.

[1353] "Hobbies and preferences" refers to information indicating the types of products and services that a user likes.

[1354] A "customized action plan" is a recommended plan of activities and destinations generated based on a user's individual needs and past behavior.

[1355] "Behavioral data" refers to real-time behavioral information such as a user's current location, travel route, and stores visited.

[1356] "Optimization" refers to the process of optimizing action plans and recommendations based on collected data.

[1357] "Sale information" refers to information about discounts and special offers offered by stores and facilities.

[1358] "Coupon information" refers to information about discount coupons and special coupons that can be used by users.

[1359] To implement the present invention, a user-owned terminal, a server, and an infrastructure for connecting them via communications are required. The specific operation of the system will be described below.

[1360] First, when a user arrives at a shopping mall or other facility, they launch a dedicated application on their smartphone, tablet, or other device and open the check-in screen. When they tap the check-in button, the device uses its GPS sensor and Wi-Fi module to obtain location information. This location information is converted into JSON format and sent over the Internet to a server. The server analyzes the information and records it in a database. This initial process confirms that the user is within a specific facility.

[1361] The server then identifies the user based on their check-in information and retrieves their profile information and purchase history from a relational database, which executes SQL queries to pull the required data. This information is then passed to the AI ​​module.

[1362] The AI ​​module in the server integrates the acquired profile information and purchase history to analyze the user's hobbies and preferences. The AI ​​module evaluates the category and price of the products the user has purchased in the past, as well as the frequency of visits, and recommends the most suitable products and stores for the user. The recommendation results are stored as a data structure.

[1363] Next, the server generates a customized action plan for the user based on the analysis results from the AI. This action plan includes the recommended order in which to visit stores and information on the user's movement within the store. The server also obtains the latest sales and coupon information provided by each store in the shopping mall and incorporates it into the action plan. This information is obtained using an API request. The generated action plan is formatted in JSON format and sent to the device.

[1364] The device analyzes the received action plan and notifies the user. Once the push notification is sent, the user can check the detailed action plan in a dedicated application. The plan includes store location information, the order in which to visit, sale information, and more.

[1365] Additionally, the device tracks the user's current location and travel route in real time and periodically sends the collected behavioral data to the server. This data is securely transmitted using HTTPS. The server then inputs the new behavioral data it receives back into the AI ​​module to dynamically optimize the plan. For example, if the user stops at an unexpected store, that information can be used to make new recommendations or adjust the plan. The new optimized plan is then sent back to the device and the user is notified.

[1366] Specific examples

[1367] For example, when a user checks in to a shopping mall, the system will recommend fashion brand stores that the user is likely to be interested in based on the user's past purchasing history. It also provides information about limited-time sales, allowing the user to efficiently search for products that interest them. Furthermore, if the user is also interested in a modeling event, they will be notified of information about that event. In this way, customized action plans maximize the user's convenience and enjoyment.

[1368] Example prompts for generative AI models

[1369] 1. "Please explain the specific steps a user takes to check in to a shopping mall."

[1370] 2. "Please use AI to analyze data based on user profile information and purchase history."

[1371] 3. "Explain how to dynamically optimize shopping plans based on real-time behavioral data."

[1372] By using these prompt sentences, it is expected that the generative AI model will understand the overall system processing and provide appropriate answers.

[1373] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1374] Step 1:

[1375] After arriving at the shopping mall, the user launches a dedicated application on their smartphone and opens the check-in screen. When they tap the "Check-in" button, the device uses the GPS sensor and Wi-Fi module to obtain their current location information. This location information is converted into JSON format and sent to the server. The input is the user's operation, and the output is location information data in JSON format. Specifically, the device calls an internal API to obtain location information and sends that data to the server.

[1376] Step 2:

[1377] The server executes a database query to identify the user ID based on the received location data. The server retrieves the user's profile information and purchase history from the database. The input is location data and user ID, and the output is profile information and purchase history data. Specifically, the server uses an SQL query to extract the required information from the database.

[1378] Step 3:

[1379] The AI ​​module installed on the server analyzes the user's hobbies and preferences using the acquired profile information and purchase history. The AI ​​module evaluates past purchase history and favorite trends and calculates the best products and stores for the user. The input is profile information and purchase history data, and the output is a list of recommended products. Specifically, the AI ​​algorithm analyzes this data and generates the analysis results.

[1380] Step 4:

[1381] The server generates an action plan customized for the user based on the analysis results from the AI ​​module. This plan includes the recommended order in which to visit stores and information on movement within the store. The server also obtains the latest sales and coupon information provided by stores in the shopping mall via API requests and incorporates this into the action plan. The input is the analysis results and store information, and the output is the action plan. Specifically, the server runs the plan generation algorithm, collects the necessary store information, and completes the action plan.

[1382] Step 5:

[1383] The server sends the generated action plan in JSON format to the device. The device analyzes the received action plan and sends a push notification saying, "We've created a recommended shopping route for you." The user can check the detailed action plan in a dedicated application. The input is the action plan data, and the output is a push notification and display information for the user. Specifically, the device analyzes the received data and notifies the user of the information.

[1384] Step 6:

[1385] The device tracks the user's current location and travel route in real time and periodically sends this behavioral data to the server. For example, it sends the data securely using HTTPS. The server inputs the new behavioral data it receives into the AI ​​module and dynamically optimizes the plan. If the user stops at an unexpected store, that information is used to make new recommendations and adjust the plan. The input is the new behavioral data, and the output is an optimized plan. Specifically, the AI ​​recalculates the plan based on the real-time data and sends the updated plan to the device.

[1386] Through these steps, the system leverages users' real-time behavioral data to provide a personalized shopping experience.

[1387] (Application example 1)

[1388] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1389] Modern shopping malls have many stores and products, making it difficult for users to shop efficiently and comfortably. It takes time and effort, especially, for users to find products and stores that interest them. Furthermore, if sales and coupon information is not provided appropriately, users may lose motivation to purchase. Furthermore, if plans are not optimized in real time according to user behavior, the user experience may be impaired.

[1390] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1391] In this invention, the server includes means for generating prompts recommended by AI based on the user's purchase history and profile information, means for notifying and optimizing an action plan including recommended stores and promotion information in real time, means for the user to check in to a facility, means for the terminal to acquire location information and send it to the server, means for the server to acquire the user's profile information and purchase history, means for the server to analyze the user's hobbies and preferences using AI and recommend products and stores, means for the server to generate and send an action plan customized for the user, means for the terminal to notify and display the action plan to the user, means for the terminal to track the user's behavioral data and send it to the server, and means for the server to analyze new behavioral data and optimize the plan. This allows users to efficiently find products and stores of their interest, and by applying sale information and coupon information, purchasing motivation is increased, and the user experience is optimized through real-time feedback.

[1392] "Checking in" refers to the act of notifying a server of a user's location information via a dedicated application when the user arrives at a facility.

[1393] A "terminal" is a device such as a smartphone or tablet operated by a user, and is a device for obtaining location information, sending notifications, displaying action plans, and the like.

[1394] "Location information" refers to geographical location data obtained using GPS or Wi-Fi, and is information used to identify a user's current location.

[1395] A "server" is a computer system that manages user profile information and purchase history, and recommends products and stores to users.

[1396] "Profile information" refers to integrated data such as a user's attributes, past purchasing history, interests and preferences, and generally refers to information about an individual user.

[1397] "Purchase history" is a record of products and services purchased by a user in the past, and is data used to analyze the user's purchasing trends.

[1398] "AI (artificial intelligence)" is a general term for algorithms and technologies that use user profile information and purchasing history to analyze a user's hobbies and preferences and recommend the most suitable products and stores.

[1399] "Hobbies and preferences" refers to the interests and concerns a user has toward specific products and services, and is an indicator for predicting user behavior and preferences.

[1400] An "action plan" is a plan generated by the server that includes a route to visit, recommended stores, product information, and sale information for a user.

[1401] "Notification" refers to a message or alert sent from the server to the user's device, and is a means of conveying action plans and recommended information to the user.

[1402] "Tracking" is the process of following a user's behavior in real time and recording the data obtained.

[1403] "Plan optimization" is the process of analyzing new user behavior data and updating the current behavior plan to an optimal form.

[1404] "Recommendation" is the act of AI selecting products and stores that a user might be interested in based on the user's profile information and purchasing history.

[1405] A "prompt" is text data that AI uses to give instructions and recommendations to users to show them the best course of action or choice.

[1406] "Real-time" means processing data instantly in response to changes in user behavior and the environment, and providing appropriate information immediately.

[1407] To implement this invention, a user-owned terminal, a server, and an infrastructure for connecting these via communications are required. The processing of the entire system will be described below.

[1408] System configuration

[1409] The system includes a terminal application for users to check in, a server that receives and processes data, and a tracking function for acquiring user behavior data. The terminal uses a GPS module and Wi-Fi adapter to obtain highly accurate location information and send it to the server. The server has a database containing user profile information and purchase history, an AI module, and a notification system.

[1410] Program Implementation

[1411] 1. User checks in

[1412] When a user arrives at a shopping mall or other facility, they launch a dedicated application on their smartphone and check in. When they tap the check-in button, the device uses GPS and Wi-Fi to obtain their current location information and sends it to a server.

[1413] 2. The server processes the data

[1414] The server identifies the user ID based on the received location information. It also retrieves the user's profile information and purchase history from the database. Based on this information, the AI ​​module analyzes the user's hobbies and preferences.

[1415] 3. Recommendations by AI module

[1416] The AI ​​module uses a user's profile information and purchase history to recommend products and stores that the user might be interested in. It does this using AI frameworks such as TensorFlow and PyTorch.

[1417] 4. Generate a customized action plan

[1418] Based on the analysis results of the AI ​​module, the server generates an optimal action plan for the user, which includes recommended store visit routes, sale information, coupon information, and more.

[1419] 5. User Notification and Interface

[1420] The terminal notifies the user of the action plan received from the server and displays it in the application. The action plan displayed includes a message such as "Here is a recommended route to the stores for you."

[1421] 6. Real-time feedback

[1422] The device tracks the user's actual behavior in real time and sends the data to the server, which then re-analyzes the data based on the new behavior data and dynamically optimizes the action plan.

[1423] The specific hardware and software used includes:

[1424] Hardware: Smartphone with built-in GPS sensor.

[1425] Software: Dedicated application, server is Django (backend framework), Python, SQLite (database), AI module is TensorFlow or PyTorch.

[1426] Specific examples

[1427] For example, when a user checks in to a shopping mall, AI analyzes their past purchase history and finds that they particularly like a certain fashion brand. Based on this, the user is notified of current sales at the fashion brand and the route to the store. Specific examples of prompts include the following:

[1428] "User profile: {Name: User A, Age: 30s, Gender: Female, Interests: Fashion, Purchase history: Fashion brand products.} Based on this data, please provide the optimal shopping plan for this user."

[1429] This allows users to enjoy shopping efficiently and improves the quality of their experience.

[1430] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1431] Step 1:

[1432] When a user arrives at a facility such as a shopping mall, they launch a dedicated application on their smartphone and tap the check-in button. The device uses a GPS module and Wi-Fi adapter to obtain current location information and sends it to a server. The input is the user's operation upon arriving at the facility, and the output is the transmission of location information to the server.

[1433] Step 2:

[1434] The server identifies the user ID based on the received location information. The server then retrieves the user's profile information and purchase history from the database. The input is location information, and the output is the extracted user ID, profile information, and purchase history.

[1435] Step 3:

[1436] The AI ​​module installed on the server integrates the acquired user profile information and purchase history. Based on this, the AI ​​module analyzes the user's hobbies and preferences. The input is the integrated profile information and purchase history, and the output is the analysis results of the user's hobbies and preferences. TensorFlow and PyTorch are used for the AI ​​module.

[1437] Step 4:

[1438] The server recommends the best products and stores for the user based on the analysis results of the AI ​​module. These recommendations include items and stores that the user is likely to be interested in. The input is the analysis results of the AI ​​module, and the output is a list of recommended products and stores.

[1439] Step 5:

[1440] The server generates a customized action plan based on the information on recommended stores and products. This plan includes the recommended store visit route, sale information, coupon information, etc. The input is a list of recommended stores and products, and the output is the customized action plan.

[1441] Step 6:

[1442] The server sends the generated action plan to the user's device. The device notifies the user of the received action plan and displays it in the application. The input is the action plan, and the output is notification and display to the user. The displayed action plan includes a message such as "Here is a recommended route to the stores for you."

[1443] Step 7:

[1444] The terminal tracks the user's actual behavior in real time and transmits the data to the server. The input is the user's behavior data, and the output is the transmission of the behavior data to the server.

[1445] Step 8:

[1446] The server then performs a second analysis based on the new behavioral data and dynamically optimizes the behavior plan. The input is the new behavioral data, and the output is the optimized behavior plan. This allows for real-time optimization of the plan according to the user's behavior.

[1447] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1448] To implement the present invention, a user-owned terminal, a server, an emotion engine, and an infrastructure for connecting these via communications are required. The processing flow of the entire system will be specifically explained below.

[1449] Overall system processing flow

[1450] 1. A user checks in to a shopping mall

[1451] When a user arrives at the shopping mall, they launch the dedicated application on their device and open the check-in screen. When they tap the check-in button, the device obtains the user's location information. This location information is obtained with high accuracy using GPS and Wi-Fi.

[1452] The device sends the acquired location information to a server, which confirms that the user is located within a specific shopping mall.

[1453] 2. Obtaining user profile information and emotion data

[1454] The server identifies the user ID based on the received check-in information and retrieves the user's profile information and purchase history from the database.

[1455] The user's facial expressions and tone of voice are captured by the device's camera and microphone and sent to the emotion engine.

[1456] The emotion engine analyzes the user's facial expression data and tone of voice to determine their current emotional state.

[1457] 3. Data analysis and product recommendations using AI and emotion engines

[1458] The AI ​​module installed on the server integrates the acquired user profile information, current location information, and emotional data to analyze the user's hobbies and preferences.

[1459] Based on the analysis, the AI ​​and emotion engine will recommend the best products and stores for the user, including items and stores that are likely to interest the user.

[1460] 4. Generate a customized action plan

[1461] The server uses the AI ​​and emotion engine analysis results to generate a personalized action plan for the user, including recommended products and store routes.

[1462] The server obtains sales and coupon information provided by each store in the shopping mall and incorporates this information into an action plan.

[1463] 5. User Notification and Display

[1464] The server transmits the generated action plan to the terminal.

[1465] The device notifies the user of the received action plan and displays the details within the application. For example, it may send a notification to the user saying, "We've created a recommended shopping route for you."

[1466] 6. Real-time feedback and optimization

[1467] The device tracks the user's actions in real time, in particular recording the stores the user actually visits and the route they take.

[1468] The device sends the collected behavioral data to the server.

[1469] The server analyzes the new behavioral data and updated emotional data it receives, and the AI ​​dynamically optimizes the plan based on this. For example, if the user stops at an unexpected store, it will use that information to make new recommendations or adjust the plan.

[1470] Specific examples

[1471] For example, when a user checks in to a shopping mall, the system will recommend fashion brand stores that the user is likely to be interested in based on the user's past purchasing history and current emotional state. It also provides information about products on limited-time sales, allowing users to efficiently search for products that interest them. Furthermore, the system will notify users of events that may interest them (e.g., modeling events). In this way, customized action plans maximize user convenience and enjoyment. By utilizing feedback from the emotion engine, if the user is excited about a particular product, it is also possible to suggest other products in that genre or related events in real time.

[1472] The processing flow will be explained below.

[1473] Step 1:

[1474] When the user arrives at the shopping mall, they launch the dedicated application on their device, open the check-in screen, and tap the check-in button.

[1475] Step 2:

[1476] The device uses GPS sensors and Wi-Fi information to obtain the user's location, which is determined with a high degree of accuracy.

[1477] Step 3:

[1478] The device sends the acquired location information to the server, which confirms that the user is located within a specific shopping mall.

[1479] Step 4:

[1480] The server identifies the user ID based on the check-in information it receives, and retrieves the user's profile information and purchase history from the database.

[1481] Step 5:

[1482] The device captures the user's facial expression data with a camera and their tone of voice with a microphone, and sends the acquired data to the emotion engine.

[1483] Step 6:

[1484] The emotion engine analyzes the user's facial expression data and tone of voice to determine their current emotional state, which can include happiness, excitement, focus, etc.

[1485] Step 7:

[1486] The AI ​​module on the server integrates the acquired user profile information, current location information, and emotional data, and analyzes the user's hobbies and preferences.

[1487] Step 8:

[1488] Based on the results of analysis by AI and an emotion engine, the system recommends the most suitable products and stores for users, including items and stores that are likely to interest them.

[1489] Step 9:

[1490] The server generates a personalized action plan based on the analysis results of the AI ​​and emotion engine, including recommended products and store visit routes.

[1491] Step 10:

[1492] The server incorporates sale information and coupon information provided by each store in the shopping mall into the action plan.

[1493] Step 11:

[1494] The server transmits the generated action plan to the terminal.

[1495] Step 12:

[1496] The device notifies the user of the action plan it has received, such as sending a notification that a "recommended shopping route has been created."

[1497] Step 13:

[1498] When the user taps the notification, the details of the action plan will be displayed on the device, and the user can check the recommended stores, products, and sales information.

[1499] Step 14:

[1500] The device tracks the user's actions in real time, including recording the stores they visit and the route they take.

[1501] Step 15:

[1502] The device transmits the collected behavioral data to a server.

[1503] Step 16:

[1504] The server analyzes the new behavioral data and updated emotional data received. The AI ​​dynamically optimizes the plan based on this data. If the user stops at an unexpected store, the AI ​​will use this information to make new recommendations or adjust the plan.

[1505] As a specific example, if the emotion engine determines that a user is feeling excited in front of a specific store in a shopping mall after checking in, it will recommend products and information about other stores that fit that state of excitement in real time. For example, new products from a fashion brand or related accessory stores will be recommended. Also, if the user is feeling relaxed, information about cafes and rest areas will be provided. This allows users to enjoy a shopping experience optimized for their interests and emotions.

[1506] Example 2

[1507] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1508] Conventional shopping support systems typically recommend products and stores based on a user's location information and purchase history. However, these systems are unable to fully consider a user's momentary emotions or specific hobbies and preferences, making it difficult to provide optimal recommendations and action plans. As a result, users' purchasing experiences are limited, reducing the effectiveness of the system.

[1509] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1510] In this invention, the server includes means for acquiring user profile information and purchase history, means for acquiring the user's facial expression data and voice data and transmitting them to the emotion analysis device, means for the emotion analysis device to analyze the user's facial expression data and voice data to determine the user's emotional state, and means for analyzing the user's hobbies, preferences, and emotional state using AI and recommending products and stores. This makes it possible to provide more accurate product and store recommendations and action plans that take the user's emotional state into consideration.

[1511] "User" means a person who uses the system.

[1512] "Facilities" refer to places that users visit, such as shopping malls and commercial facilities.

[1513] "Device" refers to an electronic device such as a smartphone or tablet owned by a user.

[1514] "Location information" means data about a user's current location obtained using GPS, Wi-Fi, etc.

[1515] "Server" refers to a computer system that stores, processes, and distributes data over a network.

[1516] "Profile information" refers to data about a user's personal information and their interests and preferences.

[1517] "Purchase history" means a record of products and services purchased by a user in the past.

[1518] "Facial expression data" refers to digital data relating to the user's facial expressions captured by the device's camera.

[1519] "Voice data" refers to digital data relating to the user's voice, including the tone and content of the user's voice.

[1520] An "emotion analysis device" refers to a system that analyzes facial expression data and voice data to determine a user's emotional state.

[1521] "AI" refers to the technology that uses artificial intelligence to analyze data and understand and predict users' hobbies, preferences, and emotional states.

[1522] "Hobbies and preferences" refers to items and categories in which a user has interests or preferences.

[1523] An "action plan" refers to a plan that includes recommended routes for users to move around the facility efficiently and comfortably, as well as stores and products of interest.

[1524] "Tracking" means tracking a user's actual behavior and movement path in real time.

[1525] "Optimization" refers to dynamically adjusting and improving existing action plans based on new user behavior data.

[1526] "Sale Information" means information about discounts and special offers offered by stores within the shopping mall.

[1527] "Coupon information" refers to information about discount coupons and special offers that users can use.

[1528] To implement the present invention, the following elements are required: a terminal owned by a user, a server, an emotion analysis device, and infrastructure for connecting these via communications. A detailed description of the system is provided below.

[1529] Hardware and Software Configuration

[1530] Device: A smartphone or tablet is used. These devices are equipped with a GPS sensor, camera, microphone, etc. A dedicated application is also installed and is used by the user to check in and collect emotional data.

[1531] Server: A computer system that stores, processes, and distributes data over a network. It uses a database (SQL Server, MySQL, etc.) to manage user profile information and purchase history. It is equipped with an AI module (TensorFlow, PyTorch, etc.) and integrates and analyzes data.

[1532] Emotion analysis device: A system for analyzing facial expression data and voice data. It uses a facial expression analysis algorithm (such as OpenCV) or a voice emotion analysis module (such as Google Cloud Speech-to-Text).

[1533] Communication infrastructure: Connect devices, servers, and emotion analysis devices using the internet or Wi-Fi.

[1534] Details of data processing and data calculation

[1535] When a user arrives at a shopping mall, they launch the dedicated app and display the check-in screen. When the user taps the check-in button, the device uses GPS and Wi-Fi to obtain location information and sends that information to the server. The server identifies the user ID based on this location information and retrieves profile information and purchase history from the database.

[1536] The device then uses a camera and microphone to capture the user's facial expression and voice data, which are then sent to an emotion analyzer. The emotion analyzer analyzes the user's emotional state in real time and sends the results back to the server. The server then uses an AI module to integrate the profile information, location information, and emotion data to analyze the user's interests and preferences.

[1537] As a result, the server recommends the most suitable products and stores for the user and generates a customized action plan based on this. This action plan includes a recommended route and a list of recommended products. The server also obtains sales and coupon information provided by each store in the shopping mall and integrates it into the action plan.

[1538] The generated action plan is sent to the device and notified to the user. The device receives it and displays it to the user. As the user walks the recommended route, the device tracks their actions and collects new action data. This data is sent to the server, and the AI ​​module dynamically optimizes the plan.

[1539] Specific examples

[1540] For example, when a user checks in to a shopping mall, the system will recommend fashion brand stores that the user is likely to be interested in based on the user's past purchasing history and current emotional state. It will also provide information on products on limited-time sales. This allows users to efficiently search for products that interest them. In addition, the system will notify users of events that may interest them (e.g., modeling events). Using feedback from an emotion analyzer, if the user becomes excited in front of a particular product, it will be possible to suggest other products in that genre or related events in real time.

[1541] Prompt Sentence Examples

[1542] "When a user checks in to a shopping mall, the emotion analyzer analyzes the user's emotional state based on data acquired from the device's camera and microphone. Based on the results, recommend the most suitable products and stores for the user."

[1543] In this way, in order to implement the invention, by appropriately using the above-mentioned hardware and software and coordinating the flow of the entire system, it is possible to provide users with an optimal purchasing experience.

[1544] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1545] Step 1:

[1546] A user checks in to a facility

[1547] When a user arrives at the shopping mall, they launch the dedicated application and open the check-in screen. When they tap the check-in button, the device acquires the user's location information.

[1548] Input: User action (tapping the check-in button)

[1549] Output: Obtain location information (latitude, longitude data)

[1550] Specific operation: The device detects the user's current location using the GPS sensor and Wi-Fi and sends the location information to the server.

[1551] Step 2:

[1552] The device acquires location information and sends it to the server.

[1553] The device uses the acquired location information to confirm that the user is within a specific facility, and sends this location information in JSON format to the server.

[1554] Input: Location information (latitude, longitude)

[1555] Output: Send location information to the server

[1556] Specific operation: The device uses the location information API to format the acquired location data into JSON format and send it to the server to determine the user's current location.

[1557] Step 3:

[1558] The server retrieves the user's profile information and purchase history.

[1559] The server identifies the user ID based on the received location information and retrieves profile information and purchase history from the database.

[1560] Input: Location information, user ID

[1561] Output: Profile information, purchase history

[1562] What happens: The server executes a database query to retrieve the user's profile information and purchase history data, which is used in the subsequent analysis process.

[1563] Step 4:

[1564] The device acquires the user's facial expression and voice data and sends it to the emotion analysis device.

[1565] The user's facial expressions and tone of voice are collected from a camera and microphone, and this data is sent to an emotion analysis device.

[1566] Input: facial expression data, voice data

[1567] Output: Send data to emotion analyzer

[1568] Specific operation: The device captures the user's facial expressions using the camera function and records audio using the microphone. This data is then sent to the emotion analysis device in real time.

[1569] Step 5:

[1570] The emotion analyzer analyzes the user's facial expression and voice data to determine their emotional state.

[1571] The emotion analyzer uses facial expression analysis algorithms and voice emotion analysis modules to analyze the user's emotional state.

[1572] Input: facial expression data, voice data

[1573] Output: Emotional state discrimination result

[1574] Specific operation: The emotion analysis device uses tools such as OpenCV and Google Cloud Speech-to-Text to analyze the transmitted facial expression and voice data to determine the user's current emotional state.

[1575] Step 6:

[1576] The server uses AI to analyze the user's tastes and emotional state and recommend products and stores.

[1577] The AI ​​module installed on the server integrates user profile information, purchase history, and emotional state data to analyze hobbies and preferences.

[1578] Input: Profile information, purchase history, emotional state data

[1579] Output: Product and store recommendation results

[1580] Specific operation: The server's AI module uses TensorFlow and PyTorch to analyze user data and generate a list of the most suitable products and stores.

[1581] Step 7:

[1582] The server generates and sends a customized action plan to the user.

[1583] Based on the recommendations from the AI, the server generates an optimal action plan for the user and sends it to the device.

[1584] Input: Product and store recommendation results

[1585] Output: A customized action plan

[1586] Specific operation: The action plan includes the order of stores to visit, the route, and a list of recommended products. The server also obtains sales and coupon information from the shopping mall and incorporates this information into the action plan.

[1587] Step 8:

[1588] The device notifies and displays the action plan to the user.

[1589] The terminal notifies the user of the action plan received from the server and displays the details within the application.

[1590] Input: Customized Action Plan

[1591] Output: User notification and display

[1592] What it does: The device uses in-app notifications to notify the user of the action plan and display a message such as "We've created a recommended shopping route for you."

[1593] Step 9:

[1594] The device tracks user behavior data and sends it to the server.

[1595] The device tracks users' real-time behavior and collects data on specific stores and routes.

[1596] Input: User behavior data

[1597] Output: Sending tracking data to the server

[1598] Specific operation: The device uses GPS and Bluetooth beacons to record the user's movements and periodically transmits them to the server.

[1599] Step 10:

[1600] The server analyzes new behavioral data and optimizes the plan

[1601] The server dynamically optimizes the behavior plan based on the new behavior data received.

[1602] Input: New behavioral data

[1603] Output: Optimized action plan

[1604] Specific behavior: The server's AI takes into account new behavioral data and emotional state, adjusting and updating its behavior plan in real time based on unplanned behaviors and changing interests.

[1605] (Application example 2)

[1606] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1607] In today's shopping experience, users lack the information and guidance they need to make efficient and satisfying purchases. In particular, it is difficult to provide personalized product and store recommendations based on a user's preferences and current emotional state in brick-and-mortar stores. There is also a need for action plans that effectively incorporate real-time sales and coupon information. It is necessary to solve these issues and provide users with an optimal shopping experience.

[1608] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1609] In this invention, the server includes means for acquiring facial expression and voice data from the device's camera and microphone and sending it to an emotion engine, means for the emotion engine to analyze the facial expression and voice data and determine the user's emotional state, and means for the server to use AI to integrate and analyze the user's hobbies, preferences, and emotional data and recommend products and stores. This makes it possible to recommend appropriate products and stores based on the user's hobbies, preferences, and emotional state, and to provide a more personalized action plan by tracking the user's behavioral data.

[1610] A "user" is a consumer who visits a facility or physical store and is the person who receives an action plan or product recommendations.

[1611] "Facility" refers to a collection of commercial facilities and physical stores that users visit, such as shopping malls and department stores.

[1612] "Checking in" is the act of a user arriving at a facility and identifying their location using a smartphone or other device.

[1613] A "terminal" is a mobile device such as a smartphone or tablet that is owned by the user, and is a device that acquires location information, collects emotional data, notifies and displays action plans, etc.

[1614] "Location information" is data that indicates a user's current location and movement patterns, determined using GPS sensors and Wi-Fi information.

[1615] The "server" is a central management system that manages user profile information, purchase history, and emotional data, and analyzes and processes them using AI.

[1616] "Profile information" is user data that includes personalized information such as the user's age, gender, preferences, and past purchasing history.

[1617] "Purchase history" is a record of products and services purchased by a user in the past.

[1618] "AI" is a system that uses artificial intelligence technology to analyze users' preferences and emotions and recommend the most suitable products and stores.

[1619] The "Emotion Engine" is a system that uses facial expression recognition and voice analysis technologies to analyze and determine the user's emotional state.

[1620] "Hobbies and preferences" are attributes preferred by individual users, such as areas of interest and concern, favorite things and activities, etc.

[1621] "Recommending products and stores" means presenting products that may interest the user and stores that they should visit based on data analyzed by AI and an emotion engine.

[1622] An "action plan" is a schedule that includes a list of stores and products to visit and a travel route, and is generated by the server to enable the user to enjoy shopping efficiently.

[1623] "Notification" refers to a push notification or alert that the device sends to inform the user of the generated action plan or recommended information.

[1624] "Tracking" refers to a device recording a user's behavioral data in real time and sending it to a server.

[1625] "Sale information" is product information about discounts and special prices offered by stores.

[1626] "Coupon information" is information about discount coupons and special offers offered by stores.

[1627] "Optimization" is the process by which AI dynamically adjusts and improves its action plan based on the user's new behavioral data and updated emotional data.

[1628] To implement this invention, a server, a user terminal, an emotion engine, and a communication infrastructure are required. Below, an outline of the program processing of the entire system will be explained in detail.

[1629] First, the user arrives at the facility and checks in by launching a dedicated application on their device. The user's device uses a GPS sensor and Wi-Fi information to obtain highly accurate location information. This location information is sent to the server, which confirms that the user is within a specific facility. After check-in, the server retrieves the user's profile information and purchase history from a database based on the user ID.

[1630] The device then uses a camera and microphone to capture the user's facial expressions and voice. This data is sent to the emotion engine, which uses facial expression recognition and voice analysis technologies to determine the user's current emotional state. The emotion engine utilizes facial expression recognition APIs (e.g., Google Face API) and voice analysis APIs (e.g., Google Speech-to-Text).

[1631] Once the user's emotional state is determined, the server integrates and analyzes the profile information, purchase history, location information, and emotional data. This analysis uses artificial intelligence (AI) technology, such as machine learning models using TensorFlow or PyTorch. The server then recommends products and stores based on the user's preferences and emotional state.

[1632] The server then generates a customized itinerary, which includes information such as the stores and products to visit, sales and coupons, and the best route to travel. The itinerary is then sent to the user's device, which notifies the user and displays it within the application. Notifications are sent using Firebase Cloud Messaging (FCM) and Apple Push Notification Service (APNS).

[1633] When a user begins to take action, the device tracks the user's movements and destinations in real time and sends that data to the server. The server analyzes the newly collected behavioral data and updated emotional data, and the AI ​​optimizes the action plan in real time. This allows the device to quickly respond to any new stores or products the user shows interest in.

[1634] For example, when a user checks in to a fashion brand store within the facility, the system will identify the brand that excites them based on their past purchase history and facial recognition, and notify them of the store's sale information in real time. Also, if the user shows an extended interest in a particular product, it will be possible to present related products and events in that genre.

[1635] To accomplish this process, the following example prompts can be used:

[1636] "Create an application that guides users when they check in to a physical store. Based on their past purchase history and current emotional data, it needs to recommend the most suitable stores and products to users. The application will use the smartphone's GPS and Wi-Fi to obtain the user's location, and facial recognition and speech analysis APIs to collect emotional data. Finally, it will push a generated action plan to the smartphone. Technologies used should include the following: GPS API, Wi-Fi sensor, Google Face API, Google Speech-to-Text, AWS, TensorFlow, and Firebase Cloud Messaging."

[1637] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1638] Step 1:

[1639] The user arrives at the facility and checks in by launching a dedicated application on the terminal.

[1640] Input: Location information of user and device arriving at facility

[1641] Specific operation: The user opens the smartphone application and taps the check-in button to obtain location information. The device uses the GPS sensor and Wi-Fi information to obtain the precise location.

[1642] Output: Obtained user location information

[1643] Step 2:

[1644] The location information acquired by the device is sent to the server.

[1645] Input: User's location

[1646] Specific operation: The device creates a communication request to send the acquired location information packet to the server and sends it to the server API endpoint.

[1647] Output: User location information received by the server

[1648] Step 3:

[1649] The server retrieves profile information and purchase history from a database based on the user ID.

[1650] Input: User's location and user ID

[1651] Specific operation: The server executes a database query using the location information and user ID as keys to obtain the user's profile information and purchase history.

[1652] Output: User profile information and purchase history

[1653] Step 4:

[1654] The device uses a camera and microphone to capture facial and voice data and sends it to the emotion engine.

[1655] Input: Real-time facial and voice data of the user

[1656] Specific operation: The device camera takes a picture of the user's face and records their voice with the microphone. A request is created to send this data to the emotion engine, and then it is sent.

[1657] Output: Facial and vocal data sent to the emotion engine

[1658] Step 5:

[1659] An emotion engine analyzes facial and voice data to determine the user's emotional state.

[1660] Input: facial expression and voice data

[1661] How it works: The emotion engine uses facial expression recognition APIs and voice analysis APIs to analyze data and determine the user's emotional state (excited, happy, calm, etc.).

[1662] Output: Determined user's emotional state

[1663] Step 6:

[1664] The server integrates and analyzes profile information, purchase history, location information, and emotional data to recommend products and stores.

[1665] Input: Profile information, purchase history, location information, emotional data

[1666] Specific operation: The server uses a generative AI model to analyze the user's hobbies and preferences based on input data and generate a list of optimal products and stores.

[1667] Output: A list of recommended products and stores

[1668] Step 7:

[1669] The server generates and transmits a customized action plan.

[1670] Input: Recommended products and store lists, sale information, coupon information

[1671] Specific operation: The server uses an action plan generation algorithm to generate an action plan that includes the order of stores and products to visit, travel routes, and sale and coupon information. The generated action plan is then sent to the user's device.

[1672] Output: A customized action plan

[1673] Step 8:

[1674] The terminal notifies the user of the generated action plan and displays it within the application.

[1675] Input: Generated action plan

[1676] Specific operation: The device will use a push notification service (FCM or APNS) to send a notification of the action plan to the user. The application will then interactively display a detailed action plan.

[1677] Output: A plan of action communicated to the user and displayed

[1678] Step 9:

[1679] The device tracks the user's behavior in real time and transmits the data to a server.

[1680] Input: Real-time user behavior data

[1681] Specific operation: The device records the user's location and visited destinations in real time, and creates and sends a request to send that data to the server.

[1682] Output: User behavior data sent to the server

[1683] Step 10:

[1684] The server analyzes the new behavioral data and updated emotional data to optimize the behavior plan.

[1685] Input: New behavioral data and updated emotion data

[1686] How it works: Based on the data received in real time, the server uses a generative AI model to dynamically adjust and optimize the action plan, recommending new products and stores, changing the route, etc. as needed.

[1687] Output: Optimized action plan

[1688] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1689] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1690] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1691] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1692] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1693] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1694] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1695] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1696] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1697] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1698] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1699] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1700] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1701] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1702] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1703] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1704] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1705] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1706] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1707] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1708] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1709] The following is further disclosed regarding the above embodiment.

[1710] (Claim 1)

[1711] A user checks in to a facility

[1712] Means and

[1713] The device acquires location information and sends it to the server.

[1714] Means and

[1715] The server retrieves the user's profile information and purchase history.

[1716] Means and

[1717] The server uses AI to analyze the user's hobbies and preferences and recommends products and stores.

[1718] Means and

[1719] The server generates and sends a customized action plan to the user.

[1720] Means and

[1721] The device notifies and displays the action plan to the user.

[1722] Means and

[1723] The device tracks user behavior data and sends it to the server.

[1724] Means and

[1725] The server analyzes new behavioral data and optimizes the plan

[1726] means and

[1727] A system including:

[1728] (Claim 2)

[1729] Uses GPS sensor and Wi-Fi information to obtain user location information

[1730] 10. The system of claim 1, comprising:

[1731] (Claim 3)

[1732] The server incorporates sales and coupon information into the action plan.

[1733] 10. The system of claim 1, comprising:

[1734] "Example 1"

[1735] (Claim 1)

[1736] A user checks in to a facility

[1737] Means and

[1738] The device acquires location information and sends it to the server.

[1739] Means and

[1740] The server retrieves the user's profile information and purchase history.

[1741] Means and

[1742] The server uses AI to analyze the user's hobbies and preferences and recommends products and stores.

[1743] Means and

[1744] The server generates and sends a customized action plan to the user.

[1745] Means and

[1746] The device notifies and displays the action plan to the user.

[1747] Means and

[1748] The device tracks user behavior data and sends it to the server.

[1749] Means and

[1750] The server analyzes new behavioral data and optimizes the plan

[1751] means and

[1752] A system including:

[1753] (Claim 2)

[1754] Uses GPS sensor and Wi-Fi information to obtain user location information

[1755] 10. The system of claim 1, comprising:

[1756] (Claim 3)

[1757] The server incorporates sales and discount information into the action plan.

[1758] 10. The system of claim 1, comprising:

[1759] "Application Example 1"

[1760] (Claim 1)

[1761] a means for a user to check in to a facility;

[1762] A means for the terminal to acquire location information and transmit it to a server;

[1763] A means for the server to obtain user profile information and purchase history;

[1764] The server uses AI to analyze the user's tastes and preferences and recommend products and stores.

[1765] means for the server to generate and transmit a customized action plan to the user;

[1766] means for the terminal to notify and display the action plan to the user;

[1767] A means for the terminal to track user behavior data and transmit the data to a server;

[1768] A means for the server to analyze new behavioral data and optimize the plan;

[1769] A means to notify and optimize action plans in real time, including recommended stores and promotion information.

[1770] A system that includes a means for generating AI-recommended prompts based on a user's purchasing history and profile information.

[1771] (Claim 2)

[1772] 10. The system of claim 1, further comprising means for using a GPS sensor and Wi-Fi information to obtain location information of the user.

[1773] (Claim 3)

[1774] 2. The system of claim 1, wherein the server includes means for incorporating sale information and coupon information into the action plan.

[1775] "Example 2: Combining Emotion Engines"

[1776] (Claim 1)

[1777] A user checks in to a facility

[1778] Means and

[1779] The device acquires location information and sends it to the server.

[1780] Means and

[1781] The server retrieves the user's profile information and purchase history.

[1782] Means and

[1783] The device acquires the user's facial expression and voice data and sends it to the emotion analysis device.

[1784] Means and

[1785] The emotion analyzer analyzes the user's facial expression and voice data to determine their emotional state.

[1786] Means and

[1787] The server uses AI to analyze the user's tastes and emotional state and recommend products and stores.

[1788] Means and

[1789] The server generates and sends a customized action plan to the user.

[1790] Means and

[1791] The device notifies and displays the action plan to the user.

[1792] Means and

[1793] The device tracks user behavior data and sends it to the server.

[1794] Means and

[1795] The server analyzes new behavioral data and optimizes the plan

[1796] means and

[1797] A system including:

[1798] (Claim 2)

[1799] Uses GPS sensor and Wi-Fi information to obtain user location information

[1800] 10. The system of claim 1, comprising:

[1801] (Claim 3)

[1802] The server incorporates sales and coupon information into the action plan.

[1803] 10. The system of claim 1, comprising:

[1804] "Application example 2 when combining emotion engines"

[1805] (Claim 1)

[1806] a means for a user to check in to a facility;

[1807] A means for the terminal to acquire location information and transmit it to a server;

[1808] A means for the server to obtain user profile information and purchase history;

[1809] A means for acquiring facial expression and voice data from the camera and microphone of the device and transmitting the data to the emotion engine;

[1810] means for the emotion eng...

Claims

1. A user checks into a facility Means and The device acquires location information and sends it to the server. Means and The server retrieves the user's profile information and purchase history. Means and The server uses AI to analyze the user's hobbies and preferences and recommends products and stores. Means and The server generates and sends a customized action plan to the user. Means and The device notifies and displays the action plan to the user. Means and The device tracks user behavior data and sends it to the server. Means and The server analyzes new behavioral data and optimizes the plan means and A system including:

2. Uses GPS sensor and Wi-Fi information to obtain user location information 10. The system of claim 1, comprising:

3. The server incorporates sales and coupon information into the action plan.

10. The system of claim 1, comprising:

Images