system

Abstract

We provide the system. [Solution] An identification device for recognizing customer characteristics, A storage device for obtaining customer history information, A computing device for acquiring location information of items within a store and generating guidance information based on customer history information, A communication device for providing guidance information to the customer's terminal, A system that includes this.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor, and includes steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] The present invention solves the problem that in a store visited for the first time or a store with a changed layout, customers cannot quickly determine the arrangement of the items they want to purchase, resulting in a waste of time due to getting lost in the store. Also, on the store side, it aims to solve the problem that it is difficult to acquire new customers and there is a lack of means to break away from an operation relying on regular customers.

Means for Solving the Problems

[0005] This invention identifies customers using an identification device that recognizes their characteristics. It then retrieves customer history information stored in a storage device and compares it with location information of items within the store to generate guidance information based on the customer's history. Furthermore, by providing this guidance information to the customer's terminal via a communication device, customers can efficiently purchase items. Additionally, by using facial recognition technology in the identification device, smooth customer recognition is achieved, and providing guidance information in voice format enables more intuitive use.

[0006] An "identification device" is a device used to recognize and identify the characteristics of a customer.

[0007] "Historical information" refers to data about a customer's past purchasing activities and indicates their preferences.

[0008] A "storage device" is a device used to store and manage data such as historical information.

[0009] "Location information" refers to data about the arrangement and location of items within a store.

[0010] A "computing device" is a device that performs calculations and data processing based on acquired information and generates necessary guidance information.

[0011] A "communication device" is a device used to transmit generated guidance information to the customer's terminal.

[0012] A "terminal" is an information display device owned by a customer, used to receive guidance information from a communication device.

[0013] "Guidance information" refers to location guidance and recommended routes that help customers efficiently find and purchase items. [Brief explanation of the drawing]

[0014] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] It is a conceptual diagram showing an example of the main functions of a data processing device and a smart device according to the first embodiment. [Figure 3] It is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] It is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] It is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] It is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] It is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] It is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] It shows an emotion map to which a plurality of emotions are mapped. [Figure 10] It shows an emotion map to which a plurality of emotions are mapped. [Figure 11] It is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] It is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] It is a sequence diagram showing the processing flow of the data processing system in Example 2 when an emotion engine is combined. [Figure 14] It is a sequence diagram showing the processing flow of the data processing system in Application Example 2 when an emotion engine is combined.

Mode for Carrying Out the Invention

[0015] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0016] First, the terms used in the following description will be explained.

[0017] In the following embodiments, a processor with a reference numeral (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0018] In the following embodiments, a RAM (Random Access Memory) with a reference numeral is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0019] In the following embodiments, a storage with a reference numeral is one or more non-volatile storage devices that store various programs and various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes, and the like.

[0020] In the following embodiments, a communication I / F (Interface) with a reference numeral is an interface including a communication processor and an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark), and the like.

[0021] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0022] [First Embodiment]

[0023] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0024] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0025] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0026] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0027] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0028] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0029] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0030] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0031] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0032] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0033] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0034] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0035] This invention provides a system that enables users visiting a store to efficiently find and purchase the items they need. The system includes an identification device, a storage device for storing historical information, a computing device for acquiring location information, a communication device, and a terminal carried by the user.

[0036] First, when a user enters the store, an identification device is activated via a terminal and captures the user's face. The server then compares the captured facial data with existing customer data stored internally to identify the user. Because facial recognition technology is used in this process, the user is identified smoothly.

[0037] Next, the server retrieves user history information from storage. This history information includes past purchase history and user preferences, which are used to determine the user's needs and preferences. The server also retrieves location information for the latest products in the store. This location information indicates the placement of each item within the store.

[0038] Subsequently, the server generates personalized guidance information for the user based on historical and location data. This guidance information not only shows the location of items the user wishes to purchase, but also includes the optimal route within the store. Furthermore, it generates product recommendations tailored to the user's preferences.

[0039] The server transmits the generated guidance information to the user's terminal via a communication device. The terminal displays the received information on its screen and sometimes provides guidance to the user through voice guidance. This allows the user to efficiently move around the store and purchase the items they need. This guidance includes specific location information such as "Product XX is in the left aisle" and information such as "Seasonal items are on sale."

[0040] As a concrete example, in a large supermarket visited by a user for the first time, a camera near the entrance recognizes the user's face and guides them to the location of everyday items they have purchased in the past, allowing the user to complete their necessary shopping in a short amount of time. This system makes the user's shopping experience more efficient and comfortable, while also enabling the store to make effective product suggestions based on purchasing trends.

[0041] The following describes the processing flow.

[0042] Step 1:

[0043] The user passes through the store entrance. As a result, a camera installed at the entrance captures the user's face.

[0044] Step 2:

[0045] The device acquires facial image data captured by the device and sends that data to the server.

[0046] Step 3:

[0047] The server identifies users by matching the received facial image data with an internal customer database. A facial recognition algorithm is used to efficiently execute this process.

[0048] Step 4:

[0049] The server retrieves the user's history information from its storage device. This history information includes past purchase history and preference data.

[0050] Step 5:

[0051] The server retrieves the latest item placement data from the store management system. This data includes the precise location information of each item within the store.

[0052] Step 6:

[0053] The server generates personalized guidance information for the user based on historical data and item placement data. This information includes the location of desired items, the optimal route within the store, and even recommended products.

[0054] Step 7:

[0055] The server generates guidance information and transmits it to the user's terminal via a communication device.

[0056] Step 8:

[0057] The device displays the received guidance information on the screen. If necessary, it plays audio guidance to provide instructions in an easy-to-understand format for the user.

[0058] Step 9:

[0059] The user follows the instructions to move around the store and purchase the desired item.

[0060] Step 10:

[0061] The server saves the user's new purchase information to its storage device and updates the history information. The updated information will be used to generate future notifications.

[0062] (Example 1)

[0063] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0064] In current stores, it is difficult for customers to efficiently find the items they need, and this is particularly time-consuming in larger stores. Furthermore, product recommendations based on customers' past purchase history and preferences are not being implemented effectively, resulting in an inability to provide customers with the optimal shopping experience.

[0065] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0066] In this invention, the server includes means for photographing a customer's face and recognizing its features, storage means for acquiring the customer's purchase history and preference data, and information processing means for aggregating information on the placement of items in the store and generating personalized guidance based on the customer's history and preferences. This enables customers to quickly find the items they need and receive product suggestions that match their preferences.

[0067] The term "customer" refers to people who visit a store and receive goods or services.

[0068] A "device that photographs faces and recognizes features" is a device that uses cameras and sensors to acquire images of customers' faces and analyze their feature points.

[0069] "Purchase history" is a concept that refers to information about products and services that a customer has purchased in the past.

[0070] "Preference data" refers to data that has been accumulated to include information related to customers' preferences and interests.

[0071] The term "memory device" refers to an electronic device or mechanism used to store and retrieve information as needed.

[0072] "Item placement information" refers to information about the location and placement of each product within a store.

[0073] "Information processing means" refers to computer devices and algorithms used to analyze data and generate results tailored to specific purposes.

[0074] "Communication means" is a concept that refers to the technology or devices used to send and receive information.

[0075] "Personalized guidance" refers to customized information provided based on a customer's specific circumstances and history.

[0076] The invention will now be described in detail. This system streamlines the shopping experience for customers in stores and provides personalized product recommendations. The following are specific embodiments of this system.

[0077] The terminal uses an identification device to capture the face of a user entering the store and sends the image to a server. The server uses facial recognition software to compare the transmitted facial data with existing customer data to identify the user. Common facial recognition technologies used in this process include, for example, FaceNet and DeepFace.

[0078] The server retrieves purchase history and preference data of specific users stored in storage devices. This history data may be stored in either an SQL or NoSQL database. The server also aggregates information on the placement of each item within the store. This information is collected using beacons and RFID tags installed within the store.

[0079] Next, the server uses an information processing device to analyze the user's history information and store location information to generate personalized guidance information. Dijkstra's algorithm and A algorithm are used for route optimization based on location information to provide the optimal route for the user to move efficiently within the store. In addition, collaborative filtering and content-based filtering are used to recommend relevant products based on the user's preferences.

[0080] The generated guidance information is transmitted from the server to the user's terminal via a communication device. The terminal displays the received guidance information on its screen and provides voice guidance as needed to support the user's movement within the store. The voice guidance is implemented using text-to-speech technology.

[0081] As a concrete example, in a large supermarket that a user is visiting for the first time, a terminal recognizes the user's face using a camera near the entrance. At this point, the user receives location guidance for everyday items they have purchased in the past and products that match their preferences, allowing them to complete their necessary shopping in a short amount of time. An example of a prompt message could be, "Design a system that uses facial recognition technology to provide product guidance based on the user's purchase history."

[0082] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0083] Step 1:

[0084] The terminal captures the faces of users entering the store using an identification device. The input is a real-time captured image of the user's face. The output is generated as facial feature point data. This data is sent to a server and serves as the basis for identifying users using facial recognition technology. Examples of facial recognition technologies used here include FaceNet.

[0085] Step 2:

[0086] The server uses the received facial feature point data to match it with existing customer data in the database. The input includes facial feature point data and the customer's face database. The server uses a database search algorithm to find matching data and obtains an identified user ID as output. This completes the user identification process.

[0087] Step 3:

[0088] The server retrieves the user's purchase history and preference data from storage based on the identified user ID. The user ID is provided as input. By executing a database query and extracting the user's historical dataset from the database, the server can obtain past purchase history and preference data as output.

[0089] Step 4:

[0090] The server aggregates location information for items within the store. Inputs include real-time location data collected from beacons and RFID devices installed within the store. The server analyzes this data and generates up-to-date item placement information as output. This information is used to pinpoint the location of items of interest.

[0091] Step 5:

[0092] The server generates optimal guidance information using the user's history information and acquired item placement information. Input includes user history information, preference data, and item placement information. Data processing uses collaborative filtering algorithms to recommend products likely to interest the user, resulting in customized guidance information as output.

[0093] Step 6:

[0094] The server transmits the generated guidance information to the user's terminal via a communication method. Customized guidance information is used as input. As output, route guidance on a map displayed on the user's terminal and voice guidance information are provided. This allows the user to move efficiently within the store and easily find the desired product.

[0095] (Application Example 1)

[0096] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0097] In traditional stores, it was difficult and time-consuming for customers to efficiently find products that met their needs. Furthermore, it was difficult to offer product suggestions that adequately reflected customer preferences, potentially leading to decreased customer satisfaction.

[0098] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0099] In this invention, the server includes authentication means for identifying individual customer information, storage means for acquiring past customer transaction information, calculation means for acquiring product placement information within the store and creating route information based on the customer transaction information, and calculation means for generating movement guidance and recommended product information based on past transaction information and product placement information. This enables customers to quickly find the optimal products according to their preferences and purchase history, move efficiently within the store, and make purchases.

[0100] An "authentication device" is a device used to identify individual customer information and has the function of identifying customers using facial recognition technology.

[0101] A "storage device" is a memory device that stores a customer's past transaction information and allows it to be retrieved as needed.

[0102] A "processing unit" is a device that acquires information on the placement of products within a store and has the function of creating route information and recommended product information based on customer transaction information.

[0103] A "communication device" is a communication device that transmits information in order to provide routing information to a customer's device.

[0104] "Route information" refers to information that shows the optimal path for customers to move efficiently within a store and reach their desired product.

[0105] "Recommended product information" refers to information about products that are highly relevant to the customer, suggested based on the customer's past transaction history and preferences.

[0106] To realize this invention, a program using the following system components is important.

[0107] The server first identifies the individual information of users entering the store via a camera that acts as an authentication device. This is done using a facial recognition library such as OpenCV. The recognized facial data is sent to the server's database and compared against past transaction information. Firebase is an example of a database management system used in this process.

[0108] Next, the server obtains information on the placement of products within the store using location services such as the Google® Maps API. Based on this information, it generates navigation guidance and recommended product information based on the user's preferences and transaction history. This data processing is realized by analysis algorithms and computing devices performed on the server.

[0109] Subsequently, the server functions as a communication device, providing the generated route information and recommended product information to the customer's smartphone. The smartphone app visually displays the route and guides the user using voice guidance. This allows users to easily find the products they want in a short amount of time, improving their shopping experience.

[0110] For example, when a user visits a large shopping mall, the application recognizes the user through cameras near the entrance and displays the shortest route to the sports equipment they have previously purchased. It can also provide voice notifications about discounts on the latest sports equipment.

[0111] An example of a prompt used when optimizing recommended product information based on customer preferences using a generative AI model is: "Identify the product the user is looking for in the store based on their past purchase history and preferences. Then, suggest the best way to purchase that product."

[0112] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0113] Step 1:

[0114] The server enables the camera on a terminal that functions as an authentication device and captures a facial image when a user enters the store. The input is the raw facial image data captured by the camera, and the output is the facial data with extracted features. Facial recognition software such as OpenCV is used to extract features and prepare them for transmission to the database.

[0115] Step 2:

[0116] The server matches the customer's past transaction information stored in the database. The input is processed feature facial data, and the output is the profile data of the matching customer. The matching process uses a management system such as Firebase to search for past purchase history and preference information.

[0117] Step 3:

[0118] The server uses location services to obtain product placement information within the store. Input is product identification information obtained from the product's barcode or QR code (registered trademark), and output is coordinate data indicating the product's location. Real-time location information is obtained using the Google Maps API and used for in-store navigation.

[0119] Step 4:

[0120] The server uses a generative AI model to generate optimal route guidance and recommended product information based on the user's transaction history and acquired product placement information. The input is customer profile data and product placement data, and the output is route guidance information and a list of recommended products. The prompt used for the generative AI model is, "Identify the product the user is looking for based on their past purchase history and preferences, and suggest the optimal purchase route for that product."

[0121] Step 5:

[0122] The terminal acts as a communication device, providing users with route guidance information and recommended product information received from the server. Input is data transmitted from the server, and output is route information displayed on the screen and voice guidance. The terminal combines visual map display and voice guidance to enable users to efficiently navigate to their desired products.

[0123] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0124] This invention is a support system for users to visit stores and shop efficiently and comfortably. The system comprises an identification device, a memory device, a computing device, a communication device, and an emotion engine that analyzes and adjusts the user's emotions.

[0125] When a user enters a store, the terminal's identification device uses facial recognition technology to identify the user. The server retrieves the necessary data from a storage device that contains the user's purchase history and preferences. The server then obtains the location information of items within the store and generates guidance information based on the historical information. This guidance information includes recommended routes and product lists derived from past purchase history.

[0126] Furthermore, an emotion engine operates to recognize the user's emotions in real time. The emotion engine analyzes facial expressions captured by the camera and voice tone obtained through the microphone to infer the user's emotional state. For example, if the system detects that the user is irritated, the server updates the guidance information, presenting a concise route and products to ensure a smooth shopping experience. The tone of the voice guide is also adjusted according to the user's emotions, creating a sense of friendliness and reassurance.

[0127] Users can receive guidance information transmitted to their terminals via communication devices and view it on their terminals. Guidance information is provided not only on the screen but also via audio, allowing for stress-free shopping using both sight and hearing. Furthermore, the emotion engine provides product recommendations optimized for the user's emotional state.

[0128] As a concrete example, when a user enters a large supermarket, they are identified by facial recognition, and the location of recommended products is suggested based on their past purchase history. At the same time, an emotion engine senses that the user is relaxed and sets the voice guidance to a friendly tone. This allows the user to comfortably browse the store and efficiently purchase what they need.

[0129] This system allows users to improve their shopping efficiency, and enables stores to enhance the customer experience.

[0130] The following describes the processing flow.

[0131] Step 1:

[0132] A user passes through the store entrance. At that moment, a camera installed at the entrance captures the user's face.

[0133] Step 2:

[0134] The device acquires facial image data captured by the device and sends that data to the server.

[0135] Step 3:

[0136] The server uses the received facial image data to match it with the customer database and identify the user.

[0137] Step 4:

[0138] The server retrieves the purchase history and preferences of identified users from its storage device.

[0139] Step 5:

[0140] The server retrieves the latest item placement data from the store and generates guidance information based on the user's purchase history. This information includes the location of the desired item and the optimal route to take.

[0141] Step 6:

[0142] The device's built-in emotion engine uses the camera and microphone to analyze the user's facial expressions and voice, and infer their current emotional state.

[0143] Step 7:

[0144] The server adjusts the guidance information based on data from the emotion engine, according to the user's emotional state. For example, if the user is excited, the guidance will be changed to a gentler tone.

[0145] Step 8:

[0146] The server sends the coordinated guidance information to the user's terminal via a communication device.

[0147] Step 9:

[0148] The device displays the received guidance information on the screen and plays audio guides as needed.

[0149] Step 10:

[0150] Users utilize guidance from their devices to find and purchase the items they are looking for in the store.

[0151] Step 11:

[0152] The server saves the user's new purchase information to storage and updates the purchase history and preference data.

[0153] (Example 2)

[0154] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0155] Modern consumers desire an efficient and comfortable shopping experience, but the vast product range and crowds often make shopping stressful. Furthermore, while there is a demand for services that cater to consumer emotions, systems capable of analyzing and instantly responding to individual consumer sentiments are lacking.

[0156] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0157] In this invention, the server includes identification means for detecting consumer characteristics, storage means for collecting consumer history information, calculation means for collecting product location information within the store and generating guidance information based on the consumer history information, and an emotion analysis engine for analyzing the consumer's emotional state and optimizing the guidance information. This makes it possible to provide personalized services based on consumer characteristics, thereby realizing an effective and comfortable shopping experience.

[0158] A "consumer" refers to an individual who purchases or uses goods or services.

[0159] An "identification device" is a combination of hardware and software used to detect and identify the individual characteristics of a consumer.

[0160] A "memory device" is a data storage system that stores consumer history information and preferences and retrieves them as needed.

[0161] A "computational means" is a computer system that processes information to generate guidance information based on the collected information.

[0162] "Communication means" refers to the network infrastructure and protocols used by a server to transmit information to a consumer's terminal.

[0163] An "emotion analysis engine" is software that analyzes a consumer's facial expressions and tone of voice to estimate their emotional state and adjust the way a service is delivered accordingly.

[0164] "Guidance information" refers to information such as the location of products and recommended routes provided to consumers to help them shop efficiently.

[0165] This invention is an information processing system that supports consumers in shopping efficiently within a store. Specifically, it begins when the user enters the store.

[0166] When a user enters a store, an identification device on the terminal uses an image processing library to capture the user's face and identifies the individual based on a facial recognition algorithm. To do this, the terminal applies common facial recognition technology and compares it with a user database.

[0167] Next, the server accesses storage to retrieve the user's past purchase history and preference data. This retrieved data is used to recommend products that match the user's preferences. The server also uses computing resources to collect product locations from the store's location information system and generate an optimal shopping route. Wi-Fi and beacon technology are used for location processing in this process.

[0168] Furthermore, an emotion analysis engine operates to analyze the user's emotions in real time. Specifically, it analyzes the user's facial expressions using the device's camera and captures their voice tone using the built-in microphone to infer their emotions. This allows for the optimization of guidance information according to the user's current emotional state.

[0169] The generated guidance information is transmitted to the user's device via communication means. The user can visually confirm the guidance information on their device and also listen to the information through an audio guide. This audio guide is adjusted based on the emotional state estimated by the emotion analysis engine and is delivered in an appropriate tone.

[0170] As a concrete example, when a user visits a large shopping center, this system identifies the user through facial recognition and guides them to the floor with recommended products by referring to their past purchase history. At the same time, if the emotion analysis engine detects that the user is in a calm mood, the voice guide explains the shopping route in a gentle tone. This guidance allows the user to shop efficiently and comfortably within the store.

[0171] An example of a prompt message might be: "When a user is looking for the optimal shopping route in the store, provide guidance that reflects the results of sentiment analysis."

[0172] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0173] Step 1:

[0174] When a user enters a store, a camera installed on the terminal captures an image of the user's face. The facial image, as input, is processed in real time, and the output is a unique ID necessary for identifying the user. Specifically, a facial recognition algorithm analyzes the image and identifies the user by comparing it with pre-registered facial data.

[0175] Step 2:

[0176] The server takes the identified user ID as input, accesses a storage device, and retrieves the user's purchase history and preference information from the database. The output is a list of user history and preferences. This list is generated by quickly collecting data related to the user's preferences using SQL queries.

[0177] Step 3:

[0178] The server obtains product location information from the store's location information system. The input to this process is a list of products to be handled, and the output is the location information of those products. Location information is collected using tracking technologies such as Wi-Fi and beacons. Subsequently, a computing system integrates user preference information and location information to generate the optimal shopping route.

[0179] Step 4:

[0180] The device's built-in emotion analysis engine analyzes the user's emotional state in real time. Inputs are facial expression data from the camera and voice tone from the microphone, while output is a status indicating the user's emotional state. Specifically, a machine learning model analyzes this data to estimate the user's emotions.

[0181] Step 5:

[0182] The server optimizes guidance information based on the user's emotional state, adjusting the content and presentation of the information provided to the user. The input is the analyzed emotional status, and the output is the adjusted guidance information. This information is dynamically updated as the tone and display content of the audio guide.

[0183] Step 6:

[0184] The server transmits the final guidance information to the user's terminal via communication means. The input is optimized guidance information, and the output is the information displayed on the user's terminal. Users can efficiently navigate the store using visual displays or audio guidance on their terminals. In this case, the audio guidance is provided in a friendly tone that matches the user's emotions, allowing the user to shop more comfortably.

[0185] (Application Example 2)

[0186] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0187] There is a demand to provide customers with an efficient and comfortable shopping experience when they shop at physical stores. Conventional systems do not adequately optimize guidance information based on customer emotions and preferences, which can cause stress during shopping. This invention aims to improve the customer experience by analyzing the customer's real-time emotional state and adjusting guidance information based on this analysis.

[0188] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0189] In this invention, the server includes identification means for recognizing customer characteristics, storage means for acquiring customer history information, calculation means for acquiring location information of items in the store and generating guidance information, and emotion analysis means for analyzing customer emotions and optimizing guidance information. This makes it possible to provide optimal shopping guidance based on the customer's preferences and emotional state.

[0190] A "customer" is a person who visits a store and intends to purchase goods or services.

[0191] An "identification device" is a device equipped with technology to recognize the characteristics of a customer.

[0192] "Facial recognition technology" is an image processing technology used to identify a customer's face.

[0193] A "storage device" is a device used to store and retrieve customer history information.

[0194] "History information" refers to information that shows the products a customer has purchased in the past and their purchasing trends.

[0195] A "computational device" is a device that collects and analyzes data to generate the desired information.

[0196] A "communication device" is a device used to transmit information generated by a computing device to a customer's terminal.

[0197] An "emotion analysis device" is a device equipped with technology that analyzes a customer's emotional state and takes the results into consideration.

[0198] "Information" refers to information provided to support customer behavior within a store.

[0199] A "terminal" is a personal information terminal owned by a customer, and is a device that receives information provided by a communication device.

[0200] The system implementing this invention is designed to provide customers with an efficient and comfortable shopping experience when they visit a physical store. The server first recognizes the customer's characteristics using an identification device. This identification uses facial recognition technology, and a camera device acquires image data of the customer's face. The acquired data is analyzed by software called FaceRecognitionAPI to identify the customer.

[0201] Subsequently, the server retrieves the customer's past history information from storage, and the computing device combines this with location information of items within the store to generate guidance information. This process uses the StoreMapAPI, which provides store map information. The generated guidance information is optimized according to the customer's emotional state. The EmotionAnalysisAPI is used for emotion analysis, and emotion analysis is performed based on data acquired from cameras and microphones.

[0202] This optimized guidance information is transmitted to the customer's device via a communication device. The device displays the information on a smartphone application and provides audio guidance, allowing the customer to continue shopping without stress.

[0203] As a concrete example, a system could be envisioned where, upon entering an electronics store, a customer is guided to the shortest route to where their favorite brand's new products are displayed, while simultaneously receiving promotional information. In this case, if the user is relaxed, the voice guidance would be delivered in a calm and friendly tone.

[0204] An example of a prompt using a generative AI model is: "Consider a scenario for an app that detects a user's face with a camera, analyzes their emotions, and provides real-time recommendations for home appliances based on their purchase history." This prompt is used when simulating the system's emotional state-based optimization process.

[0205] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0206] Step 1:

[0207] The server acquires images of the user's face through camera devices installed in the store. The facial image is provided as input, and the image data is analyzed using the FaceRecognition API to identify the user. The output after analysis is the user's ID. Through this process, the identification device obtains the user's characteristic information.

[0208] Step 2:

[0209] The server retrieves the user's past purchase history from storage based on the identified user's ID. By using the user ID as input and searching the history database, the server obtains the user's purchase history data as output. This data is used to understand the user's preferences.

[0210] Step 3:

[0211] The server obtains location information for items within the store using the StoreMapAPI. Store map data is imported as input, and the location of items is displayed on the map as output. The computing unit integrates this map information with the user's purchase history to generate navigation information. This navigation information includes product locations and recommended routes based on the user's interests.

[0212] Step 4:

[0213] The server inputs facial and audio data acquired from the camera and microphone into the EmotionAnalysisAPI to analyze the user's emotions. The output is the user's emotional state, which the emotion analysis device uses. Based on this, the guidance information is adjusted and optimized guidance is prepared.

[0214] Step 5:

[0215] The server uses a communication device to transmit optimized guidance information to the user's terminal. The pre-adjusted guidance information is used as input, and the output includes audio and visual guides displayed on the terminal. This makes it easier for the user to navigate the store based on the information displayed on their terminal.

[0216] Step 6:

[0217] Users receive guidance information visually and audibly through a smartphone application. The guidance includes friendly voice guidance tailored to the user's emotional state. This reduces shopping stress and allows users to efficiently find the products they need.

[0218] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0219] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0220] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0221] [Second Embodiment]

[0222] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0223] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0224] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0225] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0226] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0227] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0228] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0229] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0230] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0231] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0232] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0233] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0234] This invention provides a system that enables users visiting a store to efficiently find and purchase the items they need. The system includes an identification device, a storage device for storing historical information, a computing device for acquiring location information, a communication device, and a terminal carried by the user.

[0235] First, when a user enters the store, an identification device is activated via a terminal and captures the user's face. The server then compares the captured facial data with existing customer data stored internally to identify the user. Because facial recognition technology is used in this process, the user is identified smoothly.

[0236] Next, the server retrieves user history information from storage. This history information includes past purchase history and user preferences, which are used to determine the user's needs and preferences. The server also retrieves location information for the latest products in the store. This location information indicates the placement of each item within the store.

[0237] Subsequently, the server generates personalized guidance information for the user based on historical and location data. This guidance information not only shows the location of items the user wishes to purchase, but also includes the optimal route within the store. Furthermore, it generates product recommendations tailored to the user's preferences.

[0238] The server transmits the generated guidance information to the user's terminal via a communication device. The terminal displays the received information on its screen and sometimes provides guidance to the user through voice guidance. This allows the user to efficiently move around the store and purchase the items they need. This guidance includes specific location information such as "Product XX is in the left aisle" and information such as "Seasonal items are on sale."

[0239] As a concrete example, in a large supermarket visited by a user for the first time, a camera near the entrance recognizes the user's face and guides them to the location of everyday items they have purchased in the past, allowing the user to complete their necessary shopping in a short amount of time. This system makes the user's shopping experience more efficient and comfortable, while also enabling the store to make effective product suggestions based on purchasing trends.

[0240] The following describes the processing flow.

[0241] Step 1:

[0242] The user passes through the store entrance. As a result, a camera installed at the entrance captures the user's face.

[0243] Step 2:

[0244] The device acquires facial image data captured by the device and sends that data to the server.

[0245] Step 3:

[0246] The server identifies users by matching the received facial image data with an internal customer database. A facial recognition algorithm is used to efficiently execute this process.

[0247] Step 4:

[0248] The server retrieves the user's history information from its storage device. This history information includes past purchase history and preference data.

[0249] Step 5:

[0250] The server retrieves the latest item placement data from the store management system. This data includes the precise location information of each item within the store.

[0251] Step 6:

[0252] The server generates personalized guidance information for the user based on historical data and item placement data. This information includes the location of desired items, the optimal route within the store, and even recommended products.

[0253] Step 7:

[0254] The server generates guidance information and transmits it to the user's terminal via a communication device.

[0255] Step 8:

[0256] The device displays the received guidance information on the screen. If necessary, it plays audio guidance to provide instructions in an easy-to-understand format for the user.

[0257] Step 9:

[0258] The user follows the instructions to move around the store and purchase the desired item.

[0259] Step 10:

[0260] The server saves the user's new purchase information to its storage device and updates the history information. The updated information will be used to generate future notifications.

[0261] (Example 1)

[0262] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0263] In current stores, it is difficult for customers to efficiently find the items they need, and this is particularly time-consuming in larger stores. Furthermore, product recommendations based on customers' past purchase history and preferences are not being implemented effectively, resulting in an inability to provide customers with the optimal shopping experience.

[0264] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0265] In this invention, the server includes means for photographing a customer's face and recognizing its features, storage means for acquiring the customer's purchase history and preference data, and information processing means for aggregating information on the placement of items in the store and generating personalized guidance based on the customer's history and preferences. This enables customers to quickly find the items they need and receive product suggestions that match their preferences.

[0266] The term "customer" refers to people who visit a store and receive goods or services.

[0267] A "device that photographs faces and recognizes features" is a device that uses cameras and sensors to acquire images of customers' faces and analyze their feature points.

[0268] "Purchase history" is a concept that refers to information about products and services that a customer has purchased in the past.

[0269] "Preference data" refers to data that has been accumulated to include information related to customers' preferences and interests.

[0270] The term "memory device" refers to an electronic device or mechanism used to store and retrieve information as needed.

[0271] "Item placement information" refers to information about the location and placement of each product within a store.

[0272] "Information processing means" refers to computer devices and algorithms used to analyze data and generate results tailored to specific purposes.

[0273] "Communication means" is a concept that refers to the technology or devices used to send and receive information.

[0274] "Personalized guidance" refers to customized information provided based on a customer's specific circumstances and history.

[0275] The invention will now be described in detail. This system streamlines the shopping experience for customers in stores and provides personalized product recommendations. The following are specific embodiments of this system.

[0276] The terminal uses an identification device to capture the face of a user entering the store and sends the image to a server. The server uses facial recognition software to compare the transmitted facial data with existing customer data to identify the user. Common facial recognition technologies used in this process include, for example, FaceNet and DeepFace.

[0277] The server retrieves purchase history and preference data of specific users stored in storage devices. This history data may be stored in either an SQL or NoSQL database. The server also aggregates information on the placement of each item within the store. This information is collected using beacons and RFID tags installed within the store.

[0278] Next, the server uses an information processing device to analyze the user's history information and store location information to generate personalized guidance information. Dijkstra's algorithm and A algorithm are used for route optimization based on location information to provide the optimal route for the user to move efficiently within the store. In addition, collaborative filtering and content-based filtering are used to recommend relevant products based on the user's preferences.

[0279] The generated guidance information is transmitted from the server to the user's terminal via a communication device. The terminal displays the received guidance information on its screen and provides voice guidance as needed to support the user's movement within the store. The voice guidance is implemented using text-to-speech technology.

[0280] As a concrete example, in a large supermarket that a user is visiting for the first time, a terminal recognizes the user's face using a camera near the entrance. At this point, the user receives location guidance for everyday items they have purchased in the past and products that match their preferences, allowing them to complete their necessary shopping in a short amount of time. An example of a prompt message could be, "Design a system that uses facial recognition technology to provide product guidance based on the user's purchase history."

[0281] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0282] Step 1:

[0283] The terminal captures the face of the user who enters the store using an identification device. As input, it uses the real-time acquired face image of the user. As output, facial feature point data is generated. This data is sent to the server and serves as the basic data for identifying the user using face recognition technology. Examples of the face recognition technology used here include FaceNet.

[0284] Step 2:

[0285] Based on the received facial feature point data, the server compares it with the existing customer data in the database. The input includes the facial feature point data and the customer face database. The server uses a database search algorithm to find matching data and obtains the identified user ID as output. Thus, the identification of the user is completed.

[0286] Step 3:

[0287] Based on the identified user ID, the server acquires the purchase history and preference data of the corresponding user from the storage device. The user ID is given as input. By executing a database query and extracting the user's historical data set from the database, past purchase history and preference data can be obtained as output.

[0288] Step 4:

[0289] The server aggregates the location information of the items in the store. The input includes the real-time location information data collected from beacons and RFID installed in the store. The server analyzes this data and generates the latest item placement information as output. This information is utilized to identify the location of the items of interest.

[0290] Step 5:

[0291] The server generates optimal guidance information using the user's history information and acquired item placement information. Input includes user history information, preference data, and item placement information. Data processing uses collaborative filtering algorithms to recommend products likely to interest the user, resulting in customized guidance information as output.

[0292] Step 6:

[0293] The server transmits the generated guidance information to the user's terminal via a communication method. Customized guidance information is used as input. As output, route guidance on a map displayed on the user's terminal and voice guidance information are provided. This allows the user to move efficiently within the store and easily find the desired product.

[0294] (Application Example 1)

[0295] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0296] In traditional stores, it was difficult and time-consuming for customers to efficiently find products that met their needs. Furthermore, it was difficult to offer product suggestions that adequately reflected customer preferences, potentially leading to decreased customer satisfaction.

[0297] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0298] In this invention, the server includes authentication means for identifying individual customer information, storage means for acquiring past customer transaction information, calculation means for acquiring product placement information within the store and creating route information based on the customer transaction information, and calculation means for generating movement guidance and recommended product information based on past transaction information and product placement information. This enables customers to quickly find the optimal products according to their preferences and purchase history, move efficiently within the store, and make purchases.

[0299] An "authentication device" is a device used to identify individual customer information and has the function of identifying customers using facial recognition technology.

[0300] A "storage device" is a memory device that stores a customer's past transaction information and allows it to be retrieved as needed.

[0301] A "processing unit" is a device that acquires information on the placement of products within a store and has the function of creating route information and recommended product information based on customer transaction information.

[0302] A "communication device" is a communication device that transmits information in order to provide routing information to a customer's device.

[0303] "Route information" refers to information that shows the optimal path for customers to move efficiently within a store and reach their desired product.

[0304] "Recommended product information" refers to information about products that are highly relevant to the customer, suggested based on the customer's past transaction history and preferences.

[0305] To realize this invention, a program using the following system components is important.

[0306] First, the server functions as an authentication device and uses a camera to identify the personal information of the user who enters the store. For this, a face recognition library such as OpenCV is used. The recognized face data is sent to the server's database and compared with past transaction information. As an example of the database management system used here, Firebase can be cited.

[0307] Next, the server uses a location information service such as the Google Maps API to obtain the product placement information within the store. Then, based on this information, it generates route guidance and recommended product information based on the user's preferences and transaction history. This data processing is realized by the analysis algorithms and computing devices on the server.

[0308] After that, the server functions as a communication device and provides the generated route information and recommended product information to the customer's smartphone terminal. The smartphone application visually displays the route and guides the user using voice guidance. This enables the user to easily find the products they want in a short time and improve the shopping experience.

[0309] For example, when a certain user visits a large shopping mall, the application recognizes the user through the camera near the entrance and displays the shortest route to the sports supplies he has purchased before. It can also inform the user audibly that the latest sports supplies are on sale.

[0310] As an example of the prompt text when optimizing the recommended product information according to the customer's preferences using the generated AI model, "Please identify the product that the user is looking for in the store based on past purchase history and preferences. Then, propose the optimal purchase route for that product." can be cited.

[0311] The flow of the specific process in Application Example 1 will be described using FIG. 12.

[0312] Step 1:

[0313] The server enables the camera on a terminal that functions as an authentication device and captures a facial image when a user enters the store. The input is the raw facial image data captured by the camera, and the output is the facial data with extracted features. Facial recognition software such as OpenCV is used to extract features and prepare them for transmission to the database.

[0314] Step 2:

[0315] The server matches the customer's past transaction information stored in the database. The input is processed feature facial data, and the output is the profile data of the matching customer. The matching process uses a management system such as Firebase to search for past purchase history and preference information.

[0316] Step 3:

[0317] The server uses location services to obtain product placement information within the store. Input is product identification information obtained from the product's barcode or QR code, and output is coordinate data indicating the product's location. Real-time location information is obtained using the Google Maps API and used for in-store navigation.

[0318] Step 4:

[0319] The server uses a generative AI model to generate optimal route guidance and recommended product information based on the user's transaction history and acquired product placement information. The input is customer profile data and product placement data, and the output is route guidance information and a list of recommended products. The prompt used for the generative AI model is, "Identify the product the user is looking for based on their past purchase history and preferences, and suggest the optimal purchase route for that product."

[0320] Step 5:

[0321] The terminal acts as a communication device, providing users with route guidance information and recommended product information received from the server. Input is data transmitted from the server, and output is route information displayed on the screen and voice guidance. The terminal combines visual map display and voice guidance to enable users to efficiently navigate to their desired products.

[0322] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0323] This invention is a support system for users to visit stores and shop efficiently and comfortably. The system comprises an identification device, a memory device, a computing device, a communication device, and an emotion engine that analyzes and adjusts the user's emotions.

[0324] When a user enters a store, the terminal's identification device uses facial recognition technology to identify the user. The server retrieves the necessary data from a storage device that contains the user's purchase history and preferences. The server then obtains the location information of items within the store and generates guidance information based on the historical information. This guidance information includes recommended routes and product lists derived from past purchase history.

[0325] Furthermore, an emotion engine operates to recognize the user's emotions in real time. The emotion engine analyzes facial expressions captured by the camera and voice tone obtained through the microphone to infer the user's emotional state. For example, if the system detects that the user is irritated, the server updates the guidance information, presenting a concise route and products to ensure a smooth shopping experience. The tone of the voice guide is also adjusted according to the user's emotions, creating a sense of friendliness and reassurance.

[0326] Users can receive guidance information transmitted to their terminals via communication devices and view it on their terminals. Guidance information is provided not only on the screen but also via audio, allowing for stress-free shopping using both sight and hearing. Furthermore, the emotion engine provides product recommendations optimized for the user's emotional state.

[0327] As a concrete example, when a user enters a large supermarket, they are identified by facial recognition, and the location of recommended products is suggested based on their past purchase history. At the same time, an emotion engine senses that the user is relaxed and sets the voice guidance to a friendly tone. This allows the user to comfortably browse the store and efficiently purchase what they need.

[0328] This system allows users to improve their shopping efficiency, and enables stores to enhance the customer experience.

[0329] The following describes the processing flow.

[0330] Step 1:

[0331] A user passes through the store entrance. At that moment, a camera installed at the entrance captures the user's face.

[0332] Step 2:

[0333] The device acquires facial image data captured by the device and sends that data to the server.

[0334] Step 3:

[0335] The server uses the received facial image data to match it with the customer database and identify the user.

[0336] Step 4:

[0337] The server retrieves the purchase history and preferences of identified users from its storage device.

[0338] Step 5:

[0339] The server retrieves the latest item placement data from the store and generates guidance information based on the user's purchase history. This information includes the location of the desired item and the optimal route to take.

[0340] Step 6:

[0341] The device's built-in emotion engine uses the camera and microphone to analyze the user's facial expressions and voice, and infer their current emotional state.

[0342] Step 7:

[0343] The server adjusts the guidance information based on data from the emotion engine, according to the user's emotional state. For example, if the user is excited, the guidance will be changed to a gentler tone.

[0344] Step 8:

[0345] The server sends the coordinated guidance information to the user's terminal via a communication device.

[0346] Step 9:

[0347] The device displays the received guidance information on the screen and plays audio guides as needed.

[0348] Step 10:

[0349] Users utilize guidance from their devices to find and purchase the items they are looking for in the store.

[0350] Step 11:

[0351] The server saves the user's new purchase information to storage and updates the purchase history and preference data.

[0352] (Example 2)

[0353] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0354] Modern consumers desire an efficient and comfortable shopping experience, but the vast product range and crowds often make shopping stressful. Furthermore, while there is a demand for services that cater to consumer emotions, systems capable of analyzing and instantly responding to individual consumer sentiments are lacking.

[0355] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0356] In this invention, the server includes identification means for detecting consumer characteristics, storage means for collecting consumer history information, calculation means for collecting product location information within the store and generating guidance information based on the consumer history information, and an emotion analysis engine for analyzing the consumer's emotional state and optimizing the guidance information. This makes it possible to provide personalized services based on consumer characteristics, thereby realizing an effective and comfortable shopping experience.

[0357] A "consumer" refers to an individual who purchases or uses goods or services.

[0358] An "identification device" is a combination of hardware and software used to detect and identify the individual characteristics of a consumer.

[0359] A "memory device" is a data storage system that stores consumer history information and preferences and retrieves them as needed.

[0360] A "computational means" is a computer system that processes information to generate guidance information based on the collected information.

[0361] "Communication means" refers to the network infrastructure and protocols used by a server to transmit information to a consumer's terminal.

[0362] An "emotion analysis engine" is software that analyzes a consumer's facial expressions and tone of voice to estimate their emotional state and adjust the way a service is delivered accordingly.

[0363] "Guidance information" refers to information such as the location of products and recommended routes provided to consumers to help them shop efficiently.

[0364] This invention is an information processing system that supports consumers in shopping efficiently within a store. Specifically, it begins when the user enters the store.

[0365] When a user enters a store, an identification device on the terminal uses an image processing library to capture the user's face and identifies the individual based on a facial recognition algorithm. To do this, the terminal applies common facial recognition technology and compares it with a user database.

[0366] Next, the server accesses storage to retrieve the user's past purchase history and preference data. This retrieved data is used to recommend products that match the user's preferences. The server also uses computing resources to collect product locations from the store's location information system and generate an optimal shopping route. Wi-Fi and beacon technology are used for location processing in this process.

[0367] Furthermore, an emotion analysis engine operates to analyze the user's emotions in real time. Specifically, it analyzes the user's facial expressions using the device's camera and captures their voice tone using the built-in microphone to infer their emotions. This allows for the optimization of guidance information according to the user's current emotional state.

[0368] The generated guidance information is transmitted to the user's device via communication means. The user can visually confirm the guidance information on their device and also listen to the information through an audio guide. This audio guide is adjusted based on the emotional state estimated by the emotion analysis engine and is delivered in an appropriate tone.

[0369] As a concrete example, when a user visits a large shopping center, this system identifies the user through facial recognition and guides them to the floor with recommended products by referring to their past purchase history. At the same time, if the emotion analysis engine detects that the user is in a calm mood, the voice guide explains the shopping route in a gentle tone. This guidance allows the user to shop efficiently and comfortably within the store.

[0370] An example of a prompt message might be: "When a user is looking for the optimal shopping route in the store, provide guidance that reflects the results of sentiment analysis."

[0371] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0372] Step 1:

[0373] When a user enters a store, a camera installed on the terminal captures an image of the user's face. The facial image, as input, is processed in real time, and the output is a unique ID necessary for identifying the user. Specifically, a facial recognition algorithm analyzes the image and identifies the user by comparing it with pre-registered facial data.

[0374] Step 2:

[0375] The server takes the identified user ID as input, accesses a storage device, and retrieves the user's purchase history and preference information from the database. The output is a list of user history and preferences. This list is generated by quickly collecting data related to the user's preferences using SQL queries.

[0376] Step 3:

[0377] The server obtains product location information from the store's location information system. The input to this process is a list of products to be handled, and the output is the location information of those products. Location information is collected using tracking technologies such as Wi-Fi and beacons. Subsequently, a computing system integrates user preference information and location information to generate the optimal shopping route.

[0378] Step 4:

[0379] The device's built-in emotion analysis engine analyzes the user's emotional state in real time. Inputs are facial expression data from the camera and voice tone from the microphone, while output is a status indicating the user's emotional state. Specifically, a machine learning model analyzes this data to estimate the user's emotions.

[0380] Step 5:

[0381] The server optimizes guidance information based on the user's emotional state, adjusting the content and presentation of the information provided to the user. The input is the analyzed emotional status, and the output is the adjusted guidance information. This information is dynamically updated as the tone and display content of the audio guide.

[0382] Step 6:

[0383] The server transmits the final guidance information to the user's terminal via communication means. The input is optimized guidance information, and the output is the information displayed on the user's terminal. Users can efficiently navigate the store using visual displays or audio guidance on their terminals. In this case, the audio guidance is provided in a friendly tone that matches the user's emotions, allowing the user to shop more comfortably.

[0384] (Application Example 2)

[0385] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0386] There is a demand to provide customers with an efficient and comfortable shopping experience when they shop at physical stores. Conventional systems do not adequately optimize guidance information based on customer emotions and preferences, which can cause stress during shopping. This invention aims to improve the customer experience by analyzing the customer's real-time emotional state and adjusting guidance information based on this analysis.

[0387] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0388] In this invention, the server includes identification means for recognizing customer characteristics, storage means for acquiring customer history information, calculation means for acquiring location information of items in the store and generating guidance information, and emotion analysis means for analyzing customer emotions and optimizing guidance information. This makes it possible to provide optimal shopping guidance based on the customer's preferences and emotional state.

[0389] A "customer" is a person who visits a store and intends to purchase goods or services.

[0390] An "identification device" is a device equipped with technology to recognize the characteristics of a customer.

[0391] "Facial recognition technology" is an image processing technology used to identify a customer's face.

[0392] A "storage device" is a device used to store and retrieve customer history information.

[0393] "History information" refers to information that shows the products a customer has purchased in the past and their purchasing trends.

[0394] A "computational device" is a device that collects and analyzes data to generate the desired information.

[0395] A "communication device" is a device used to transmit information generated by a computing device to a customer's terminal.

[0396] An "emotion analysis device" is a device equipped with technology that analyzes a customer's emotional state and takes the results into consideration.

[0397] "Information" refers to information provided to support customer behavior within a store.

[0398] A "terminal" is a personal information terminal owned by a customer, and is a device that receives information provided by a communication device.

[0399] The system implementing this invention is designed to provide customers with an efficient and comfortable shopping experience when they visit a physical store. The server first recognizes the customer's characteristics using an identification device. This identification uses facial recognition technology, and a camera device acquires image data of the customer's face. The acquired data is analyzed by software called FaceRecognitionAPI to identify the customer.

[0400] Subsequently, the server retrieves the customer's past history information from storage, and the computing device combines this with location information of items within the store to generate guidance information. This process uses the StoreMapAPI, which provides store map information. The generated guidance information is optimized according to the customer's emotional state. The EmotionAnalysisAPI is used for emotion analysis, and emotion analysis is performed based on data acquired from cameras and microphones.

[0401] This optimized guidance information is transmitted to the customer's device via a communication device. The device displays the information on a smartphone application and provides audio guidance, allowing the customer to continue shopping without stress.

[0402] As a concrete example, a system could be envisioned where, upon entering an electronics store, a customer is guided to the shortest route to where their favorite brand's new products are displayed, while simultaneously receiving promotional information. In this case, if the user is relaxed, the voice guidance would be delivered in a calm and friendly tone.

[0403] An example of a prompt using a generative AI model is: "Consider a scenario for an app that detects a user's face with a camera, analyzes their emotions, and provides real-time recommendations for home appliances based on their purchase history." This prompt is used when simulating the system's emotional state-based optimization process.

[0404] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0405] Step 1:

[0406] The server acquires images of the user's face through camera devices installed in the store. The facial image is provided as input, and the image data is analyzed using the FaceRecognition API to identify the user. The output after analysis is the user's ID. Through this process, the identification device obtains the user's characteristic information.

[0407] Step 2:

[0408] The server retrieves the user's past purchase history from storage based on the identified user's ID. By using the user ID as input and searching the history database, the server obtains the user's purchase history data as output. This data is used to understand the user's preferences.

[0409] Step 3:

[0410] The server obtains location information for items within the store using the StoreMapAPI. Store map data is imported as input, and the location of items is displayed on the map as output. The computing unit integrates this map information with the user's purchase history to generate navigation information. This navigation information includes product locations and recommended routes based on the user's interests.

[0411] Step 4:

[0412] The server inputs facial and audio data acquired from the camera and microphone into the EmotionAnalysisAPI to analyze the user's emotions. The output is the user's emotional state, which the emotion analysis device uses. Based on this, the guidance information is adjusted and optimized guidance is prepared.

[0413] Step 5:

[0414] The server uses a communication device to transmit optimized guidance information to the user's terminal. The pre-adjusted guidance information is used as input, and the output includes audio and visual guides displayed on the terminal. This makes it easier for the user to navigate the store based on the information displayed on their terminal.

[0415] Step 6:

[0416] Users receive guidance information visually and audibly through a smartphone application. The guidance includes friendly voice guidance tailored to the user's emotional state. This reduces shopping stress and allows users to efficiently find the products they need.

[0417] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0418] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet Search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0419] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0420] [Third Embodiment]

[0421] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0422] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0423] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0424] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0425] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0426] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0427] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0428] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0429] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0430] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0431] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0432] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0433] This invention provides a system that enables users visiting a store to efficiently find and purchase the items they need. The system includes an identification device, a storage device for storing historical information, a computing device for acquiring location information, a communication device, and a terminal carried by the user.

[0434] First, when a user enters the store, an identification device is activated via a terminal and captures the user's face. The server then compares the captured facial data with existing customer data stored internally to identify the user. Because facial recognition technology is used in this process, the user is identified smoothly.

[0435] Next, the server retrieves user history information from storage. This history information includes past purchase history and user preferences, which are used to determine the user's needs and preferences. The server also retrieves location information for the latest products in the store. This location information indicates the placement of each item within the store.

[0436] Subsequently, the server generates personalized guidance information for the user based on historical and location data. This guidance information not only shows the location of items the user wishes to purchase, but also includes the optimal route within the store. Furthermore, it generates product recommendations tailored to the user's preferences.

[0437] The server transmits the generated guidance information to the user's terminal via a communication device. The terminal displays the received information on its screen and sometimes provides guidance to the user through voice guidance. This allows the user to efficiently move around the store and purchase the items they need. This guidance includes specific location information such as "Product XX is in the left aisle" and information such as "Seasonal items are on sale."

[0438] As a concrete example, in a large supermarket visited by a user for the first time, a camera near the entrance recognizes the user's face and guides them to the location of everyday items they have purchased in the past, allowing the user to complete their necessary shopping in a short amount of time. This system makes the user's shopping experience more efficient and comfortable, while also enabling the store to make effective product suggestions based on purchasing trends.

[0439] The following describes the processing flow.

[0440] Step 1:

[0441] The user passes through the store entrance. As a result, a camera installed at the entrance captures the user's face.

[0442] Step 2:

[0443] The device acquires facial image data captured by the device and sends that data to the server.

[0444] Step 3:

[0445] The server identifies users by matching the received facial image data with an internal customer database. A facial recognition algorithm is used to efficiently execute this process.

[0446] Step 4:

[0447] The server retrieves the user's history information from its storage device. This history information includes past purchase history and preference data.

[0448] Step 5:

[0449] The server retrieves the latest item placement data from the store management system. This data includes the precise location information of each item within the store.

[0450] Step 6:

[0451] The server generates personalized guidance information for the user based on historical data and item placement data. This information includes the location of desired items, the optimal route within the store, and even recommended products.

[0452] Step 7:

[0453] The server generates guidance information and transmits it to the user's terminal via a communication device.

[0454] Step 8:

[0455] The device displays the received guidance information on the screen. If necessary, it plays audio guidance to provide instructions in an easy-to-understand format for the user.

[0456] Step 9:

[0457] The user follows the instructions to move around the store and purchase the desired item.

[0458] Step 10:

[0459] The server saves the user's new purchase information to its storage device and updates the history information. The updated information will be used to generate future notifications.

[0460] (Example 1)

[0461] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0462] In current stores, it is difficult for customers to efficiently find the items they need, and this is particularly time-consuming in larger stores. Furthermore, product recommendations based on customers' past purchase history and preferences are not being implemented effectively, resulting in an inability to provide customers with the optimal shopping experience.

[0463] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0464] In this invention, the server includes means for photographing a customer's face and recognizing its features, storage means for acquiring the customer's purchase history and preference data, and information processing means for aggregating information on the placement of items in the store and generating personalized guidance based on the customer's history and preferences. This enables customers to quickly find the items they need and receive product suggestions that match their preferences.

[0465] The term "customer" refers to people who visit a store and receive goods or services.

[0466] A "device that photographs faces and recognizes features" is a device that uses cameras and sensors to acquire images of customers' faces and analyze their feature points.

[0467] "Purchase history" is a concept that refers to information about products and services that a customer has purchased in the past.

[0468] "Preference data" refers to data that has been accumulated to include information related to customers' preferences and interests.

[0469] The term "memory device" refers to an electronic device or mechanism used to store and retrieve information as needed.

[0470] "Item placement information" refers to information about the location and placement of each product within a store.

[0471] "Information processing means" refers to computer devices and algorithms used to analyze data and generate results tailored to specific purposes.

[0472] "Communication means" is a concept that refers to the technology or devices used to send and receive information.

[0473] "Personalized guidance" refers to customized information provided based on a customer's specific circumstances and history.

[0474] The invention will now be described in detail. This system streamlines the shopping experience for customers in stores and provides personalized product recommendations. The following are specific embodiments of this system.

[0475] The terminal uses an identification device to capture the face of a user entering the store and sends the image to a server. The server uses facial recognition software to compare the transmitted facial data with existing customer data to identify the user. Common facial recognition technologies used in this process include, for example, FaceNet and DeepFace.

[0476] The server retrieves purchase history and preference data of specific users stored in storage devices. This history data may be stored in either an SQL or NoSQL database. The server also aggregates information on the placement of each item within the store. This information is collected using beacons and RFID tags installed within the store.

[0477] Next, the server uses an information processing device to analyze the user's history information and store location information to generate personalized guidance information. Dijkstra's algorithm and A algorithm are used for route optimization based on location information to provide the optimal route for the user to move efficiently within the store. In addition, collaborative filtering and content-based filtering are used to recommend relevant products based on the user's preferences.

[0478] The generated guidance information is transmitted from the server to the user's terminal via a communication device. The terminal displays the received guidance information on its screen and provides voice guidance as needed to support the user's movement within the store. The voice guidance is implemented using text-to-speech technology.

[0479] As a concrete example, in a large supermarket that a user is visiting for the first time, a terminal recognizes the user's face using a camera near the entrance. At this point, the user receives location guidance for everyday items they have purchased in the past and products that match their preferences, allowing them to complete their necessary shopping in a short amount of time. An example of a prompt message could be, "Design a system that uses facial recognition technology to provide product guidance based on the user's purchase history."

[0480] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0481] Step 1:

[0482] The terminal captures the faces of users entering the store using an identification device. The input is a real-time captured image of the user's face. The output is generated as facial feature point data. This data is sent to a server and serves as the basis for identifying users using facial recognition technology. Examples of facial recognition technologies used here include FaceNet.

[0483] Step 2:

[0484] The server uses the received facial feature point data to match it with existing customer data in the database. The input includes facial feature point data and the customer's face database. The server uses a database search algorithm to find matching data and obtains an identified user ID as output. This completes the user identification process.

[0485] Step 3:

[0486] The server retrieves the user's purchase history and preference data from storage based on the identified user ID. The user ID is provided as input. By executing a database query and extracting the user's historical dataset from the database, the server can obtain past purchase history and preference data as output.

[0487] Step 4:

[0488] The server aggregates location information for items within the store. Inputs include real-time location data collected from beacons and RFID devices installed within the store. The server analyzes this data and generates up-to-date item placement information as output. This information is used to pinpoint the location of items of interest.

[0489] Step 5:

[0490] The server generates optimal guidance information using the user's history information and acquired item placement information. Input includes user history information, preference data, and item placement information. Data processing uses collaborative filtering algorithms to recommend products likely to interest the user, resulting in customized guidance information as output.

[0491] Step 6:

[0492] The server transmits the generated guidance information to the user's terminal via a communication method. Customized guidance information is used as input. As output, route guidance on a map displayed on the user's terminal and voice guidance information are provided. This allows the user to move efficiently within the store and easily find the desired product.

[0493] (Application Example 1)

[0494] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0495] In traditional stores, it was difficult and time-consuming for customers to efficiently find products that met their needs. Furthermore, it was difficult to offer product suggestions that adequately reflected customer preferences, potentially leading to decreased customer satisfaction.

[0496] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0497] In this invention, the server includes authentication means for identifying individual customer information, storage means for acquiring past customer transaction information, calculation means for acquiring product placement information within the store and creating route information based on the customer transaction information, and calculation means for generating movement guidance and recommended product information based on past transaction information and product placement information. This enables customers to quickly find the optimal products according to their preferences and purchase history, move efficiently within the store, and make purchases.

[0498] An "authentication device" is a device used to identify individual customer information and has the function of identifying customers using facial recognition technology.

[0499] A "storage device" is a memory device that stores a customer's past transaction information and allows it to be retrieved as needed.

[0500] A "processing unit" is a device that acquires information on the placement of products within a store and has the function of creating route information and recommended product information based on customer transaction information.

[0501] A "communication device" is a communication device that transmits information in order to provide routing information to a customer's device.

[0502] "Route information" refers to information that shows the optimal path for customers to move efficiently within a store and reach their desired product.

[0503] "Recommended product information" refers to information about products that are highly relevant to the customer, suggested based on the customer's past transaction history and preferences.

[0504] To realize this invention, a program using the following system components is important.

[0505] The server first identifies the individual information of users entering the store via a camera that acts as an authentication device. This is done using a facial recognition library such as OpenCV. The recognized facial data is sent to the server's database and compared against past transaction information. Firebase is an example of a database management system used in this process.

[0506] Next, the server obtains information about the placement of products within the store using location services such as the Google Maps API. Based on this information, it generates navigation guidance and recommended product information based on the user's preferences and transaction history. This data processing is realized by analysis algorithms and computing devices performed on the server.

[0507] Subsequently, the server functions as a communication device, providing the generated route information and recommended product information to the customer's smartphone. The smartphone app visually displays the route and guides the user using voice guidance. This allows users to easily find the products they want in a short amount of time, improving their shopping experience.

[0508] For example, when a user visits a large shopping mall, the application recognizes the user through cameras near the entrance and displays the shortest route to the sports equipment they have previously purchased. It can also provide voice notifications about discounts on the latest sports equipment.

[0509] An example of a prompt used when optimizing recommended product information based on customer preferences using a generative AI model is: "Identify the product the user is looking for in the store based on their past purchase history and preferences. Then, suggest the best way to purchase that product."

[0510] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0511] Step 1:

[0512] The server enables the camera on a terminal that functions as an authentication device and captures a facial image when a user enters the store. The input is the raw facial image data captured by the camera, and the output is the facial data with extracted features. Facial recognition software such as OpenCV is used to extract features and prepare them for transmission to the database.

[0513] Step 2:

[0514] The server matches the customer's past transaction information stored in the database. The input is processed feature facial data, and the output is the profile data of the matching customer. The matching process uses a management system such as Firebase to search for past purchase history and preference information.

[0515] Step 3:

[0516] The server uses location services to obtain product placement information within the store. Input is product identification information obtained from the product's barcode or QR code, and output is coordinate data indicating the product's location. Real-time location information is obtained using the Google Maps API and used for in-store navigation.

[0517] Step 4:

[0518] The server uses a generative AI model to generate optimal route guidance and recommended product information based on the user's transaction history and acquired product placement information. The input is customer profile data and product placement data, and the output is route guidance information and a list of recommended products. The prompt used for the generative AI model is, "Identify the product the user is looking for based on their past purchase history and preferences, and suggest the optimal purchase route for that product."

[0519] Step 5:

[0520] The terminal acts as a communication device, providing users with route guidance information and recommended product information received from the server. Input is data transmitted from the server, and output is route information displayed on the screen and voice guidance. The terminal combines visual map display and voice guidance to enable users to efficiently navigate to their desired products.

[0521] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0522] This invention is a support system for users to visit stores and shop efficiently and comfortably. The system comprises an identification device, a memory device, a computing device, a communication device, and an emotion engine that analyzes and adjusts the user's emotions.

[0523] When a user enters a store, the terminal's identification device uses facial recognition technology to identify the user. The server retrieves the necessary data from a storage device that contains the user's purchase history and preferences. The server then obtains the location information of items within the store and generates guidance information based on the historical information. This guidance information includes recommended routes and product lists derived from past purchase history.

[0524] Furthermore, an emotion engine operates to recognize the user's emotions in real time. The emotion engine analyzes facial expressions captured by the camera and voice tone obtained through the microphone to infer the user's emotional state. For example, if the system detects that the user is irritated, the server updates the guidance information, presenting a concise route and products to ensure a smooth shopping experience. The tone of the voice guide is also adjusted according to the user's emotions, creating a sense of friendliness and reassurance.

[0525] Users can receive guidance information transmitted to their terminals via communication devices and view it on their terminals. Guidance information is provided not only on the screen but also via audio, allowing for stress-free shopping using both sight and hearing. Furthermore, the emotion engine provides product recommendations optimized for the user's emotional state.

[0526] As a concrete example, when a user enters a large supermarket, they are identified by facial recognition, and the location of recommended products is suggested based on their past purchase history. At the same time, an emotion engine senses that the user is relaxed and sets the voice guidance to a friendly tone. This allows the user to comfortably browse the store and efficiently purchase what they need.

[0527] This system allows users to improve their shopping efficiency, and enables stores to enhance the customer experience.

[0528] The following describes the processing flow.

[0529] Step 1:

[0530] A user passes through the store entrance. At that moment, a camera installed at the entrance captures the user's face.

[0531] Step 2:

[0532] The device acquires facial image data captured by the device and sends that data to the server.

[0533] Step 3:

[0534] The server uses the received facial image data to match it with the customer database and identify the user.

[0535] Step 4:

[0536] The server retrieves the purchase history and preferences of identified users from its storage device.

[0537] Step 5:

[0538] The server retrieves the latest item placement data from the store and generates guidance information based on the user's purchase history. This information includes the location of the desired item and the optimal route to take.

[0539] Step 6:

[0540] The device's built-in emotion engine uses the camera and microphone to analyze the user's facial expressions and voice, and infer their current emotional state.

[0541] Step 7:

[0542] The server adjusts the guidance information based on data from the emotion engine, according to the user's emotional state. For example, if the user is excited, the guidance will be changed to a gentler tone.

[0543] Step 8:

[0544] The server sends the coordinated guidance information to the user's terminal via a communication device.

[0545] Step 9:

[0546] The device displays the received guidance information on the screen and plays audio guides as needed.

[0547] Step 10:

[0548] Users utilize guidance from their devices to find and purchase the items they are looking for in the store.

[0549] Step 11:

[0550] The server saves the user's new purchase information to storage and updates the purchase history and preference data.

[0551] (Example 2)

[0552] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0553] Modern consumers desire an efficient and comfortable shopping experience, but the vast product range and crowds often make shopping stressful. Furthermore, while there is a demand for services that cater to consumer emotions, systems capable of analyzing and instantly responding to individual consumer sentiments are lacking.

[0554] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0555] In this invention, the server includes identification means for detecting consumer characteristics, storage means for collecting consumer history information, calculation means for collecting product location information within the store and generating guidance information based on the consumer history information, and an emotion analysis engine for analyzing the consumer's emotional state and optimizing the guidance information. This makes it possible to provide personalized services based on consumer characteristics, thereby realizing an effective and comfortable shopping experience.

[0556] A "consumer" refers to an individual who purchases or uses goods or services.

[0557] An "identification device" is a combination of hardware and software used to detect and identify the individual characteristics of a consumer.

[0558] A "memory device" is a data storage system that stores consumer history information and preferences and retrieves them as needed.

[0559] A "computational means" is a computer system that processes information to generate guidance information based on the collected information.

[0560] "Communication means" refers to the network infrastructure and protocols used by a server to transmit information to a consumer's terminal.

[0561] An "emotion analysis engine" is software that analyzes a consumer's facial expressions and tone of voice to estimate their emotional state and adjust the way a service is delivered accordingly.

[0562] "Guidance information" refers to information such as the location of products and recommended routes provided to consumers to help them shop efficiently.

[0563] This invention is an information processing system that supports consumers in shopping efficiently within a store. Specifically, it begins when the user enters the store.

[0564] When a user enters a store, an identification device on the terminal uses an image processing library to capture the user's face and identifies the individual based on a facial recognition algorithm. To do this, the terminal applies common facial recognition technology and compares it with a user database.

[0565] Next, the server accesses storage to retrieve the user's past purchase history and preference data. This retrieved data is used to recommend products that match the user's preferences. The server also uses computing resources to collect product locations from the store's location information system and generate an optimal shopping route. Wi-Fi and beacon technology are used for location processing in this process.

[0566] Furthermore, an emotion analysis engine operates to analyze the user's emotions in real time. Specifically, it analyzes the user's facial expressions using the device's camera and captures their voice tone using the built-in microphone to infer their emotions. This allows for the optimization of guidance information according to the user's current emotional state.

[0567] The generated guidance information is transmitted to the user's device via communication means. The user can visually confirm the guidance information on their device and also listen to the information through an audio guide. This audio guide is adjusted based on the emotional state estimated by the emotion analysis engine and is delivered in an appropriate tone.

[0568] As a concrete example, when a user visits a large shopping center, this system identifies the user through facial recognition and guides them to the floor with recommended products by referring to their past purchase history. At the same time, if the emotion analysis engine detects that the user is in a calm mood, the voice guide explains the shopping route in a gentle tone. This guidance allows the user to shop efficiently and comfortably within the store.

[0569] An example of a prompt message might be: "When a user is looking for the optimal shopping route in the store, provide guidance that reflects the results of sentiment analysis."

[0570] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0571] Step 1:

[0572] When a user enters a store, a camera installed on the terminal captures an image of the user's face. The facial image, as input, is processed in real time, and the output is a unique ID necessary for identifying the user. Specifically, a facial recognition algorithm analyzes the image and identifies the user by comparing it with pre-registered facial data.

[0573] Step 2:

[0574] The server takes the identified user ID as input, accesses a storage device, and retrieves the user's purchase history and preference information from the database. The output is a list of user history and preferences. This list is generated by quickly collecting data related to the user's preferences using SQL queries.

[0575] Step 3:

[0576] The server obtains product location information from the store's location information system. The input to this process is a list of products to be handled, and the output is the location information of those products. Location information is collected using tracking technologies such as Wi-Fi and beacons. Subsequently, a computing system integrates user preference information and location information to generate the optimal shopping route.

[0577] Step 4:

[0578] The device's built-in emotion analysis engine analyzes the user's emotional state in real time. Inputs are facial expression data from the camera and voice tone from the microphone, while output is a status indicating the user's emotional state. Specifically, a machine learning model analyzes this data to estimate the user's emotions.

[0579] Step 5:

[0580] The server optimizes guidance information based on the user's emotional state, adjusting the content and presentation of the information provided to the user. The input is the analyzed emotional status, and the output is the adjusted guidance information. This information is dynamically updated as the tone and display content of the audio guide.

[0581] Step 6:

[0582] The server transmits the final guidance information to the user's terminal via communication means. The input is optimized guidance information, and the output is the information displayed on the user's terminal. Users can efficiently navigate the store using visual displays or audio guidance on their terminals. In this case, the audio guidance is provided in a friendly tone that matches the user's emotions, allowing the user to shop more comfortably.

[0583] (Application Example 2)

[0584] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0585] There is a demand to provide customers with an efficient and comfortable shopping experience when they shop at physical stores. Conventional systems do not adequately optimize guidance information based on customer emotions and preferences, which can cause stress during shopping. This invention aims to improve the customer experience by analyzing the customer's real-time emotional state and adjusting guidance information based on this analysis.

[0586] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0587] In this invention, the server includes identification means for recognizing customer characteristics, storage means for acquiring customer history information, calculation means for acquiring location information of items in the store and generating guidance information, and emotion analysis means for analyzing customer emotions and optimizing guidance information. This makes it possible to provide optimal shopping guidance based on the customer's preferences and emotional state.

[0588] A "customer" is a person who visits a store and intends to purchase goods or services.

[0589] An "identification device" is a device equipped with technology to recognize the characteristics of a customer.

[0590] "Facial recognition technology" is an image processing technology used to identify a customer's face.

[0591] A "storage device" is a device used to store and retrieve customer history information.

[0592] "History information" refers to information that shows the products a customer has purchased in the past and their purchasing trends.

[0593] A "computational device" is a device that collects and analyzes data to generate the desired information.

[0594] A "communication device" is a device used to transmit information generated by a computing device to a customer's terminal.

[0595] An "emotion analysis device" is a device equipped with technology that analyzes a customer's emotional state and takes the results into consideration.

[0596] "Information" refers to information provided to support customer behavior within a store.

[0597] A "terminal" is a personal information terminal owned by a customer, and is a device that receives information provided by a communication device.

[0598] The system implementing this invention is designed to provide customers with an efficient and comfortable shopping experience when they visit a physical store. The server first recognizes the customer's characteristics using an identification device. This identification uses facial recognition technology, and a camera device acquires image data of the customer's face. The acquired data is analyzed by software called FaceRecognitionAPI to identify the customer.

[0599] Subsequently, the server retrieves the customer's past history information from storage, and the computing device combines this with location information of items within the store to generate guidance information. This process uses the StoreMapAPI, which provides store map information. The generated guidance information is optimized according to the customer's emotional state. The EmotionAnalysisAPI is used for emotion analysis, and emotion analysis is performed based on data acquired from cameras and microphones.

[0600] This optimized guidance information is transmitted to the customer's device via a communication device. The device displays the information on a smartphone application and provides audio guidance, allowing the customer to continue shopping without stress.

[0601] As a concrete example, a system could be envisioned where, upon entering an electronics store, a customer is guided to the shortest route to where their favorite brand's new products are displayed, while simultaneously receiving promotional information. In this case, if the user is relaxed, the voice guidance would be delivered in a calm and friendly tone.

[0602] An example of a prompt using a generative AI model is: "Consider a scenario for an app that detects a user's face with a camera, analyzes their emotions, and provides real-time recommendations for home appliances based on their purchase history." This prompt is used when simulating the system's emotional state-based optimization process.

[0603] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0604] Step 1:

[0605] The server acquires images of the user's face through camera devices installed in the store. The facial image is provided as input, and the image data is analyzed using the FaceRecognition API to identify the user. The output after analysis is the user's ID. Through this process, the identification device obtains the user's characteristic information.

[0606] Step 2:

[0607] The server retrieves the user's past purchase history from storage based on the identified user's ID. By using the user ID as input and searching the history database, the server obtains the user's purchase history data as output. This data is used to understand the user's preferences.

[0608] Step 3:

[0609] The server obtains location information for items within the store using the StoreMapAPI. Store map data is imported as input, and the location of items is displayed on the map as output. The computing unit integrates this map information with the user's purchase history to generate navigation information. This navigation information includes product locations and recommended routes based on the user's interests.

[0610] Step 4:

[0611] The server inputs facial and audio data acquired from the camera and microphone into the EmotionAnalysisAPI to analyze the user's emotions. The output is the user's emotional state, which the emotion analysis device uses. Based on this, the guidance information is adjusted and optimized guidance is prepared.

[0612] Step 5:

[0613] The server uses a communication device to transmit optimized guidance information to the user's terminal. The pre-adjusted guidance information is used as input, and the output includes audio and visual guides displayed on the terminal. This makes it easier for the user to navigate the store based on the information displayed on their terminal.

[0614] Step 6:

[0615] Users receive guidance information visually and audibly through a smartphone application. The guidance includes friendly voice guidance tailored to the user's emotional state. This reduces shopping stress and allows users to efficiently find the products they need.

[0616] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0617] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet Search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0618] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0619] [Fourth Embodiment]

[0620] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0621] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0622] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0623] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0624] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0625] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0626] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0627] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0628] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0629] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0630] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0631] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0632] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0633] This invention provides a system that enables users visiting a store to efficiently find and purchase the items they need. The system includes an identification device, a storage device for storing historical information, a computing device for acquiring location information, a communication device, and a terminal carried by the user.

[0634] First, when a user enters the store, an identification device is activated via a terminal and captures the user's face. The server then compares the captured facial data with existing customer data stored internally to identify the user. Because facial recognition technology is used in this process, the user is identified smoothly.

[0635] Next, the server retrieves user history information from storage. This history information includes past purchase history and user preferences, which are used to determine the user's needs and preferences. The server also retrieves location information for the latest products in the store. This location information indicates the placement of each item within the store.

[0636] Subsequently, the server generates personalized guidance information for the user based on historical and location data. This guidance information not only shows the location of items the user wishes to purchase, but also includes the optimal route within the store. Furthermore, it generates product recommendations tailored to the user's preferences.

[0637] The server transmits the generated guidance information to the user's terminal via a communication device. The terminal displays the received information on its screen and sometimes provides guidance to the user through voice guidance. This allows the user to efficiently move around the store and purchase the items they need. This guidance includes specific location information such as "Product XX is in the left aisle" and information such as "Seasonal items are on sale."

[0638] As a concrete example, in a large supermarket visited by a user for the first time, a camera near the entrance recognizes the user's face and guides them to the location of everyday items they have purchased in the past, allowing the user to complete their necessary shopping in a short amount of time. This system makes the user's shopping experience more efficient and comfortable, while also enabling the store to make effective product suggestions based on purchasing trends.

[0639] The following describes the processing flow.

[0640] Step 1:

[0641] The user passes through the store entrance. As a result, a camera installed at the entrance captures the user's face.

[0642] Step 2:

[0643] The device acquires facial image data captured by the device and sends that data to the server.

[0644] Step 3:

[0645] The server identifies users by matching the received facial image data with an internal customer database. A facial recognition algorithm is used to efficiently execute this process.

[0646] Step 4:

[0647] The server retrieves the user's history information from its storage device. This history information includes past purchase history and preference data.

[0648] Step 5:

[0649] The server retrieves the latest item placement data from the store management system. This data includes the precise location information of each item within the store.

[0650] Step 6:

[0651] The server generates personalized guidance information for the user based on historical data and item placement data. This information includes the location of desired items, the optimal route within the store, and even recommended products.

[0652] Step 7:

[0653] The server generates guidance information and transmits it to the user's terminal via a communication device.

[0654] Step 8:

[0655] The device displays the received guidance information on the screen. If necessary, it plays audio guidance to provide instructions in an easy-to-understand format for the user.

[0656] Step 9:

[0657] The user follows the instructions to move around the store and purchase the desired item.

[0658] Step 10:

[0659] The server saves the user's new purchase information to its storage device and updates the history information. The updated information will be used to generate future notifications.

[0660] (Example 1)

[0661] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0662] In current stores, it is difficult for customers to efficiently find the items they need, and this is particularly time-consuming in larger stores. Furthermore, product recommendations based on customers' past purchase history and preferences are not being implemented effectively, resulting in an inability to provide customers with the optimal shopping experience.

[0663] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0664] In this invention, the server includes means for photographing a customer's face and recognizing its features, storage means for acquiring the customer's purchase history and preference data, and information processing means for aggregating information on the placement of items in the store and generating personalized guidance based on the customer's history and preferences. This enables customers to quickly find the items they need and receive product suggestions that match their preferences.

[0665] The term "customer" refers to people who visit a store and receive goods or services.

[0666] A "device that photographs faces and recognizes features" is a device that uses cameras and sensors to acquire images of customers' faces and analyze their feature points.

[0667] "Purchase history" is a concept that refers to information about products and services that a customer has purchased in the past.

[0668] "Preference data" refers to data that has been accumulated to include information related to customers' preferences and interests.

[0669] The term "memory device" refers to an electronic device or mechanism used to store and retrieve information as needed.

[0670] "Item placement information" refers to information about the location and placement of each product within a store.

[0671] "Information processing means" refers to computer devices and algorithms used to analyze data and generate results tailored to specific purposes.

[0672] "Communication means" is a concept that refers to the technology or devices used to send and receive information.

[0673] "Personalized guidance" refers to customized information provided based on a customer's specific circumstances and history.

[0674] The invention will now be described in detail. This system streamlines the shopping experience for customers in stores and provides personalized product recommendations. The following are specific embodiments of this system.

[0675] The terminal uses an identification device to capture the face of a user entering the store and sends the image to a server. The server uses facial recognition software to compare the transmitted facial data with existing customer data to identify the user. Common facial recognition technologies used in this process include, for example, FaceNet and DeepFace.

[0676] The server retrieves purchase history and preference data of specific users stored in storage devices. This history data may be stored in either an SQL or NoSQL database. The server also aggregates information on the placement of each item within the store. This information is collected using beacons and RFID tags installed within the store.

[0677] Next, the server uses an information processing device to analyze the user's history information and store location information to generate personalized guidance information. Dijkstra's algorithm and A algorithm are used for route optimization based on location information to provide the optimal route for the user to move efficiently within the store. In addition, collaborative filtering and content-based filtering are used to recommend relevant products based on the user's preferences.

[0678] The generated guidance information is transmitted from the server to the user's terminal via a communication device. The terminal displays the received guidance information on its screen and provides voice guidance as needed to support the user's movement within the store. The voice guidance is implemented using text-to-speech technology.

[0679] As a concrete example, in a large supermarket that a user is visiting for the first time, a terminal recognizes the user's face using a camera near the entrance. At this point, the user receives location guidance for everyday items they have purchased in the past and products that match their preferences, allowing them to complete their necessary shopping in a short amount of time. An example of a prompt message could be, "Design a system that uses facial recognition technology to provide product guidance based on the user's purchase history."

[0680] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0681] Step 1:

[0682] The terminal captures the faces of users entering the store using an identification device. The input is a real-time captured image of the user's face. The output is generated as facial feature point data. This data is sent to a server and serves as the basis for identifying users using facial recognition technology. Examples of facial recognition technologies used here include FaceNet.

[0683] Step 2:

[0684] The server uses the received facial feature point data to match it with existing customer data in the database. The input includes facial feature point data and the customer's face database. The server uses a database search algorithm to find matching data and obtains an identified user ID as output. This completes the user identification process.

[0685] Step 3:

[0686] The server retrieves the user's purchase history and preference data from storage based on the identified user ID. The user ID is provided as input. By executing a database query and extracting the user's historical dataset from the database, the server can obtain past purchase history and preference data as output.

[0687] Step 4:

[0688] The server aggregates location information for items within the store. Inputs include real-time location data collected from beacons and RFID devices installed within the store. The server analyzes this data and generates up-to-date item placement information as output. This information is used to pinpoint the location of items of interest.

[0689] Step 5:

[0690] The server generates optimal guidance information using the user's history information and acquired item placement information. Input includes user history information, preference data, and item placement information. Data processing uses collaborative filtering algorithms to recommend products likely to interest the user, resulting in customized guidance information as output.

[0691] Step 6:

[0692] The server transmits the generated guidance information to the user's terminal via a communication method. Customized guidance information is used as input. As output, route guidance on a map displayed on the user's terminal and voice guidance information are provided. This allows the user to move efficiently within the store and easily find the desired product.

[0693] (Application Example 1)

[0694] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0695] In traditional stores, it was difficult and time-consuming for customers to efficiently find products that met their needs. Furthermore, it was difficult to offer product suggestions that adequately reflected customer preferences, potentially leading to decreased customer satisfaction.

[0696] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0697] In this invention, the server includes authentication means for identifying individual customer information, storage means for acquiring past customer transaction information, calculation means for acquiring product placement information within the store and creating route information based on the customer transaction information, and calculation means for generating movement guidance and recommended product information based on past transaction information and product placement information. This enables customers to quickly find the optimal products according to their preferences and purchase history, move efficiently within the store, and make purchases.

[0698] An "authentication device" is a device used to identify individual customer information and has the function of identifying customers using facial recognition technology.

[0699] A "storage device" is a memory device that stores a customer's past transaction information and allows it to be retrieved as needed.

[0700] A "processing unit" is a device that acquires information on the placement of products within a store and has the function of creating route information and recommended product information based on customer transaction information.

[0701] A "communication device" is a communication device that transmits information in order to provide routing information to a customer's device.

[0702] "Route information" refers to information that shows the optimal path for customers to move efficiently within a store and reach their desired product.

[0703] "Recommended product information" refers to information about products that are highly relevant to the customer, suggested based on the customer's past transaction history and preferences.

[0704] To realize this invention, a program using the following system components is important.

[0705] The server first identifies the individual information of users entering the store via a camera that acts as an authentication device. This is done using a facial recognition library such as OpenCV. The recognized facial data is sent to the server's database and compared against past transaction information. Firebase is an example of a database management system used in this process.

[0706] Next, the server obtains information about the placement of products within the store using location services such as the Google Maps API. Based on this information, it generates navigation guidance and recommended product information based on the user's preferences and transaction history. This data processing is realized by analysis algorithms and computing devices performed on the server.

[0707] Subsequently, the server functions as a communication device, providing the generated route information and recommended product information to the customer's smartphone. The smartphone app visually displays the route and guides the user using voice guidance. This allows users to easily find the products they want in a short amount of time, improving their shopping experience.

[0708] For example, when a user visits a large shopping mall, the application recognizes the user through cameras near the entrance and displays the shortest route to the sports equipment they have previously purchased. It can also provide voice notifications about discounts on the latest sports equipment.

[0709] An example of a prompt used when optimizing recommended product information based on customer preferences using a generative AI model is: "Identify the product the user is looking for in the store based on their past purchase history and preferences. Then, suggest the best way to purchase that product."

[0710] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0711] Step 1:

[0712] The server enables the camera on a terminal that functions as an authentication device and captures a facial image when a user enters the store. The input is the raw facial image data captured by the camera, and the output is the facial data with extracted features. Facial recognition software such as OpenCV is used to extract features and prepare them for transmission to the database.

[0713] Step 2:

[0714] The server matches the customer's past transaction information stored in the database. The input is processed feature facial data, and the output is the profile data of the matching customer. The matching process uses a management system such as Firebase to search for past purchase history and preference information.

[0715] Step 3:

[0716] The server uses location services to obtain product placement information within the store. Input is product identification information obtained from the product's barcode or QR code, and output is coordinate data indicating the product's location. Real-time location information is obtained using the Google Maps API and used for in-store navigation.

[0717] Step 4:

[0718] The server uses a generative AI model to generate optimal route guidance and recommended product information based on the user's transaction history and acquired product placement information. The input is customer profile data and product placement data, and the output is route guidance information and a list of recommended products. The prompt used for the generative AI model is, "Identify the product the user is looking for based on their past purchase history and preferences, and suggest the optimal purchase route for that product."

[0719] Step 5:

[0720] The terminal acts as a communication device, providing users with route guidance information and recommended product information received from the server. Input is data transmitted from the server, and output is route information displayed on the screen and voice guidance. The terminal combines visual map display and voice guidance to enable users to efficiently navigate to their desired products.

[0721] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0722] This invention is a support system for users to visit stores and shop efficiently and comfortably. The system comprises an identification device, a memory device, a computing device, a communication device, and an emotion engine that analyzes and adjusts the user's emotions.

[0723] When a user enters a store, the terminal's identification device uses facial recognition technology to identify the user. The server retrieves the necessary data from a storage device that contains the user's purchase history and preferences. The server then obtains the location information of items within the store and generates guidance information based on the historical information. This guidance information includes recommended routes and product lists derived from past purchase history.

[0724] Furthermore, an emotion engine operates to recognize the user's emotions in real time. The emotion engine analyzes facial expressions captured by the camera and voice tone obtained through the microphone to infer the user's emotional state. For example, if the system detects that the user is irritated, the server updates the guidance information, presenting a concise route and products to ensure a smooth shopping experience. The tone of the voice guide is also adjusted according to the user's emotions, creating a sense of friendliness and reassurance.

[0725] Users can receive guidance information transmitted to their terminals via communication devices and view it on their terminals. Guidance information is provided not only on the screen but also via audio, allowing for stress-free shopping using both sight and hearing. Furthermore, the emotion engine provides product recommendations optimized for the user's emotional state.

[0726] As a concrete example, when a user enters a large supermarket, they are identified by facial recognition, and the location of recommended products is suggested based on their past purchase history. At the same time, an emotion engine senses that the user is relaxed and sets the voice guidance to a friendly tone. This allows the user to comfortably browse the store and efficiently purchase what they need.

[0727] This system allows users to improve their shopping efficiency, and enables stores to enhance the customer experience.

[0728] The following describes the processing flow.

[0729] Step 1:

[0730] A user passes through the store entrance. At that moment, a camera installed at the entrance captures the user's face.

[0731] Step 2:

[0732] The device acquires facial image data captured by the device and sends that data to the server.

[0733] Step 3:

[0734] The server uses the received facial image data to match it with the customer database and identify the user.

[0735] Step 4:

[0736] The server retrieves the purchase history and preferences of identified users from its storage device.

[0737] Step 5:

[0738] The server retrieves the latest item placement data from the store and generates guidance information based on the user's purchase history. This information includes the location of the desired item and the optimal route to take.

[0739] Step 6:

[0740] The device's built-in emotion engine uses the camera and microphone to analyze the user's facial expressions and voice, and infer their current emotional state.

[0741] Step 7:

[0742] The server adjusts the guidance information based on data from the emotion engine, according to the user's emotional state. For example, if the user is excited, the guidance will be changed to a gentler tone.

[0743] Step 8:

[0744] The server sends the coordinated guidance information to the user's terminal via a communication device.

[0745] Step 9:

[0746] The device displays the received guidance information on the screen and plays audio guides as needed.

[0747] Step 10:

[0748] Users utilize guidance from their devices to find and purchase the items they are looking for in the store.

[0749] Step 11:

[0750] The server saves the user's new purchase information to storage and updates the purchase history and preference data.

[0751] (Example 2)

[0752] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0753] Modern consumers desire an efficient and comfortable shopping experience, but the vast product range and crowds often make shopping stressful. Furthermore, while there is a demand for services that cater to consumer emotions, systems capable of analyzing and instantly responding to individual consumer sentiments are lacking.

[0754] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0755] In this invention, the server includes identification means for detecting consumer characteristics, storage means for collecting consumer history information, calculation means for collecting product location information within the store and generating guidance information based on the consumer history information, and an emotion analysis engine for analyzing the consumer's emotional state and optimizing the guidance information. This makes it possible to provide personalized services based on consumer characteristics, thereby realizing an effective and comfortable shopping experience.

[0756] A "consumer" refers to an individual who purchases or uses goods or services.

[0757] An "identification device" is a combination of hardware and software used to detect and identify the individual characteristics of a consumer.

[0758] A "memory device" is a data storage system that stores consumer history information and preferences and retrieves them as needed.

[0759] A "computational means" is a computer system that processes information to generate guidance information based on the collected information.

[0760] "Communication means" refers to the network infrastructure and protocols used by a server to transmit information to a consumer's terminal.

[0761] An "emotion analysis engine" is software that analyzes a consumer's facial expressions and tone of voice to estimate their emotional state and adjust the way a service is delivered accordingly.

[0762] "Guidance information" refers to information such as the location of products and recommended routes provided to consumers to help them shop efficiently.

[0763] This invention is an information processing system that supports consumers in shopping efficiently within a store. Specifically, it begins when the user enters the store.

[0764] When a user enters a store, an identification device on the terminal uses an image processing library to capture the user's face and identifies the individual based on a facial recognition algorithm. To do this, the terminal applies common facial recognition technology and compares it with a user database.

[0765] Next, the server accesses storage to retrieve the user's past purchase history and preference data. This retrieved data is used to recommend products that match the user's preferences. The server also uses computing resources to collect product locations from the store's location information system and generate an optimal shopping route. Wi-Fi and beacon technology are used for location processing in this process.

[0766] Furthermore, an emotion analysis engine operates to analyze the user's emotions in real time. Specifically, it analyzes the user's facial expressions using the device's camera and captures their voice tone using the built-in microphone to infer their emotions. This allows for the optimization of guidance information according to the user's current emotional state.

[0767] The generated guidance information is transmitted to the user's device via communication means. The user can visually confirm the guidance information on their device and also listen to the information through an audio guide. This audio guide is adjusted based on the emotional state estimated by the emotion analysis engine and is delivered in an appropriate tone.

[0768] As a concrete example, when a user visits a large shopping center, this system identifies the user through facial recognition and guides them to the floor with recommended products by referring to their past purchase history. At the same time, if the emotion analysis engine detects that the user is in a calm mood, the voice guide explains the shopping route in a gentle tone. This guidance allows the user to shop efficiently and comfortably within the store.

[0769] An example of a prompt message might be: "When a user is looking for the optimal shopping route in the store, provide guidance that reflects the results of sentiment analysis."

[0770] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0771] Step 1:

[0772] When a user enters a store, a camera installed on the terminal captures an image of the user's face. The facial image, as input, is processed in real time, and the output is a unique ID necessary for identifying the user. Specifically, a facial recognition algorithm analyzes the image and identifies the user by comparing it with pre-registered facial data.

[0773] Step 2:

[0774] The server takes the identified user ID as input, accesses a storage device, and retrieves the user's purchase history and preference information from the database. The output is a list of user history and preferences. This list is generated by quickly collecting data related to the user's preferences using SQL queries.

[0775] Step 3:

[0776] The server obtains product location information from the store's location information system. The input to this process is a list of products to be handled, and the output is the location information of those products. Location information is collected using tracking technologies such as Wi-Fi and beacons. Subsequently, a computing system integrates user preference information and location information to generate the optimal shopping route.

[0777] Step 4:

[0778] The device's built-in emotion analysis engine analyzes the user's emotional state in real time. Inputs are facial expression data from the camera and voice tone from the microphone, while output is a status indicating the user's emotional state. Specifically, a machine learning model analyzes this data to estimate the user's emotions.

[0779] Step 5:

[0780] The server optimizes guidance information based on the user's emotional state, adjusting the content and presentation of the information provided to the user. The input is the analyzed emotional status, and the output is the adjusted guidance information. This information is dynamically updated as the tone and display content of the audio guide.

[0781] Step 6:

[0782] The server transmits the final guidance information to the user's terminal via communication means. The input is optimized guidance information, and the output is the information displayed on the user's terminal. Users can efficiently navigate the store using visual displays or audio guidance on their terminals. In this case, the audio guidance is provided in a friendly tone that matches the user's emotions, allowing the user to shop more comfortably.

[0783] (Application Example 2)

[0784] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0785] There is a demand to provide customers with an efficient and comfortable shopping experience when they shop at physical stores. Conventional systems do not adequately optimize guidance information based on customer emotions and preferences, which can cause stress during shopping. This invention aims to improve the customer experience by analyzing the customer's real-time emotional state and adjusting guidance information based on this analysis.

[0786] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0787] In this invention, the server includes identification means for recognizing customer characteristics, storage means for acquiring customer history information, calculation means for acquiring location information of items in the store and generating guidance information, and emotion analysis means for analyzing customer emotions and optimizing guidance information. This makes it possible to provide optimal shopping guidance based on the customer's preferences and emotional state.

[0788] A "customer" is a person who visits a store and intends to purchase goods or services.

[0789] An "identification device" is a device equipped with technology to recognize the characteristics of a customer.

[0790] "Facial recognition technology" is an image processing technology used to identify a customer's face.

[0791] A "storage device" is a device used to store and retrieve customer history information.

[0792] "History information" refers to information that shows the products a customer has purchased in the past and their purchasing trends.

[0793] A "computational device" is a device that collects and analyzes data to generate the desired information.

[0794] A "communication device" is a device used to transmit information generated by a computing device to a customer's terminal.

[0795] An "emotion analysis device" is a device equipped with technology that analyzes a customer's emotional state and takes the results into consideration.

[0796] "Information" refers to information provided to support customer behavior within a store.

[0797] A "terminal" is a personal information terminal owned by a customer, and is a device that receives information provided by a communication device.

[0798] The system implementing this invention is designed to provide customers with an efficient and comfortable shopping experience when they visit a physical store. The server first recognizes the customer's characteristics using an identification device. This identification uses facial recognition technology, and a camera device acquires image data of the customer's face. The acquired data is analyzed by software called FaceRecognitionAPI to identify the customer.

[0799] Subsequently, the server retrieves the customer's past history information from storage, and the computing device combines this with location information of items within the store to generate guidance information. This process uses the StoreMapAPI, which provides store map information. The generated guidance information is optimized according to the customer's emotional state. The EmotionAnalysisAPI is used for emotion analysis, and emotion analysis is performed based on data acquired from cameras and microphones.

[0800] This optimized guidance information is transmitted to the customer's device via a communication device. The device displays the information on a smartphone application and provides audio guidance, allowing the customer to continue shopping without stress.

[0801] As a concrete example, a system could be envisioned where, upon entering an electronics store, a customer is guided to the shortest route to where their favorite brand's new products are displayed, while simultaneously receiving promotional information. In this case, if the user is relaxed, the voice guidance would be delivered in a calm and friendly tone.

[0802] An example of a prompt using a generative AI model is: "Consider a scenario for an app that detects a user's face with a camera, analyzes their emotions, and provides real-time recommendations for home appliances based on their purchase history." This prompt is used when simulating the system's emotional state-based optimization process.

[0803] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0804] Step 1:

[0805] The server acquires images of the user's face through camera devices installed in the store. The facial image is provided as input, and the image data is analyzed using the FaceRecognition API to identify the user. The output after analysis is the user's ID. Through this process, the identification device obtains the user's characteristic information.

[0806] Step 2:

[0807] The server retrieves the user's past purchase history from storage based on the identified user's ID. By using the user ID as input and searching the history database, the server obtains the user's purchase history data as output. This data is used to understand the user's preferences.

[0808] Step 3:

[0809] The server obtains location information for items within the store using the StoreMapAPI. Store map data is imported as input, and the location of items is displayed on the map as output. The computing unit integrates this map information with the user's purchase history to generate navigation information. This navigation information includes product locations and recommended routes based on the user's interests.

[0810] Step 4:

[0811] The server inputs facial and audio data acquired from the camera and microphone into the EmotionAnalysisAPI to analyze the user's emotions. The output is the user's emotional state, which the emotion analysis device uses. Based on this, the guidance information is adjusted and optimized guidance is prepared.

[0812] Step 5:

[0813] The server uses a communication device to transmit optimized guidance information to the user's terminal. The pre-adjusted guidance information is used as input, and the output includes audio and visual guides displayed on the terminal. This makes it easier for the user to navigate the store based on the information displayed on their terminal.

[0814] Step 6:

[0815] Users receive guidance information visually and audibly through a smartphone application. The guidance includes friendly voice guidance tailored to the user's emotional state. This reduces shopping stress and allows users to efficiently find the products they need.

[0816] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0817] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet Search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0818] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0819] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0820] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0821] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0822] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0823] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0824] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0825] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0826] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0827] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0828] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0829] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0830] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0831] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0832] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0833] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0834] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0835] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0836] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted as being incorporated by reference.

[0837] The following is further disclosed regarding the embodiments described above.

[0838] (Claim 1)

[0839] An identification device for recognizing customer characteristics,

[0840] A storage device for obtaining customer history information,

[0841] A computing device for acquiring location information of items within a store and generating guidance information based on customer history information,

[0842] A communication device for providing guidance information to the customer's terminal,

[0843] A system that includes this.

[0844] (Claim 2)

[0845] The system according to claim 1, wherein the identification device recognizes the customer using facial recognition technology.

[0846] (Claim 3)

[0847] The system according to claim 1, which provides guidance information to customers by voice.

[0848] "Example 1"

[0849] (Claim 1)

[0850] A device that photographs the customer's face and recognizes its features,

[0851] A storage means for acquiring customer purchase history and preference data,

[0852] Information processing means for aggregating information on the placement of items within a store and generating personalized guidance based on customer history and preferences,

[0853] A communication means for transferring the generated guidance information to the customer's individual terminal,

[0854] A system that includes this.

[0855] (Claim 2)

[0856] A means for quickly identifying a customer using facial recognition technology, the system according to claim 1.

[0857] (Claim 3)

[0858] A means for providing guidance information to customers visually and audibly, the system according to claim 1.

[0859] "Application Example 1"

[0860] (Claim 1)

[0861] Authentication device for identifying individual customer information,

[0862] A storage device for acquiring past transaction information of customers,

[0863] A computing device for acquiring product placement information within a store and creating route information based on customer transaction information,

[0864] A communication device for providing route information to the customer's device,

[0865] A computing device for generating customer flow guidance and recommended product information based on past transaction information and product placement information,

[0866] A system that includes this.

[0867] (Claim 2)

[0868] The system according to claim 1, wherein the authentication device uses facial recognition technology to identify the customer.

[0869] (Claim 3)

[0870] The system according to claim 1, which provides route information and recommended product information to customers by voice.

[0871] "Example 2 of combining an emotion engine"

[0872] (Claim 1)

[0873] An identification device for detecting consumer characteristics,

[0874] A storage means for collecting consumer history information,

[0875] A calculation means for collecting location information of products within a sales office and generating guidance information based on consumer history information,

[0876] A means of communication for transmitting guidance information to a consumer's device,

[0877] An emotion analysis engine to analyze consumers' emotional states and optimize guidance information,

[0878] A system that includes this.

[0879] (Claim 2)

[0880] The system according to claim 1, wherein the identification device identifies consumers using image recognition technology.

[0881] (Claim 3)

[0882] The system according to claim 1, which provides guidance information to consumers in audio and adjusts the tone of the audio guide according to the consumer's emotions.

[0883] "Application example 2 of combining emotional engines"

[0884] (Claim 1)

[0885] An identification device for recognizing customer characteristics,

[0886] A storage device for obtaining customer history information,

[0887] A computing device for acquiring location information of items within a store and generating guidance information based on customer history information,

[0888] A communication device for providing guidance information to the customer's terminal,

[0889] An emotion analysis device for analyzing customer emotions and optimizing guidance information,

[0890] A system that includes this.

[0891] (Claim 2)

[0892] The system according to claim 1, wherein the identification device recognizes the customer using facial recognition technology.

[0893] (Claim 3)

[0894] The system according to claim 1, which provides guidance information to customers by voice and adjusts the voice guide based on the customer's emotional state. [Explanation of Symbols]

[0895] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. An identification device for recognizing customer characteristics, A storage device for obtaining customer history information, A computing device for acquiring location information of items within a store and generating guidance information based on customer history information, A communication device for providing guidance information to the customer's terminal, A system that includes this.

2. The system according to claim 1, wherein the identification device recognizes the customer using facial recognition technology.

3. The system according to claim 1, which provides guidance information to customers by voice.

Images