System

Abstract

A system is provided.SOLUTION: A terminal including an information processing apparatus worn by a user and a communication means, a server that receives visual information and audio information collected by the terminal, a generation AI means that analyzes and patterns the collected visual information and audio information in the server, a data providing means that provides an analysis result obtained by the generation AI means to a company, and a reward management means that provides a reward to a user.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In modern society, acquiring data on actual product purchases is important for companies to promote marketing and digital transformation (DX). However, conventional methods have limited ways to efficiently visualize consumers' in-store movements and product selection processes. This has made it difficult to collect detailed data that can be used to optimize marketing strategies and store operations. There is a need to provide a new system that can solve these problems and make effective use of consumer behavior data. [Means for solving the problem]

[0005] The present invention solves the above-mentioned problems with a system that includes a terminal equipped with an information processing device and communication means worn by a user, a server that receives visual and audio information collected by the terminal, a generation AI means in the server that analyzes and patterns the collected visual and audio information, a data provision means that provides companies with the analysis results obtained by the generation AI means, and a reward management means that awards rewards to users. Furthermore, because the terminal is smart glasses worn by the user, it is possible to collect data on the user's natural behavior, and the reward management means allows the user to earn points in exchange for providing the data. This realizes a win-win situation in which companies can obtain highly accurate marketing data and consumers also receive rewards.

[0006] "User" refers to a consumer who wears an information processing device and provides visual and audio information.

[0007] "Information processing device" refers to a terminal that collects visual and audio information and is equipped with a communication means.

[0008] "Communication means" refers to the interface or protocol that has the function of sending collected information to the server.

[0009] "Terminal" refers to an information processing device worn by a user, which has the function of collecting visual and audio information.

[0010] "Visual information" refers to video data that is visually recognized by the user.

[0011] "Audio information" refers to audio data including user speech and environmental sounds.

[0012] "Server" refers to a computer system for receiving and analyzing collected visual and audio information.

[0013] "Generative AI means" refers to artificial intelligence for analyzing and patterning collected visual and audio information.

[0014] "Data provision means" refers to the mechanism for providing companies with data analyzed by the generating AI means.

[0015] "Reward management means" refers to a system for providing rewards to users in exchange for providing data.

[0016] "Smart glasses" refers to glasses-type information processing devices that are worn by users and have the function of collecting visual and audio information.

[0017] "Company" refers to the organization that receives the analyzed data and uses it to optimize marketing strategies and store operations.

[0018] "Points" refers to the rewards that users receive in exchange for providing data. [Brief explanation of the drawings]

[0019] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0020] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0021] First, the terms used in the following description will be explained.

[0022] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0023] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0024] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0025] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0026] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0027] [First embodiment]

[0028] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0029] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0030] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0031] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0032] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0033] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0034] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0035] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0036] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0037] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0038] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0039] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0040] The present invention is a system that uses a terminal equipped with an information processing device and communication means worn by the user, a server, a generating AI means, a data providing means, and a reward management means, and collects, analyzes, and provides highly accurate data that is useful for optimizing corporate marketing and store operations. Specific examples are shown below.

[0041] Data collection by terminal

[0042] The user wears the information processing device and launches a dedicated app. The information processing device takes the form of smart glasses, and collects visual and audio information through communication means and sends it to a cloud server via a smartphone.

[0043] When a user enters a store, the smart glasses automatically begin recording. The user's visual information (e.g., their behavior as they look at the shelves) and audio information (e.g., their comments about products) are collected in real time. This information is then collected on the smartphone and sent to a cloud server via the Internet.

[0044] Data analysis by server

[0045] The server analyzes the received visual and audio information using a generation AI means. The generation AI means uses advanced algorithms such as deep learning and natural language processing (NLP) to analyze eye movement patterns and speech content, creating patterns. These patterns make it possible to extract the user's behavioral tendencies and interests.

[0046] Providing data to companies

[0047] The analyzed data is provided to companies through data provision means. Based on this data, companies can optimize product display locations, develop promotional strategies, and arrange products to attract consumer interest, thereby improving their marketing strategies and store operations.

[0048] Rewarding

[0049] In addition, users are rewarded through a reward management means. Specifically, points are awarded to users based on the collected and provided data. These points can be used for the next purchase or use of other services. This increases users' motivation to participate and enables the acquisition of more abundant data.

[0050] Specific examples

[0051] For example, consider a scenario in which a user wears smart glasses in a store and walks around the grocery section. While enjoying their usual shopping routine, the user may pause at a particular shelf for a long time or comment aloud about a particular product. This data is collected in real time and sent to a server via smartphone. On the server, a generative AI tool analyzes the user's behavior and provides companies with insights such as which products are likely to attract attention and which areas are often overlooked. At the same time, the user earns points in exchange for providing the data, which can be used on their next purchase.

[0052] The present invention aims to improve the quality and quantity of data collection by not only utilizing the data collected in this way for corporate marketing strategies and store operations, but also by providing direct rewards to users.

[0053] The processing flow will be explained below.

[0054] Step 1:

[0055] The user wears the smart glasses and launches a dedicated app on their smartphone, which connects to the smart glasses via Bluetooth or Wi-Fi.

[0056] Step 2:

[0057] When a user enters a store, a dedicated app automatically activates the smart glasses' camera and microphone to begin collecting visual and audio information.

[0058] Step 3:

[0059] The device (smart glasses) collects the user's visual information (eye movements and products they are looking at) and audio information (user comments and ambient sounds) in real time, and the collected data is sent sequentially to a smartphone.

[0060] Step 4:

[0061] The user purchases an item at the cash register and pays using a cashless payment method such as PayPay. The smartphone app associates the purchase data obtained through the payment with visual and audio information.

[0062] Step 5:

[0063] When a user stops recording using the app, the devices (smart glasses and smartphone) upload visual, audio, and payment information to a cloud server.

[0064] Step 6:

[0065] The server receives the uploaded data and analyzes the visual and audio information using a generative AI, which uses a deep learning algorithm to generate patterns from the data and extract user behavioral trends and interests.

[0066] Step 7:

[0067] The server stores the analysis results in a database, organizes the stored data in a format that companies can access, and provides it to them through data provision means.

[0068] Step 8:

[0069] Based on the data provided, companies can optimize product display locations and develop promotional strategies.

[0070] Step 9:

[0071] In return for providing the data, the server uses a reward management means to give the user points, which the user can use for their next purchase.

[0072] In this way, this system efficiently collects visual and audio information from users, analyzes and patterns it using generative AI, and provides companies with valuable marketing data.In return, users are awarded points in return for providing the data, creating a win-win situation.

[0073] Example 1

[0074] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0075] Conventional marketing data collection systems have difficulty analyzing user behavior and interests with high accuracy, limiting the quality and quantity of collected data. Furthermore, incentives for users are insufficient, making it difficult to obtain proactive data provision. This makes it difficult for companies to optimize their marketing strategies and store operations.

[0076] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0077] In this invention, the server includes a terminal equipped with an information processing device and communication means worn by the user, a means for receiving visual and audio information collected by the terminal, a data analysis means for analyzing and patterning the visual and audio information collected by the server, an information provision means for providing companies with the analysis results obtained by the data analysis means, a reward management means for rewarding users, a communication means for aggregating data collected from the terminal on a smartphone and transmitting it to a cloud server, a processing means for analyzing the visual and audio data using a generative AI model in the cloud, and a patterning means for extracting user behavioral trends and interests based on the collected data. This enables highly accurate analysis of user behavior and interests and the collection of high-quality data. Furthermore, rewarding users encourages proactive data provision.

[0078] An "information processing device" is a wearable device worn by a user that collects visual and audio information.

[0079] The "communication means" is a means having a function of transferring data from an information processing device to a smartphone and transmitting data from the smartphone to a cloud server.

[0080] A "server" is a computer system that operates on the cloud and has the ability to receive and analyze collected visual and audio information.

[0081] "Data analysis means" refers to algorithms or software that analyzes collected visual and audio information and patterns user behavior and interests.

[0082] "Information provision means" refers to the means for providing analyzed data to companies.

[0083] The "reward management means" is a means for giving rewards to users for providing data, and specifically has a function of giving points.

[0084] "Processing means" refers to a means that has the function of analyzing visual and audio information using a generative AI model on the cloud.

[0085] "Patterning means" refers to a means for extracting and patterning a user's behavioral tendencies and interests.

[0086] A "terminal" is a device that is worn by a user and has an information processing device and communication means.

[0087] A "generative AI model" is an artificial intelligence model that uses deep learning and natural language processing to analyze data and extract user behavior patterns.

[0088] This invention is a system that uses a terminal equipped with an information processing device and communication means worn by the user, a server, a generating AI means, a data providing means, and a reward management means, and collects, analyzes, and provides highly accurate data that is useful for optimizing corporate marketing and store operations.

[0089] Data collection by terminal

[0090] The user wears the information processing device and launches a dedicated app. The information processing device takes the form of smart glasses, which collect visual and audio information and send it to a cloud server via a smartphone. The smart glasses are designed to automatically start recording when the user enters a store. Visual information (e.g., the user's actions as they look at product shelves) and audio information (e.g., comments about products) are collected in real time. This information is first collected on the smartphone and then sent to the cloud server via the Internet.

[0091] Data analysis by server

[0092] The server analyzes the received visual and audio information using a generative AI means. The generative AI means uses advanced algorithms such as deep learning and natural language processing (NLP) to generate patterns of eye movement and speech content. This analysis makes it possible to extract the user's behavioral tendencies and interests.

[0093] Providing data to companies

[0094] The analyzed data is provided to companies through data provision means. Based on this data, companies can optimize product display locations, develop promotional strategies, and position products to attract consumer interest. For example, by positioning specific products in locations that are likely to attract attention, companies can improve their marketing strategies and store operations.

[0095] Rewarding

[0096] Users are rewarded through a reward management system. Specifically, based on the data collected and provided, users are awarded points. These points can be used for their next purchase or for using other services, which increases user participation and allows for the acquisition of more data.

[0097] Specific examples

[0098] For example, consider a scenario in which a user wears smart glasses in a store and walks around the grocery section. While shopping as usual, the user may pause at a particular shelf for a long time or comment aloud about a particular product. This data is collected in real time and sent to a server via smartphone. On the server, a generative AI tool analyzes the user's behavior and provides companies with insights such as which products are likely to attract attention and which areas are often overlooked. At the same time, the user earns points in exchange for providing the data, which can be used on their next purchase.

[0099] Prompt Sentence Examples

[0100] "Users gazed at a specific product shelf for about 10 seconds, then commented out loud on the name of the product. The system analyzed their eye movements and the content of their comments, and provided companies with information on which products were most likely to attract attention."

[0101] This invention aims to improve the quality of collected data by analyzing user behavior and interests with high accuracy, and also to increase motivation for providing data by providing rewards to users.

[0102] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0103] Step 1:

[0104] The user wears the information processing device and launches a dedicated app on their smartphone. The device (smart glasses) synchronizes with the launch of the app and begins collecting data. At this time, the device collects the user's visual information (which shelf they are looking at) and audio information (what they are saying) and converts it into digital format. The input is the visual and audio data captured by the smart glasses, and the output is the visual and audio data converted into digital format.

[0105] Specific examples of behavior:

[0106] When a user enters a store, the smart glasses automatically start recording.

[0107] When the app is launched, the device captures the user's eye movements and voice in real time.

[0108] Step 2:

[0109] The device transfers the collected visual and audio information to a smartphone, which temporarily stores the data and sends it to a cloud server via the Internet. The input is the digital data sent from the device, and the output is the data sent to the cloud server.

[0110] Specific examples of behavior:

[0111] As the user moves around the store, the smart glasses continuously transmit the information they collect to their smartphone.

[0112] The smartphone sends the data to a cloud server at regular intervals.

[0113] Step 3:

[0114] The server runs on the cloud and receives visual and audio information sent from the smartphone. This data is then analyzed by the generative AI model. The input is the visual and audio data sent to the cloud server, and the output is the analyzed behavioral patterns and interest data. The generative AI model performs advanced analysis using deep learning and natural language processing.

[0115] Specific examples of behavior:

[0116] The server analyzes the received visual data and determines which product shelf the user looked at and for how long.

[0117] The voice data is analyzed and information about products that have piqued the user's interest is extracted from the content of the user's comments.

[0118] Step 4:

[0119] The data analyzed on the server is provided to companies through data provision means. Companies use this data to optimize product display locations and promotion strategies. The input is data on behavioral patterns and interests as a result of the analysis, and the output is insight data provided to companies.

[0120] Specific examples of behavior:

[0121] The server generates reports and automatically sends them to the company's marketing system.

[0122] Based on the reports they receive, companies can take action such as changing the location of product displays.

[0123] Step 5:

[0124] The server provides rewards to users through the reward management means. Specifically, points are given to users for the collected and provided data. The input is the behavioral data as the analysis result, and the output is the user's reward point data.

[0125] Specific examples of behavior:

[0126] Points are automatically credited to the user's account based on the data provided by the user.

[0127] A notification regarding the points being awarded will be sent to the user's smartphone.

[0128] In this way, through a series of processing steps, a system is completed that analyzes user behavior and interests with high accuracy, collects high-quality data, and provides it to companies. Users are rewarded, so they are expected to actively provide data.

[0129] (Application example 1)

[0130] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0131] With conventional methods of collecting marketing data, it was difficult to accurately grasp users' actual purchasing behavior and interests. Furthermore, the analysis of collected data was delayed, making it difficult to change strategies in real time. Furthermore, there was insufficient incentive for users to voluntarily participate in providing data.

[0132] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0133] In this invention, the server includes a means for transmitting visual information to a cloud storage service in real time, a means for analyzing the visual and audio information using a generative AI model, and a means for providing the analysis results to a company's API endpoint. This allows for the collection and analysis of users' visual and audio information in real time and the immediate provision of the analysis results to the company. Furthermore, by awarding users points that can be used toward their next purchase, incentives for data provision are increased, enabling the collection of higher quality data.

[0134] An "information processing device" is a device worn by a user that collects visual and audio information and transmits the data in real time.

[0135] "Communication means" refers to means for transmitting and receiving data between an information processing device and a server, and includes communication via the Internet or cloud storage.

[0136] "Visual information" refers to video data that a user perceives visually, and is information collected using a photographic device such as a camera.

[0137] "Audio information" refers to audio data including the user's speech and surrounding sounds, and is information collected using a microphone or the like.

[0138] "Server" refers to a computer system for receiving and analyzing collected visual and audio information.

[0139] "Generative AI methods" refer to algorithms and systems that use deep learning and natural language processing to analyze collected data and pattern user behavioral trends.

[0140] "Data provision means" refers to the means for providing the analysis results of the generation AI means to the company, and sends the data to the company's API endpoint via the Internet.

[0141] "Reward management means" refers to a system that provides rewards to users in exchange for providing data, such as a mechanism for awarding points.

[0142] A "cloud storage service" is a storage service that enables the storage and retrieval of data via the Internet, and is used to store video data and analysis results.

[0143] "Generative AI model" refers to an advanced machine learning model used to analyze visual and audio information, identify user behavior patterns, and analyze areas of interest, etc.

[0144] "API endpoint" refers to the interface provided by a company to receive data and is used to automatically transmit analysis results.

[0145] This invention is a system that uses a terminal equipped with an information processing device and communication means worn by the user, a server, a generating AI means, a data providing means, and a reward management means. This system collects and analyzes the visual and audio information of the user and provides it to companies to support the optimization of marketing strategies and store operations.

[0146] First, the device includes an information processing device (smart glasses) worn by the user and a communication means (smartphone). The smart glasses are equipped with a camera and microphone to collect visual and audio information from the user. The communication means is responsible for sending the collected information to a cloud server via the Internet.

[0147] The server functions as a cloud server and receives the collected visual and audio information. The server then analyzes the data using a generative AI model that utilizes deep learning and natural language processing to identify patterns of user behavior.

[0148] The data analyzed by the Generative AI Method is provided to the company via the Data Provision Method. Specifically, the analysis results are automatically sent to the API endpoint provided by the company. This allows companies to understand consumer interests in real time and use this information to improve their product displays and promotional strategies.

[0149] Meanwhile, the reward management means acts as an incentive for users. Points are awarded based on the data provided by the user, and these points can be used for future purchases, etc. This increases user participation and allows for the collection of more data.

[0150] As a concrete example, consider a scenario where a user visits the fruit section of a supermarket. While wearing smart glasses, the user looks at a particular fruit for a long time and comments aloud about the fruit's quality. This information is sent to a cloud server in real time and analyzed by a generative AI model, which provides the company with analytical results such as "bananas are attracting the most attention" or "many consumers are commenting on the price of apples." Based on these analytical results, the company can make more effective product placement and pricing decisions.

[0151] An example prompt is:

[0152] "Based on scenarios where users walk through a store and their eyes are drawn to specific products, create prompts that analyze which products and areas are most relevant. Also, analyze comments users make about specific products to gauge their level of interest."

[0153] As described above, this invention solves conventional problems and realizes optimization of marketing and store operations by using an information processing device, communication means, server, generation AI means, data provision means, and reward management means.

[0154] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0155] Step 1:

[0156] A user puts on smart glasses and launches the application. The smart glasses use a camera and microphone to collect visual information (video data) and audio information (audio data). The input data is the user's view of the shelves and store interior, as well as the user's conversations and comments. The output is raw video and audio data.

[0157] Step 2:

[0158] The device (smartphone) receives the visual and audio information collected from the smart glasses and temporarily stores it in storage. The input is the video and audio data sent from the smart glasses, and the output is the temporary data stored on the smartphone.

[0159] Step 3:

[0160] The smartphone transmits the collected visual and audio information via the Internet to a cloud server, where the data is appropriately compressed and uploaded to a cloud storage service. The input is the video and audio data stored on the smartphone, and the output is the data stored in the cloud storage service.

[0161] Step 4:

[0162] The server receives visual and audio information from cloud storage and analyzes it using a generative AI model. Specifically, it analyzes eye movement patterns using deep learning and audio information using natural language processing (NLP). The input is the video and audio data stored in cloud storage, and the output is the analysis results (e.g., eye movement patterns, content of user comments).

[0163] Step 5:

[0164] The server generates a prompt based on the analysis results and provides the analysis results to the company's API endpoint. The input is the analyzed data and the prompt from the generative AI model, and the output is the analysis results sent to the company. Specifically, the server invokes the generative AI model and sends the analysis results to the company's API as a POST request.

[0165] Step 6:

[0166] The server calculates the reward for the user's data provision and assigns points to the user's account through the reward management means. The input is the analysis result and the amount of data provided, and the output is the assigned point information. Specifically, the server accesses the reward management system and updates the points in the user's account.

[0167] Step 7:

[0168] The user uses the points they have earned on their next purchase. The input is the points information earned on the user's account, and the output is the purchase history using the points. Specifically, the user can use the points to receive a discount or exchange them for a specific product at a store.

[0169] Through this series of steps, highly accurate data is collected while users enjoy shopping in stores, and the analysis results are provided to companies, allowing users to receive rewards at the same time.

[0170] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0171] The present invention is a system that uses a terminal equipped with an information processing device and communication means worn by the user, a server, a generating AI means, a data providing means, a reward management means, and an emotion engine to collect, analyze, and provide highly accurate data that is useful for optimizing corporate marketing and store operations. Specific examples are shown below.

[0172] Data collection by terminal

[0173] The user wears the information processing device and launches a dedicated app on their smartphone. The information processing device takes the form of smart glasses, collects visual and audio information, and sends the data to a cloud server via the smartphone using a communication means.

[0174] When a user enters a store, the smart glasses automatically start recording and begin collecting visual and audio information. The user's visual information (eye movements and products they are looking at) and audio information (user comments and ambient sounds) are collected in real time. This visual and audio data is then collected on the smartphone and sent to a cloud server via the Internet.

[0175] Introducing the Emotion Engine

[0176] The emotion engine recognizes the user's emotions based on visual and audio information collected in real time. The emotion engine analyzes eye movements, facial expressions, and tone of voice to identify the user's emotional state. This emotional data, along with the visual and audio information, is sent to a cloud server and incorporated into the analysis by the generative AI method.

[0177] Data analysis by server

[0178] The server receives the uploaded visual, audio, and emotional data and analyzes it using a generative AI method. The generative AI method uses deep learning and natural language processing (NLP) to analyze the data and identify patterns in the user's behavioral tendencies, interests, and newly acquired emotional data. The results of this analysis are stored in a database.

[0179] Providing data to companies

[0180] The analyzed data is provided to companies through data provision means. Based on this data, companies can optimize product display placement, develop promotional strategies, position products to attract consumers' attention, and even implement marketing tactics that respond to user emotions. This will improve the accuracy of marketing strategies and store operations.

[0181] Rewarding

[0182] In addition, users are given rewards through a reward management means. Specifically, users are given points in exchange for providing visual information, audio information, and emotional data. Users can use these points for their next purchase or for other services. This increases users' motivation to participate and enables the acquisition of more abundant data.

[0183] Specific examples

[0184] For example, consider a scenario in which a user wears smart glasses in a store and walks around the grocery section. While shopping as usual, the user may pause at a particular shelf for a long time or comment aloud about a particular product. At this time, the emotion engine analyzes the user's visual and audio information to recognize the user's emotions (e.g., interest, surprise, anticipation, etc.). This data is sent in real time via smartphone to a cloud server. On the server, a generative AI tool analyzes the user's behavior and emotions, providing companies with insights such as which products are likely to attract attention and which locations attract emotional interest. At the same time, the user is awarded points in exchange for providing the data, which can be used on their next purchase.

[0185] The present invention not only utilizes the collected data in this way to improve corporate marketing strategies and store operations, but also promotes improvements in the quality and quantity of data collection by offering rewards to users. The introduction of an emotion engine makes it possible to provide even more advanced analysis and insights.

[0186] The processing flow will be explained below.

[0187] Step 1:

[0188] The user wears the smart glasses and launches a dedicated app on their smartphone, which connects to the smart glasses via Bluetooth or Wi-Fi.

[0189] Step 2:

[0190] When a user enters a store, the smart glasses' camera and microphone automatically activate and begin collecting visual and audio information.

[0191] Step 3:

[0192] The device (smart glasses) collects the user's visual information (eye movements and images of products they are looking at) and audio information (user comments and surrounding environmental sounds) in real time, and the collected data is sent sequentially to a smartphone.

[0193] Step 4:

[0194] When a user pauses in front of a particular product for a long time or reads a product description, the emotion engine processes this information to recognize the user's emotions (interest, surprise, anticipation, etc.) The emotion engine analyzes visual information and changes in voice tone.

[0195] Step 5:

[0196] A user purchases an item at a cash register and uses a cashless payment method (e.g., a smartphone payment app), which links the payment information with visual and audio information.

[0197] Step 6:

[0198] When the user stops recording using the app, the devices (smart glasses and smartphone) upload the collected visual information, audio information, emotional data, and payment information to a cloud server.

[0199] Step 7:

[0200] The server receives the uploaded data and analyzes and patterns the visual, audio, and emotional data using a generative AI. The generative AI uses deep learning and natural language processing (NLP) algorithms to extract the user's behavioral tendencies, interests, and emotions.

[0201] Step 8:

[0202] The server stores the analysis results in a database, organizes the stored data in a format that companies can access, and provides it to them through data provision means.

[0203] Step 9:

[0204] Companies can use the data provided to optimize product displays and develop promotional strategies. In particular, using emotion data can enable more effective marketing measures, such as product sorting and advertising.

[0205] Step 10:

[0206] In return for providing the data, the server uses a reward management means to give the user points, which the user can use for their next purchase or other services.

[0207] These steps enable efficient collection and analysis of users' visual, audio, and emotional data, providing businesses with highly accurate marketing data. Users are rewarded, which strengthens their motivation to provide data and ensures the smooth functioning of the entire system.

[0208] Example 2

[0209] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0210] In modern marketing and store operations, it is important to collect consumer behavioral and emotional data with high accuracy in real time and analyze it efficiently. However, this requires a system that can perform advanced data analysis without requiring users to spend time or money. Conventional systems have difficulty analyzing emotions in real time or providing highly accurate data, which has hindered companies from quickly optimizing their marketing strategies.

[0211] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0212] In this invention, the server includes a terminal equipped with an information processing device and communication means worn by the user, means for receiving visual information and audio information collected by the terminal, generation AI means in the server for analyzing and patterning the collected visual information and audio information, an emotion engine for analyzing the visual information and audio information in real time and recognizing the user's emotions, data provision means for providing companies with the analysis results obtained by the generation AI means, and reward management means for awarding rewards to users. This enables high-precision collection and analysis of consumer behavior data and emotion data in real time, allowing companies to quickly and effectively optimize their marketing strategies and store operations.

[0213] An "information processing device" is a device worn by a user that collects visual and audio information.

[0214] The "communication means" is a means for transmitting data collected from an information processing device to a smartphone or other device.

[0215] "Terminal" refers to a device equipped with an information processing device and a communication means, which collects and transmits visual and audio information of a user.

[0216] "Visual information" refers to information that users perceive visually, such as the movement of their eyes and the products they notice.

[0217] "Audio information" refers to data related to audio, such as the user's speech and surrounding environmental sounds.

[0218] A "server" is a computer system that receives, analyzes, and stores visual and audio information sent from a terminal.

[0219] "Generative AI means" are means of analyzing collected data and creating patterns using technologies such as deep learning and natural language processing.

[0220] The "emotion engine" is an engine that analyzes visual and audio information collected in real time and recognizes the user's emotions.

[0221] "Data provision means" refers to the means for providing companies with the analysis results obtained by the generation AI means.

[0222] The "reward management means" is a means for awarding rewards to users, and specifically includes a method for awarding points.

[0223] "Points" are units of reward that users receive in exchange for providing data, and can be used for their next purchase or other services.

[0224] This invention is a system that uses a terminal equipped with an information processing device and communication means worn by the user, a server, a generating AI means, a data providing means, a reward management means, and an emotion engine to collect, analyze, and provide highly accurate data that is useful for optimizing corporate marketing and store operations. This system can collect and analyze user behavior and emotion data in real time.

[0225] First, the user puts on a smart-glasses-shaped information processing device. This device is designed to collect visual and audio information. Before entering the store, the user launches a dedicated app on their smartphone. At this time, the device collects visual information (e.g., the user's eye movements and products they are looking at) and audio information (e.g., the user's comments and ambient sounds) in real time and sends it to a cloud server via the smartphone. This information is then uploaded to the server via the Internet.

[0226] The server receives and analyzes the uploaded visual, audio, and emotional data. This analysis uses techniques such as deep learning and natural language processing (NLP). The generative AI means analyzes this data and creates patterns based on the user's behavioral tendencies, interests, and newly acquired emotional data. The emotion engine analyzes the visual and audio information in real time to identify the user's emotional state (e.g., interest, surprise, anticipation, etc.). This enables highly accurate emotion recognition.

[0227] The analyzed data is provided to companies through data provision means. This data includes users' behavioral trends, interests, and emotional states, and companies can use this information to optimize their product placement and promotion strategies, thereby improving the accuracy of their marketing strategies and store operations.

[0228] Furthermore, users are given rewards through a reward management means. Specifically, points are awarded in exchange for providing visual information, audio information, and emotional data. Users can use these points for their next purchase or for other services. This increases users' motivation to participate and enables the acquisition of more abundant data.

[0229] Specific examples

[0230] For example, consider a scenario in which a user wears smart glasses in a store and walks around the grocery section. While shopping, the user may pause for a long time on a particular shelf or comment aloud about a certain product. At this time, the emotion engine analyzes the user's visual and audio information to recognize the user's emotions (e.g., interest, surprise, anticipation, etc.). This data is sent in real time to a cloud server via smartphone. On the server, a generative AI tool analyzes the user's behavior and emotions, providing companies with insights such as which products are likely to attract attention and which locations attract emotional interest. At the same time, the user is awarded points in exchange for providing the data, which can be used for their next purchase.

[0231] Example prompts for generative AI models

[0232] Users wear smart glasses when visiting a store and observe product shelves. Based on gaze data and voice data, you need to analyze user sentiment and provide insights that will help optimize retail strategies. Please present how the data should be analyzed, including specific methods and expected results.

[0233] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0234] Step 1:

[0235] The user wears the smart glasses and launches a dedicated app on their smartphone. The smart glasses collect the user's visual information (e.g., eye movements and products they are looking at) and audio information (e.g., what the user says and ambient sounds). The collected visual and audio information is sent to the smartphone via Bluetooth or Wi-Fi.

[0236] Input: User's visual and audio information

[0237] Output: Visual and audio information is sent to a smartphone

[0238] Step 2:

[0239] The smartphone uploads the received visual and audio information to a cloud server via the Internet. The smartphone acts as a communication tool, transmitting data to the server in real time. The smartphone may also format and convert the data.

[0240] Input: Visual and audio information received by the smartphone

[0241] Output: Formatted visual and audio information is sent to a cloud server

[0242] Step 3:

[0243] The server receives visual and audio information uploaded to the cloud. The server first passes this data to the emotion engine, which analyzes the visual and audio information in real time to recognize the user's emotions. The emotion engine analyzes eye movements, facial expressions, and vocal tone to identify the user's emotional state.

[0244] Input: Visual and audio information sent to the cloud server

[0245] Output: User sentiment data

[0246] Step 4:

[0247] The emotional data, visual information, and audio information are returned to the server and analyzed by the Generative AI Means. The Generative AI Means uses deep learning and natural language processing (NLP) to pattern the user's behavior and emotional data. Through this analysis, the Generative AI Means extracts the user's behavioral tendencies and interests.

[0248] Input: User's emotional data, visual information, and audio information

[0249] Output: User behavioral trends and emotional patterns

[0250] Step 5:

[0251] The analyzed data is stored in a database on a server and provided to companies through data provision means. Companies use the data to develop marketing strategies and store operation improvement measures, thereby optimizing product display placement and improving the accuracy of promotion strategies.

[0252] Input: Analysis results obtained by generative AI means

[0253] Output: Data provided to the company

[0254] Step 6:

[0255] Users are rewarded through a reward management system. Points are awarded based on the provision of visual, audio, and emotional data. These points can be used for future purchases or other services. This motivates users to participate in the system and collect more data.

[0256] Input: User-provided visual, audio, and emotional data

[0257] Output: Points awarded to the user

[0258] The above steps will result in a system that collects and analyzes highly accurate consumer data in real time, optimizing corporate marketing and store operations.

[0259] (Application example 2)

[0260] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0261] In conventional marketing and store operations, there was a lack of means to accurately collect and analyze user behavior and emotional data. As a result, it was difficult for companies to develop optimal marketing strategies and store operations based on users' interests and emotions. In addition, there was little incentive for users to actively participate in providing data.

[0262] The identification processing by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a generation AI means for analyzing and patterning collected visual information and audio information, an emotion engine for recognizing a user's emotions in real time based on the user's visual information and audio information, and a data providing means for providing analysis results, including emotion data obtained by the emotion engine, to companies. This enables highly accurate data collection and analysis based on user behavior and emotions, enabling companies to implement more effective marketing strategies and store operations, and providing appropriate incentives for users to participate in data provision.

[0263] An "information processing device" is a device that collects visual and audio information in real time and exchanges data with the outside world through communication means.

[0264] The "communication means" is a means for transmitting data collected from an information processing device to a cloud server, and refers mainly to a device or system that communicates via the Internet.

[0265] "Visual information" refers to video data about the environment and objects that the user sees.

[0266] "Audio information" refers to audio data related to the user's speech and surrounding environmental sounds.

[0267] A "server" is a computer system that receives collected visual and audio information and performs analysis and data management.

[0268] "Generative AI methods" refers to artificial intelligence techniques that use deep learning and natural language processing to analyze and pattern collected visual and audio information.

[0269] An "emotion engine" is a technology or system that recognizes emotions in real time from a user's eye movements, facial expressions, and tone of voice.

[0270] "Data provision means" refers to the means for providing analyzed data to companies, and mainly refers to systems that distribute data via APIs or databases.

[0271] "Reward management means" refers to a means for providing rewards to users in exchange for providing data, and specifically refers to point systems and digital coupons.

[0272] The present invention is a system that uses a terminal equipped with an information processing device and communication means worn by the user, a server, a generating AI means, a data providing means, a reward management means, and an emotion engine, and is useful for optimizing corporate marketing and store operations.

[0273] First, the user wears an information processing device such as smart glasses and launches a dedicated application on their smartphone. The smart glasses collect visual and audio information in real time and send the data to a cloud server via the smartphone. The visual information includes the user's eye movements and information about products they are looking at, while the audio information includes the user's comments and environmental sounds.

[0274] The collected data is first collected on the smartphone and then sent via the internet to a cloud server. The server receives the collected visual and audio information and analyzes the data using a generative AI means and emotion engine. The generative AI means uses deep learning and natural language processing to extract user behavior and emotional patterns, while the emotion engine recognizes the user's emotions in real time from eye movements, facial expressions, and tone of voice.

[0275] The analysis results are stored in a database and provided to companies through data provision means. Companies can use this data to optimize product display placement, develop promotional strategies, and position products to attract consumer interest.

[0276] Furthermore, the reward management means has a function of awarding points to users in exchange for providing data. Users can use these points for their next purchase or for other services, which increases users' motivation to participate and enables the acquisition of higher quality data.

[0277] As a concrete example, consider a scenario in which a user wears smart glasses in a store and walks around the grocery section. While enjoying their usual shopping, the user may pause at a particular shelf for a long time or comment aloud about a particular product. At this time, the emotion engine analyzes the user's visual and audio information and recognizes the user's emotions (interest, surprise, expectation, etc.). This data is sent in real time to a cloud server, where a generative AI tool analyzes the user's behavior and emotions and provides insights useful for store management.

[0278] A specific example of an input prompt sentence using a generative AI model is as follows:

[0279] Example prompt sentence:

[0280] Use user behavioral and emotional data to perform a detailed analysis of the products and emotions that users are interested in within the grocery store. Specifically, report on which shelves users spend the most time looking at and which products they show interest or anticipation for. Provide detailed insights by taking into account users' eye movements, facial expressions, and tone of voice.

[0281] In this way, the present invention enhances corporate marketing strategies and store operations, while at the same time encouraging improvements in the quality and quantity of data collection by providing rewards to users.

[0282] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0283] Step 1:

[0284] The user puts on the smart glasses and launches a dedicated application on their smartphone. At this point, the smart glasses begin collecting visual and audio information. Inputs include the user's eye movements, speech, and ambient sounds, and the output is data stored in the smart glasses. Specifically, the smart glasses capture video and audio using a camera and microphone and transfer this to the smartphone.

[0285] Step 2:

[0286] The device (smartphone) receives visual and audio information from the smart glasses and sends it to a cloud server. The input is the video and audio data transferred from the smart glasses, and the output is that this data is uploaded to the server via the internet. Specifically, the smartphone connects to the internet and sends the collected data to the cloud server in real time.

[0287] Step 3:

[0288] The server analyzes the visual and audio information it receives using a generation AI means and emotion engine. Inputs include video data, audio data, and the user's emotional state uploaded to the cloud server. Outputs include analyzing the user's behavioral patterns and emotional data, and generating patterned data. Specifically, the generation AI means analyzes the video and audio using deep learning, and the emotion engine identifies emotions from eye movements and facial expressions.

[0289] Step 4:

[0290] The server provides the generated analytical data to the company through the data provision means. The input includes data patterned by the generative AI means and the emotion engine, and the output includes insight data received by the company. Specifically, the analytical data is stored in a database, and access rights are granted to the company via an API.

[0291] Step 5:

[0292] The reward management means awards rewards to users. The input is the user's record of providing behavioral data and emotional data, and the output is points awarded to the user. Specifically, points are automatically calculated according to the amount of data provided by the user and added to the user's account.

[0293] This series of processes will enable the realization of a system that collects and analyzes user behavior and emotion data, provides useful insights to companies, and awards appropriate rewards to users. The use of a generative AI model and emotion engine enables highly accurate analysis, which is expected to be extremely useful in optimizing corporate marketing strategies and store operations.

[0294] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0295] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0296] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0297] [Second embodiment]

[0298] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0299] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0300] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0301] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0302] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0303] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0304] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0305] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0306] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0307] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0308] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0309] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0310] The present invention is a system that uses a terminal equipped with an information processing device and communication means worn by the user, a server, a generating AI means, a data providing means, and a reward management means, and collects, analyzes, and provides highly accurate data that is useful for optimizing corporate marketing and store operations. Specific examples are shown below.

[0311] Data collection by terminal

[0312] The user wears the information processing device and launches a dedicated app. The information processing device takes the form of smart glasses, and collects visual and audio information through communication means and sends it to a cloud server via a smartphone.

[0313] When a user enters a store, the smart glasses automatically begin recording. The user's visual information (e.g., their behavior as they look at the shelves) and audio information (e.g., their comments about products) are collected in real time. This information is then collected on the smartphone and sent to a cloud server via the Internet.

[0314] Data analysis by server

[0315] The server analyzes the received visual and audio information using a generation AI means. The generation AI means uses advanced algorithms such as deep learning and natural language processing (NLP) to analyze eye movement patterns and speech content, creating patterns. These patterns make it possible to extract the user's behavioral tendencies and interests.

[0316] Providing data to companies

[0317] The analyzed data is provided to companies through data provision means. Based on this data, companies can optimize product display locations, develop promotional strategies, and arrange products to attract consumer interest, thereby improving their marketing strategies and store operations.

[0318] Rewarding

[0319] In addition, users are rewarded through a reward management means. Specifically, points are awarded to users based on the collected and provided data. These points can be used for the next purchase or use of other services. This increases users' motivation to participate and enables the acquisition of more abundant data.

[0320] Specific examples

[0321] For example, consider a scenario in which a user wears smart glasses in a store and walks around the grocery section. While enjoying their usual shopping routine, the user may pause at a particular shelf for a long time or comment aloud about a particular product. This data is collected in real time and sent to a server via smartphone. On the server, a generative AI tool analyzes the user's behavior and provides companies with insights such as which products are likely to attract attention and which areas are often overlooked. At the same time, the user earns points in exchange for providing the data, which can be used on their next purchase.

[0322] The present invention aims to improve the quality and quantity of data collection by not only utilizing the data collected in this way for corporate marketing strategies and store operations, but also by providing direct rewards to users.

[0323] The processing flow will be explained below.

[0324] Step 1:

[0325] The user wears the smart glasses and launches a dedicated app on their smartphone, which connects to the smart glasses via Bluetooth or Wi-Fi.

[0326] Step 2:

[0327] When a user enters a store, a dedicated app automatically activates the smart glasses' camera and microphone to begin collecting visual and audio information.

[0328] Step 3:

[0329] The device (smart glasses) collects the user's visual information (eye movements and products they are looking at) and audio information (user comments and ambient sounds) in real time, and the collected data is sent sequentially to a smartphone.

[0330] Step 4:

[0331] The user purchases an item at the cash register and pays using a cashless payment method such as PayPay. The smartphone app associates the purchase data obtained through the payment with visual and audio information.

[0332] Step 5:

[0333] When a user stops recording using the app, the devices (smart glasses and smartphone) upload visual, audio, and payment information to a cloud server.

[0334] Step 6:

[0335] The server receives the uploaded data and analyzes the visual and audio information using a generative AI, which uses a deep learning algorithm to generate patterns from the data and extract user behavioral trends and interests.

[0336] Step 7:

[0337] The server stores the analysis results in a database, organizes the stored data in a format that companies can access, and provides it to them through data provision means.

[0338] Step 8:

[0339] Based on the data provided, companies can optimize product display locations and develop promotional strategies.

[0340] Step 9:

[0341] In return for providing the data, the server uses a reward management means to give the user points, which the user can use for their next purchase.

[0342] In this way, this system efficiently collects visual and audio information from users, analyzes and patterns it using generative AI, and provides companies with valuable marketing data.In return, users are awarded points in return for providing the data, creating a win-win situation.

[0343] Example 1

[0344] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0345] Conventional marketing data collection systems have difficulty analyzing user behavior and interests with high accuracy, limiting the quality and quantity of collected data. Furthermore, incentives for users are insufficient, making it difficult to obtain proactive data provision. This makes it difficult for companies to optimize their marketing strategies and store operations.

[0346] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0347] In this invention, the server includes a terminal equipped with an information processing device and communication means worn by the user, a means for receiving visual and audio information collected by the terminal, a data analysis means for analyzing and patterning the visual and audio information collected by the server, an information provision means for providing companies with the analysis results obtained by the data analysis means, a reward management means for rewarding users, a communication means for aggregating data collected from the terminal on a smartphone and transmitting it to a cloud server, a processing means for analyzing the visual and audio data using a generative AI model in the cloud, and a patterning means for extracting user behavioral trends and interests based on the collected data. This enables highly accurate analysis of user behavior and interests and the collection of high-quality data. Furthermore, rewarding users encourages proactive data provision.

[0348] An "information processing device" is a wearable device worn by a user that collects visual and audio information.

[0349] The "communication means" is a means having a function of transferring data from an information processing device to a smartphone and transmitting data from the smartphone to a cloud server.

[0350] A "server" is a computer system that operates on the cloud and has the ability to receive and analyze collected visual and audio information.

[0351] "Data analysis means" refers to algorithms or software that analyzes collected visual and audio information and patterns user behavior and interests.

[0352] "Information provision means" refers to the means for providing analyzed data to companies.

[0353] The "reward management means" is a means for giving rewards to users for providing data, and specifically has a function of giving points.

[0354] "Processing means" refers to a means that has the function of analyzing visual and audio information using a generative AI model on the cloud.

[0355] "Patterning means" refers to a means for extracting and patterning a user's behavioral tendencies and interests.

[0356] A "terminal" is a device that is worn by a user and has an information processing device and communication means.

[0357] A "generative AI model" is an artificial intelligence model that uses deep learning and natural language processing to analyze data and extract user behavior patterns.

[0358] This invention is a system that uses a terminal equipped with an information processing device and communication means worn by the user, a server, a generating AI means, a data providing means, and a reward management means, and collects, analyzes, and provides highly accurate data that is useful for optimizing corporate marketing and store operations.

[0359] Data collection by terminal

[0360] The user wears the information processing device and launches a dedicated app. The information processing device takes the form of smart glasses, which collect visual and audio information and send it to a cloud server via a smartphone. The smart glasses are designed to automatically start recording when the user enters a store. Visual information (e.g., the user's actions as they look at product shelves) and audio information (e.g., comments about products) are collected in real time. This information is first collected on the smartphone and then sent to the cloud server via the Internet.

[0361] Data analysis by server

[0362] The server analyzes the received visual and audio information using a generative AI means. The generative AI means uses advanced algorithms such as deep learning and natural language processing (NLP) to generate patterns of eye movement and speech content. This analysis makes it possible to extract the user's behavioral tendencies and interests.

[0363] Providing data to companies

[0364] The analyzed data is provided to companies through data provision means. Based on this data, companies can optimize product display locations, develop promotional strategies, and position products to attract consumer interest. For example, by positioning specific products in locations that are likely to attract attention, companies can improve their marketing strategies and store operations.

[0365] Rewarding

[0366] Users are rewarded through a reward management system. Specifically, based on the data collected and provided, users are awarded points. These points can be used for their next purchase or for using other services, which increases user participation and allows for the acquisition of more data.

[0367] Specific examples

[0368] For example, consider a scenario in which a user wears smart glasses in a store and walks around the grocery section. While shopping as usual, the user may pause at a particular shelf for a long time or comment aloud about a particular product. This data is collected in real time and sent to a server via smartphone. On the server, a generative AI tool analyzes the user's behavior and provides companies with insights such as which products are likely to attract attention and which areas are often overlooked. At the same time, the user earns points in exchange for providing the data, which can be used on their next purchase.

[0369] Prompt Sentence Examples

[0370] "Users gazed at a specific product shelf for about 10 seconds, then commented out loud on the name of the product. The system analyzed their eye movements and the content of their comments, and provided companies with information on which products were most likely to attract attention."

[0371] This invention aims to improve the quality of collected data by analyzing user behavior and interests with high accuracy, and also to increase motivation for providing data by providing rewards to users.

[0372] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0373] Step 1:

[0374] The user wears the information processing device and launches a dedicated app on their smartphone. The device (smart glasses) synchronizes with the launch of the app and begins collecting data. At this time, the device collects the user's visual information (which shelf they are looking at) and audio information (what they are saying) and converts it into digital format. The input is the visual and audio data captured by the smart glasses, and the output is the visual and audio data converted into digital format.

[0375] Specific examples of behavior:

[0376] When a user enters a store, the smart glasses automatically start recording.

[0377] When the app is launched, the device captures the user's eye movements and voice in real time.

[0378] Step 2:

[0379] The device transfers the collected visual and audio information to a smartphone, which temporarily stores the data and sends it to a cloud server via the Internet. The input is the digital data sent from the device, and the output is the data sent to the cloud server.

[0380] Specific examples of behavior:

[0381] As the user moves around the store, the smart glasses continuously transmit the information they collect to their smartphone.

[0382] The smartphone sends the data to a cloud server at regular intervals.

[0383] Step 3:

[0384] The server runs on the cloud and receives visual and audio information sent from the smartphone. This data is then analyzed by the generative AI model. The input is the visual and audio data sent to the cloud server, and the output is the analyzed behavioral patterns and interest data. The generative AI model performs advanced analysis using deep learning and natural language processing.

[0385] Specific examples of behavior:

[0386] The server analyzes the received visual data and determines which product shelf the user looked at and for how long.

[0387] The voice data is analyzed and information about products that have piqued the user's interest is extracted from the content of the user's comments.

[0388] Step 4:

[0389] The data analyzed on the server is provided to companies through data provision means. Companies use this data to optimize product display locations and promotion strategies. The input is data on behavioral patterns and interests as a result of the analysis, and the output is insight data provided to companies.

[0390] Specific examples of behavior:

[0391] The server generates reports and automatically sends them to the company's marketing system.

[0392] Based on the reports they receive, companies can take action such as changing the location of product displays.

[0393] Step 5:

[0394] The server provides rewards to users through the reward management means. Specifically, points are given to users for the collected and provided data. The input is the behavioral data as the analysis result, and the output is the user's reward point data.

[0395] Specific examples of behavior:

[0396] Points are automatically credited to the user's account based on the data provided by the user.

[0397] A notification regarding the points being awarded will be sent to the user's smartphone.

[0398] In this way, through a series of processing steps, a system is completed that analyzes user behavior and interests with high accuracy, collects high-quality data, and provides it to companies. Users are rewarded, so they are expected to actively provide data.

[0399] (Application example 1)

[0400] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0401] With conventional methods of collecting marketing data, it was difficult to accurately grasp users' actual purchasing behavior and interests. Furthermore, the analysis of collected data was delayed, making it difficult to change strategies in real time. Furthermore, there was insufficient incentive for users to voluntarily participate in providing data.

[0402] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0403] In this invention, the server includes a means for transmitting visual information to a cloud storage service in real time, a means for analyzing the visual and audio information using a generative AI model, and a means for providing the analysis results to a company's API endpoint. This allows for the collection and analysis of users' visual and audio information in real time and the immediate provision of the analysis results to the company. Furthermore, by awarding users points that can be used toward their next purchase, incentives for data provision are increased, enabling the collection of higher quality data.

[0404] An "information processing device" is a device worn by a user that collects visual and audio information and transmits the data in real time.

[0405] "Communication means" refers to means for transmitting and receiving data between an information processing device and a server, and includes communication via the Internet or cloud storage.

[0406] "Visual information" refers to video data that a user perceives visually, and is information collected using a photographic device such as a camera.

[0407] "Audio information" refers to audio data including the user's speech and surrounding sounds, and is information collected using a microphone or the like.

[0408] "Server" refers to a computer system for receiving and analyzing collected visual and audio information.

[0409] "Generative AI methods" refer to algorithms and systems that use deep learning and natural language processing to analyze collected data and pattern user behavioral trends.

[0410] "Data provision means" refers to the means for providing the analysis results of the generation AI means to the company, and sends the data to the company's API endpoint via the Internet.

[0411] "Reward management means" refers to a system that provides rewards to users in exchange for providing data, such as a mechanism for awarding points.

[0412] A "cloud storage service" is a storage service that enables the storage and retrieval of data via the Internet, and is used to store video data and analysis results.

[0413] "Generative AI model" refers to an advanced machine learning model used to analyze visual and audio information, identify user behavior patterns, and analyze areas of interest, etc.

[0414] "API endpoint" refers to the interface provided by a company to receive data and is used to automatically transmit analysis results.

[0415] This invention is a system that uses a terminal equipped with an information processing device and communication means worn by the user, a server, a generating AI means, a data providing means, and a reward management means. This system collects and analyzes the visual and audio information of the user and provides it to companies to support the optimization of marketing strategies and store operations.

[0416] First, the device includes an information processing device (smart glasses) worn by the user and a communication means (smartphone). The smart glasses are equipped with a camera and microphone to collect visual and audio information from the user. The communication means is responsible for sending the collected information to a cloud server via the Internet.

[0417] The server functions as a cloud server and receives the collected visual and audio information. The server then analyzes the data using a generative AI model that utilizes deep learning and natural language processing to identify patterns of user behavior.

[0418] The data analyzed by the Generative AI Method is provided to the company via the Data Provision Method. Specifically, the analysis results are automatically sent to the API endpoint provided by the company. This allows companies to understand consumer interests in real time and use this information to improve their product displays and promotional strategies.

[0419] Meanwhile, the reward management means acts as an incentive for users. Points are awarded based on the data provided by the user, and these points can be used for future purchases, etc. This increases user participation and allows for the collection of more data.

[0420] As a concrete example, consider a scenario where a user visits the fruit section of a supermarket. While wearing smart glasses, the user looks at a particular fruit for a long time and comments aloud about the fruit's quality. This information is sent to a cloud server in real time and analyzed by a generative AI model, which provides the company with analytical results such as "bananas are attracting the most attention" or "many consumers are commenting on the price of apples." Based on these analytical results, the company can make more effective product placement and pricing decisions.

[0421] An example prompt is:

[0422] "Based on scenarios where users walk through a store and their eyes are drawn to specific products, create prompts that analyze which products and areas are most relevant. Also, analyze comments users make about specific products to gauge their level of interest."

[0423] As described above, this invention solves conventional problems and realizes optimization of marketing and store operations by using an information processing device, communication means, server, generation AI means, data provision means, and reward management means.

[0424] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0425] Step 1:

[0426] A user puts on smart glasses and launches the application. The smart glasses use a camera and microphone to collect visual information (video data) and audio information (audio data). The input data is the user's view of the shelves and store interior, as well as the user's conversations and comments. The output is raw video and audio data.

[0427] Step 2:

[0428] The device (smartphone) receives the visual and audio information collected from the smart glasses and temporarily stores it in storage. The input is the video and audio data sent from the smart glasses, and the output is the temporary data stored on the smartphone.

[0429] Step 3:

[0430] The smartphone transmits the collected visual and audio information via the Internet to a cloud server, where the data is appropriately compressed and uploaded to a cloud storage service. The input is the video and audio data stored on the smartphone, and the output is the data stored in the cloud storage service.

[0431] Step 4:

[0432] The server receives visual and audio information from cloud storage and analyzes it using a generative AI model. Specifically, it analyzes eye movement patterns using deep learning and audio information using natural language processing (NLP). The input is the video and audio data stored in cloud storage, and the output is the analysis results (e.g., eye movement patterns, content of user comments).

[0433] Step 5:

[0434] The server generates a prompt based on the analysis results and provides the analysis results to the company's API endpoint. The input is the analyzed data and the prompt from the generative AI model, and the output is the analysis results sent to the company. Specifically, the server invokes the generative AI model and sends the analysis results to the company's API as a POST request.

[0435] Step 6:

[0436] The server calculates the reward for the user's data provision and assigns points to the user's account through the reward management means. The input is the analysis result and the amount of data provided, and the output is the assigned point information. Specifically, the server accesses the reward management system and updates the points in the user's account.

[0437] Step 7:

[0438] The user uses the points they have earned on their next purchase. The input is the points information earned on the user's account, and the output is the purchase history using the points. Specifically, the user can use the points to receive a discount or exchange them for a specific product at a store.

[0439] Through this series of steps, highly accurate data is collected while users enjoy shopping in stores, and the analysis results are provided to companies, allowing users to receive rewards at the same time.

[0440] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0441] The present invention is a system that uses a terminal equipped with an information processing device and communication means worn by the user, a server, a generating AI means, a data providing means, a reward management means, and an emotion engine to collect, analyze, and provide highly accurate data that is useful for optimizing corporate marketing and store operations. Specific examples are shown below.

[0442] Data collection by terminal

[0443] The user wears the information processing device and launches a dedicated app on their smartphone. The information processing device takes the form of smart glasses, collects visual and audio information, and sends the data to a cloud server via the smartphone using a communication means.

[0444] When a user enters a store, the smart glasses automatically start recording and begin collecting visual and audio information. The user's visual information (eye movements and products they are looking at) and audio information (user comments and ambient sounds) are collected in real time. This visual and audio data is then collected on the smartphone and sent to a cloud server via the Internet.

[0445] Introducing the Emotion Engine

[0446] The emotion engine recognizes the user's emotions based on visual and audio information collected in real time. The emotion engine analyzes eye movements, facial expressions, and tone of voice to identify the user's emotional state. This emotional data, along with the visual and audio information, is sent to a cloud server and incorporated into the analysis by the generative AI method.

[0447] Data analysis by server

[0448] The server receives the uploaded visual, audio, and emotional data and analyzes it using a generative AI method. The generative AI method uses deep learning and natural language processing (NLP) to analyze the data and identify patterns in the user's behavioral tendencies, interests, and newly acquired emotional data. The results of this analysis are stored in a database.

[0449] Providing data to companies

[0450] The analyzed data is provided to companies through data provision means. Based on this data, companies can optimize product display placement, develop promotional strategies, position products to attract consumers' attention, and even implement marketing tactics that respond to user emotions. This will improve the accuracy of marketing strategies and store operations.

[0451] Rewarding

[0452] In addition, users are given rewards through a reward management means. Specifically, users are given points in exchange for providing visual information, audio information, and emotional data. Users can use these points for their next purchase or for other services. This increases users' motivation to participate and enables the acquisition of more abundant data.

[0453] Specific examples

[0454] For example, consider a scenario in which a user wears smart glasses in a store and walks around the grocery section. While shopping as usual, the user may pause at a particular shelf for a long time or comment aloud about a particular product. At this time, the emotion engine analyzes the user's visual and audio information to recognize the user's emotions (e.g., interest, surprise, anticipation, etc.). This data is sent in real time via smartphone to a cloud server. On the server, a generative AI tool analyzes the user's behavior and emotions, providing companies with insights such as which products are likely to attract attention and which locations attract emotional interest. At the same time, the user is awarded points in exchange for providing the data, which can be used on their next purchase.

[0455] The present invention not only utilizes the collected data in this way to improve corporate marketing strategies and store operations, but also promotes improvements in the quality and quantity of data collection by offering rewards to users. The introduction of an emotion engine makes it possible to provide even more advanced analysis and insights.

[0456] The processing flow will be explained below.

[0457] Step 1:

[0458] The user wears the smart glasses and launches a dedicated app on their smartphone, which connects to the smart glasses via Bluetooth or Wi-Fi.

[0459] Step 2:

[0460] When a user enters a store, the smart glasses' camera and microphone automatically activate and begin collecting visual and audio information.

[0461] Step 3:

[0462] The device (smart glasses) collects the user's visual information (eye movements and images of products they are looking at) and audio information (user comments and surrounding environmental sounds) in real time, and the collected data is sent sequentially to a smartphone.

[0463] Step 4:

[0464] When a user pauses in front of a particular product for a long time or reads a product description, the emotion engine processes this information to recognize the user's emotions (interest, surprise, anticipation, etc.) The emotion engine analyzes visual information and changes in voice tone.

[0465] Step 5:

[0466] A user purchases an item at a cash register and uses a cashless payment method (e.g., a smartphone payment app), which links the payment information with visual and audio information.

[0467] Step 6:

[0468] When the user stops recording using the app, the devices (smart glasses and smartphone) upload the collected visual information, audio information, emotional data, and payment information to a cloud server.

[0469] Step 7:

[0470] The server receives the uploaded data and analyzes and patterns the visual, audio, and emotional data using a generative AI. The generative AI uses deep learning and natural language processing (NLP) algorithms to extract the user's behavioral tendencies, interests, and emotions.

[0471] Step 8:

[0472] The server stores the analysis results in a database, organizes the stored data in a format that companies can access, and provides it to them through data provision means.

[0473] Step 9:

[0474] Companies can use the data provided to optimize product displays and develop promotional strategies. In particular, using emotion data can enable more effective marketing measures, such as product sorting and advertising.

[0475] Step 10:

[0476] In return for providing the data, the server uses a reward management means to give the user points, which the user can use for their next purchase or other services.

[0477] These steps enable efficient collection and analysis of users' visual, audio, and emotional data, providing businesses with highly accurate marketing data. Users are rewarded, which strengthens their motivation to provide data and ensures the smooth functioning of the entire system.

[0478] Example 2

[0479] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0480] In modern marketing and store operations, it is important to collect consumer behavioral and emotional data with high accuracy in real time and analyze it efficiently. However, this requires a system that can perform advanced data analysis without requiring users to spend time or money. Conventional systems have difficulty analyzing emotions in real time or providing highly accurate data, which has hindered companies from quickly optimizing their marketing strategies.

[0481] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0482] In this invention, the server includes a terminal equipped with an information processing device and communication means worn by the user, means for receiving visual information and audio information collected by the terminal, generation AI means in the server for analyzing and patterning the collected visual information and audio information, an emotion engine for analyzing the visual information and audio information in real time and recognizing the user's emotions, data provision means for providing companies with the analysis results obtained by the generation AI means, and reward management means for awarding rewards to users. This enables high-precision collection and analysis of consumer behavior data and emotion data in real time, allowing companies to quickly and effectively optimize their marketing strategies and store operations.

[0483] An "information processing device" is a device worn by a user that collects visual and audio information.

[0484] The "communication means" is a means for transmitting data collected from an information processing device to a smartphone or other device.

[0485] "Terminal" refers to a device equipped with an information processing device and a communication means, which collects and transmits visual and audio information of a user.

[0486] "Visual information" refers to information that users perceive visually, such as the movement of their eyes and the products they notice.

[0487] "Audio information" refers to data related to audio, such as the user's speech and surrounding environmental sounds.

[0488] A "server" is a computer system that receives, analyzes, and stores visual and audio information sent from a terminal.

[0489] "Generative AI means" are means of analyzing collected data and creating patterns using technologies such as deep learning and natural language processing.

[0490] The "emotion engine" is an engine that analyzes visual and audio information collected in real time and recognizes the user's emotions.

[0491] "Data provision means" refers to the means for providing companies with the analysis results obtained by the generation AI means.

[0492] The "reward management means" is a means for awarding rewards to users, and specifically includes a method for awarding points.

[0493] "Points" are units of reward that users receive in exchange for providing data, and can be used for their next purchase or other services.

[0494] This invention is a system that uses a terminal equipped with an information processing device and communication means worn by the user, a server, a generating AI means, a data providing means, a reward management means, and an emotion engine to collect, analyze, and provide highly accurate data that is useful for optimizing corporate marketing and store operations. This system can collect and analyze user behavior and emotion data in real time.

[0495] First, the user puts on a smart-glasses-shaped information processing device. This device is designed to collect visual and audio information. Before entering the store, the user launches a dedicated app on their smartphone. At this time, the device collects visual information (e.g., the user's eye movements and products they are looking at) and audio information (e.g., the user's comments and ambient sounds) in real time and sends it to a cloud server via the smartphone. This information is then uploaded to the server via the Internet.

[0496] The server receives and analyzes the uploaded visual, audio, and emotional data. This analysis uses techniques such as deep learning and natural language processing (NLP). The generative AI means analyzes this data and creates patterns based on the user's behavioral tendencies, interests, and newly acquired emotional data. The emotion engine analyzes the visual and audio information in real time to identify the user's emotional state (e.g., interest, surprise, anticipation, etc.). This enables highly accurate emotion recognition.

[0497] The analyzed data is provided to companies through data provision means. This data includes users' behavioral trends, interests, and emotional states, and companies can use this information to optimize their product placement and promotion strategies, thereby improving the accuracy of their marketing strategies and store operations.

[0498] Furthermore, users are given rewards through a reward management means. Specifically, points are awarded in exchange for providing visual information, audio information, and emotional data. Users can use these points for their next purchase or for other services. This increases users' motivation to participate and enables the acquisition of more abundant data.

[0499] Specific examples

[0500] For example, consider a scenario in which a user wears smart glasses in a store and walks around the grocery section. While shopping, the user may pause for a long time on a particular shelf or comment aloud about a certain product. At this time, the emotion engine analyzes the user's visual and audio information to recognize the user's emotions (e.g., interest, surprise, anticipation, etc.). This data is sent in real time to a cloud server via smartphone. On the server, a generative AI tool analyzes the user's behavior and emotions, providing companies with insights such as which products are likely to attract attention and which locations attract emotional interest. At the same time, the user is awarded points in exchange for providing the data, which can be used for their next purchase.

[0501] Example prompts for generative AI models

[0502] Users wear smart glasses when visiting a store and observe product shelves. Based on gaze data and voice data, you need to analyze user sentiment and provide insights that will help optimize retail strategies. Please present how the data should be analyzed, including specific methods and expected results.

[0503] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0504] Step 1:

[0505] The user wears the smart glasses and launches a dedicated app on their smartphone. The smart glasses collect the user's visual information (e.g., eye movements and products they are looking at) and audio information (e.g., what the user says and ambient sounds). The collected visual and audio information is sent to the smartphone via Bluetooth or Wi-Fi.

[0506] Input: User's visual and audio information

[0507] Output: Visual and audio information is sent to a smartphone

[0508] Step 2:

[0509] The smartphone uploads the received visual and audio information to a cloud server via the Internet. The smartphone acts as a communication tool, transmitting data to the server in real time. The smartphone may also format and convert the data.

[0510] Input: Visual and audio information received by the smartphone

[0511] Output: Formatted visual and audio information is sent to a cloud server

[0512] Step 3:

[0513] The server receives visual and audio information uploaded to the cloud. The server first passes this data to the emotion engine, which analyzes the visual and audio information in real time to recognize the user's emotions. The emotion engine analyzes eye movements, facial expressions, and vocal tone to identify the user's emotional state.

[0514] Input: Visual and audio information sent to the cloud server

[0515] Output: User sentiment data

[0516] Step 4:

[0517] The emotional data, visual information, and audio information are returned to the server and analyzed by the Generative AI Means. The Generative AI Means uses deep learning and natural language processing (NLP) to pattern the user's behavior and emotional data. Through this analysis, the Generative AI Means extracts the user's behavioral tendencies and interests.

[0518] Input: User's emotional data, visual information, and audio information

[0519] Output: User behavioral trends and emotional patterns

[0520] Step 5:

[0521] The analyzed data is stored in a database on a server and provided to companies through data provision means. Companies use the data to develop marketing strategies and store operation improvement measures, thereby optimizing product display placement and improving the accuracy of promotion strategies.

[0522] Input: Analysis results obtained by generative AI means

[0523] Output: Data provided to the company

[0524] Step 6:

[0525] Users are rewarded through a reward management system. Points are awarded based on the provision of visual, audio, and emotional data. These points can be used for future purchases or other services. This motivates users to participate in the system and collect more data.

[0526] Input: User-provided visual, audio, and emotional data

[0527] Output: Points awarded to the user

[0528] The above steps will result in a system that collects and analyzes highly accurate consumer data in real time, optimizing corporate marketing and store operations.

[0529] (Application example 2)

[0530] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0531] In conventional marketing and store operations, there was a lack of means to accurately collect and analyze user behavior and emotional data. As a result, it was difficult for companies to develop optimal marketing strategies and store operations based on users' interests and emotions. In addition, there was little incentive for users to actively participate in providing data.

[0532] The identification processing by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a generation AI means for analyzing and patterning collected visual information and audio information, an emotion engine for recognizing a user's emotions in real time based on the user's visual information and audio information, and a data providing means for providing analysis results, including emotion data obtained by the emotion engine, to companies. This enables highly accurate data collection and analysis based on user behavior and emotions, enabling companies to implement more effective marketing strategies and store operations, and providing appropriate incentives for users to participate in data provision.

[0533] An "information processing device" is a device that collects visual and audio information in real time and exchanges data with the outside world through communication means.

[0534] The "communication means" is a means for transmitting data collected from an information processing device to a cloud server, and refers mainly to a device or system that communicates via the Internet.

[0535] "Visual information" refers to video data about the environment and objects that the user sees.

[0536] "Audio information" refers to audio data related to the user's speech and surrounding environmental sounds.

[0537] A "server" is a computer system that receives collected visual and audio information and performs analysis and data management.

[0538] "Generative AI methods" refers to artificial intelligence techniques that use deep learning and natural language processing to analyze and pattern collected visual and audio information.

[0539] An "emotion engine" is a technology or system that recognizes emotions in real time from a user's eye movements, facial expressions, and tone of voice.

[0540] "Data provision means" refers to the means for providing analyzed data to companies, and mainly refers to systems that distribute data via APIs or databases.

[0541] "Reward management means" refers to a means for providing rewards to users in exchange for providing data, and specifically refers to point systems and digital coupons.

[0542] The present invention is a system that uses a terminal equipped with an information processing device and communication means worn by the user, a server, a generating AI means, a data providing means, a reward management means, and an emotion engine, and is useful for optimizing corporate marketing and store operations.

[0543] First, the user wears an information processing device such as smart glasses and launches a dedicated application on their smartphone. The smart glasses collect visual and audio information in real time and send the data to a cloud server via the smartphone. The visual information includes the user's eye movements and information about products they are looking at, while the audio information includes the user's comments and environmental sounds.

[0544] The collected data is first collected on the smartphone and then sent via the internet to a cloud server. The server receives the collected visual and audio information and analyzes the data using a generative AI means and emotion engine. The generative AI means uses deep learning and natural language processing to extract user behavior and emotional patterns, while the emotion engine recognizes the user's emotions in real time from eye movements, facial expressions, and tone of voice.

[0545] The analysis results are stored in a database and provided to companies through data provision means. Companies can use this data to optimize product display placement, develop promotional strategies, and position products to attract consumer interest.

[0546] Furthermore, the reward management means has a function of awarding points to users in exchange for providing data. Users can use these points for their next purchase or for other services, which increases users' motivation to participate and enables the acquisition of higher quality data.

[0547] As a concrete example, consider a scenario in which a user wears smart glasses in a store and walks around the grocery section. While enjoying their usual shopping, the user may pause at a particular shelf for a long time or comment aloud about a particular product. At this time, the emotion engine analyzes the user's visual and audio information and recognizes the user's emotions (interest, surprise, expectation, etc.). This data is sent in real time to a cloud server, where a generative AI tool analyzes the user's behavior and emotions and provides insights useful for store management.

[0548] A specific example of an input prompt sentence using a generative AI model is as follows:

[0549] Example prompt sentence:

[0550] Use user behavioral and emotional data to perform a detailed analysis of the products and emotions that users are interested in within the grocery store. Specifically, report on which shelves users spend the most time looking at and which products they show interest or anticipation for. Provide detailed insights by taking into account users' eye movements, facial expressions, and tone of voice.

[0551] In this way, the present invention enhances corporate marketing strategies and store operations, while at the same time encouraging improvements in the quality and quantity of data collection by providing rewards to users.

[0552] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0553] Step 1:

[0554] The user puts on the smart glasses and launches a dedicated application on their smartphone. At this point, the smart glasses begin collecting visual and audio information. Inputs include the user's eye movements, speech, and ambient sounds, and the output is data stored in the smart glasses. Specifically, the smart glasses capture video and audio using a camera and microphone and transfer this to the smartphone.

[0555] Step 2:

[0556] The device (smartphone) receives visual and audio information from the smart glasses and sends it to a cloud server. The input is the video and audio data transferred from the smart glasses, and the output is that this data is uploaded to the server via the internet. Specifically, the smartphone connects to the internet and sends the collected data to the cloud server in real time.

[0557] Step 3:

[0558] The server analyzes the visual and audio information it receives using a generation AI means and emotion engine. Inputs include video data, audio data, and the user's emotional state uploaded to the cloud server. Outputs include analyzing the user's behavioral patterns and emotional data, and generating patterned data. Specifically, the generation AI means analyzes the video and audio using deep learning, and the emotion engine identifies emotions from eye movements and facial expressions.

[0559] Step 4:

[0560] The server provides the generated analytical data to the company through the data provision means. The input includes data patterned by the generative AI means and the emotion engine, and the output includes insight data received by the company. Specifically, the analytical data is stored in a database, and access rights are granted to the company via an API.

[0561] Step 5:

[0562] The reward management means awards rewards to users. The input is the user's record of providing behavioral data and emotional data, and the output is points awarded to the user. Specifically, points are automatically calculated according to the amount of data provided by the user and added to the user's account.

[0563] This series of processes will enable the realization of a system that collects and analyzes user behavior and emotion data, provides useful insights to companies, and awards appropriate rewards to users. The use of a generative AI model and emotion engine enables highly accurate analysis, which is expected to be extremely useful in optimizing corporate marketing strategies and store operations.

[0564] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0565] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0566] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0567] [Third embodiment]

[0568] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0569] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0570] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0571] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0572] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0573] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0574] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0575] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0576] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0577] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0578] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0579] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0580] The present invention is a system that uses a terminal equipped with an information processing device and communication means worn by the user, a server, a generating AI means, a data providing means, and a reward management means, and collects, analyzes, and provides highly accurate data that is useful for optimizing corporate marketing and store operations. Specific examples are shown below.

[0581] Data collection by terminal

[0582] The user wears the information processing device and launches a dedicated app. The information processing device takes the form of smart glasses, and collects visual and audio information through communication means and sends it to a cloud server via a smartphone.

[0583] When a user enters a store, the smart glasses automatically begin recording. The user's visual information (e.g., their behavior as they look at the shelves) and audio information (e.g., their comments about products) are collected in real time. This information is then collected on the smartphone and sent to a cloud server via the Internet.

[0584] Data analysis by server

[0585] The server analyzes the received visual and audio information using a generation AI means. The generation AI means uses advanced algorithms such as deep learning and natural language processing (NLP) to analyze eye movement patterns and speech content, creating patterns. These patterns make it possible to extract the user's behavioral tendencies and interests.

[0586] Providing data to companies

[0587] The analyzed data is provided to companies through data provision means. Based on this data, companies can optimize product display locations, develop promotional strategies, and arrange products to attract consumer interest, thereby improving their marketing strategies and store operations.

[0588] Rewarding

[0589] In addition, users are rewarded through a reward management means. Specifically, points are awarded to users based on the collected and provided data. These points can be used for the next purchase or use of other services. This increases users' motivation to participate and enables the acquisition of more abundant data.

[0590] Specific examples

[0591] For example, consider a scenario in which a user wears smart glasses in a store and walks around the grocery section. While enjoying their usual shopping routine, the user may pause at a particular shelf for a long time or comment aloud about a particular product. This data is collected in real time and sent to a server via smartphone. On the server, a generative AI tool analyzes the user's behavior and provides companies with insights such as which products are likely to attract attention and which areas are often overlooked. At the same time, the user earns points in exchange for providing the data, which can be used on their next purchase.

[0592] The present invention aims to improve the quality and quantity of data collection by not only utilizing the data collected in this way for corporate marketing strategies and store operations, but also by providing direct rewards to users.

[0593] The processing flow will be explained below.

[0594] Step 1:

[0595] The user wears the smart glasses and launches a dedicated app on their smartphone, which connects to the smart glasses via Bluetooth or Wi-Fi.

[0596] Step 2:

[0597] When a user enters a store, a dedicated app automatically activates the smart glasses' camera and microphone to begin collecting visual and audio information.

[0598] Step 3:

[0599] The device (smart glasses) collects the user's visual information (eye movements and products they are looking at) and audio information (user comments and ambient sounds) in real time, and the collected data is sent sequentially to a smartphone.

[0600] Step 4:

[0601] The user purchases an item at the cash register and pays using a cashless payment method such as PayPay. The smartphone app associates the purchase data obtained through the payment with visual and audio information.

[0602] Step 5:

[0603] When a user stops recording using the app, the devices (smart glasses and smartphone) upload visual, audio, and payment information to a cloud server.

[0604] Step 6:

[0605] The server receives the uploaded data and analyzes the visual and audio information using a generative AI, which uses a deep learning algorithm to generate patterns from the data and extract user behavioral trends and interests.

[0606] Step 7:

[0607] The server stores the analysis results in a database, organizes the stored data in a format that companies can access, and provides it to them through data provision means.

[0608] Step 8:

[0609] Based on the data provided, companies can optimize product display locations and develop promotional strategies.

[0610] Step 9:

[0611] In return for providing the data, the server uses a reward management means to give the user points, which the user can use for their next purchase.

[0612] In this way, this system efficiently collects visual and audio information from users, analyzes and patterns it using generative AI, and provides companies with valuable marketing data.In return, users are awarded points in return for providing the data, creating a win-win situation.

[0613] Example 1

[0614] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0615] Conventional marketing data collection systems have difficulty analyzing user behavior and interests with high accuracy, limiting the quality and quantity of collected data. Furthermore, incentives for users are insufficient, making it difficult to obtain proactive data provision. This makes it difficult for companies to optimize their marketing strategies and store operations.

[0616] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0617] In this invention, the server includes a terminal equipped with an information processing device and communication means worn by the user, a means for receiving visual and audio information collected by the terminal, a data analysis means for analyzing and patterning the visual and audio information collected by the server, an information provision means for providing companies with the analysis results obtained by the data analysis means, a reward management means for rewarding users, a communication means for aggregating data collected from the terminal on a smartphone and transmitting it to a cloud server, a processing means for analyzing the visual and audio data using a generative AI model in the cloud, and a patterning means for extracting user behavioral trends and interests based on the collected data. This enables highly accurate analysis of user behavior and interests and the collection of high-quality data. Furthermore, rewarding users encourages proactive data provision.

[0618] An "information processing device" is a wearable device worn by a user that collects visual and audio information.

[0619] The "communication means" is a means having a function of transferring data from an information processing device to a smartphone and transmitting data from the smartphone to a cloud server.

[0620] A "server" is a computer system that operates on the cloud and has the ability to receive and analyze collected visual and audio information.

[0621] "Data analysis means" refers to algorithms or software that analyzes collected visual and audio information and patterns user behavior and interests.

[0622] "Information provision means" refers to the means for providing analyzed data to companies.

[0623] The "reward management means" is a means for giving rewards to users for providing data, and specifically has a function of giving points.

[0624] "Processing means" refers to a means that has the function of analyzing visual and audio information using a generative AI model on the cloud.

[0625] "Patterning means" refers to a means for extracting and patterning a user's behavioral tendencies and interests.

[0626] A "terminal" is a device that is worn by a user and has an information processing device and communication means.

[0627] A "generative AI model" is an artificial intelligence model that uses deep learning and natural language processing to analyze data and extract user behavior patterns.

[0628] This invention is a system that uses a terminal equipped with an information processing device and communication means worn by the user, a server, a generating AI means, a data providing means, and a reward management means, and collects, analyzes, and provides highly accurate data that is useful for optimizing corporate marketing and store operations.

[0629] Data collection by terminal

[0630] The user wears the information processing device and launches a dedicated app. The information processing device takes the form of smart glasses, which collect visual and audio information and send it to a cloud server via a smartphone. The smart glasses are designed to automatically start recording when the user enters a store. Visual information (e.g., the user's actions as they look at product shelves) and audio information (e.g., comments about products) are collected in real time. This information is first collected on the smartphone and then sent to the cloud server via the Internet.

[0631] Data analysis by server

[0632] The server analyzes the received visual and audio information using a generative AI means. The generative AI means uses advanced algorithms such as deep learning and natural language processing (NLP) to generate patterns of eye movement and speech content. This analysis makes it possible to extract the user's behavioral tendencies and interests.

[0633] Providing data to companies

[0634] The analyzed data is provided to companies through data provision means. Based on this data, companies can optimize product display locations, develop promotional strategies, and position products to attract consumer interest. For example, by positioning specific products in locations that are likely to attract attention, companies can improve their marketing strategies and store operations.

[0635] Rewarding

[0636] Users are rewarded through a reward management system. Specifically, based on the data collected and provided, users are awarded points. These points can be used for their next purchase or for using other services, which increases user participation and allows for the acquisition of more data.

[0637] Specific examples

[0638] For example, consider a scenario in which a user wears smart glasses in a store and walks around the grocery section. While shopping as usual, the user may pause at a particular shelf for a long time or comment aloud about a particular product. This data is collected in real time and sent to a server via smartphone. On the server, a generative AI tool analyzes the user's behavior and provides companies with insights such as which products are likely to attract attention and which areas are often overlooked. At the same time, the user earns points in exchange for providing the data, which can be used on their next purchase.

[0639] Prompt Sentence Examples

[0640] "Users gazed at a specific product shelf for about 10 seconds, then commented out loud on the name of the product. The system analyzed their eye movements and the content of their comments, and provided companies with information on which products were most likely to attract attention."

[0641] This invention aims to improve the quality of collected data by analyzing user behavior and interests with high accuracy, and also to increase motivation for providing data by providing rewards to users.

[0642] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0643] Step 1:

[0644] The user wears the information processing device and launches a dedicated app on their smartphone. The device (smart glasses) synchronizes with the launch of the app and begins collecting data. At this time, the device collects the user's visual information (which shelf they are looking at) and audio information (what they are saying) and converts it into digital format. The input is the visual and audio data captured by the smart glasses, and the output is the visual and audio data converted into digital format.

[0645] Specific examples of behavior:

[0646] When a user enters a store, the smart glasses automatically start recording.

[0647] When the app is launched, the device captures the user's eye movements and voice in real time.

[0648] Step 2:

[0649] The device transfers the collected visual and audio information to a smartphone, which temporarily stores the data and sends it to a cloud server via the Internet. The input is the digital data sent from the device, and the output is the data sent to the cloud server.

[0650] Specific examples of behavior:

[0651] As the user moves around the store, the smart glasses continuously transmit the information they collect to their smartphone.

[0652] The smartphone sends the data to a cloud server at regular intervals.

[0653] Step 3:

[0654] The server runs on the cloud and receives visual and audio information sent from the smartphone. This data is then analyzed by the generative AI model. The input is the visual and audio data sent to the cloud server, and the output is the analyzed behavioral patterns and interest data. The generative AI model performs advanced analysis using deep learning and natural language processing.

[0655] Specific examples of behavior:

[0656] The server analyzes the received visual data and determines which product shelf the user looked at and for how long.

[0657] The voice data is analyzed and information about products that have piqued the user's interest is extracted from the content of the user's comments.

[0658] Step 4:

[0659] The data analyzed on the server is provided to companies through data provision means. Companies use this data to optimize product display locations and promotion strategies. The input is data on behavioral patterns and interests as a result of the analysis, and the output is insight data provided to companies.

[0660] Specific examples of behavior:

[0661] The server generates reports and automatically sends them to the company's marketing system.

[0662] Based on the reports they receive, companies can take action such as changing the location of product displays.

[0663] Step 5:

[0664] The server provides rewards to users through the reward management means. Specifically, points are given to users for the collected and provided data. The input is the behavioral data as the analysis result, and the output is the user's reward point data.

[0665] Specific examples of behavior:

[0666] Points are automatically credited to the user's account based on the data provided by the user.

[0667] A notification regarding the points being awarded will be sent to the user's smartphone.

[0668] In this way, through a series of processing steps, a system is completed that analyzes user behavior and interests with high accuracy, collects high-quality data, and provides it to companies. Users are rewarded, so they are expected to actively provide data.

[0669] (Application example 1)

[0670] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0671] With conventional methods of collecting marketing data, it was difficult to accurately grasp users' actual purchasing behavior and interests. Furthermore, the analysis of collected data was delayed, making it difficult to change strategies in real time. Furthermore, there was insufficient incentive for users to voluntarily participate in providing data.

[0672] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0673] In this invention, the server includes a means for transmitting visual information to a cloud storage service in real time, a means for analyzing the visual and audio information using a generative AI model, and a means for providing the analysis results to a company's API endpoint. This allows for the collection and analysis of users' visual and audio information in real time and the immediate provision of the analysis results to the company. Furthermore, by awarding users points that can be used toward their next purchase, incentives for data provision are increased, enabling the collection of higher quality data.

[0674] An "information processing device" is a device worn by a user that collects visual and audio information and transmits the data in real time.

[0675] "Communication means" refers to means for transmitting and receiving data between an information processing device and a server, and includes communication via the Internet or cloud storage.

[0676] "Visual information" refers to video data that a user perceives visually, and is information collected using a photographic device such as a camera.

[0677] "Audio information" refers to audio data including the user's speech and surrounding sounds, and is information collected using a microphone or the like.

[0678] "Server" refers to a computer system for receiving and analyzing collected visual and audio information.

[0679] "Generative AI methods" refer to algorithms and systems that use deep learning and natural language processing to analyze collected data and pattern user behavioral trends.

[0680] "Data provision means" refers to the means for providing the analysis results of the generation AI means to the company, and sends the data to the company's API endpoint via the Internet.

[0681] "Reward management means" refers to a system that provides rewards to users in exchange for providing data, such as a mechanism for awarding points.

[0682] A "cloud storage service" is a storage service that enables the storage and retrieval of data via the Internet, and is used to store video data and analysis results.

[0683] "Generative AI model" refers to an advanced machine learning model used to analyze visual and audio information, identify user behavior patterns, and analyze areas of interest, etc.

[0684] "API endpoint" refers to the interface provided by a company to receive data and is used to automatically transmit analysis results.

[0685] This invention is a system that uses a terminal equipped with an information processing device and communication means worn by the user, a server, a generating AI means, a data providing means, and a reward management means. This system collects and analyzes the visual and audio information of the user and provides it to companies to support the optimization of marketing strategies and store operations.

[0686] First, the device includes an information processing device (smart glasses) worn by the user and a communication means (smartphone). The smart glasses are equipped with a camera and microphone to collect visual and audio information from the user. The communication means is responsible for sending the collected information to a cloud server via the Internet.

[0687] The server functions as a cloud server and receives the collected visual and audio information. The server then analyzes the data using a generative AI model that utilizes deep learning and natural language processing to identify patterns of user behavior.

[0688] The data analyzed by the Generative AI Method is provided to the company via the Data Provision Method. Specifically, the analysis results are automatically sent to the API endpoint provided by the company. This allows companies to understand consumer interests in real time and use this information to improve their product displays and promotional strategies.

[0689] Meanwhile, the reward management means acts as an incentive for users. Points are awarded based on the data provided by the user, and these points can be used for future purchases, etc. This increases user participation and allows for the collection of more data.

[0690] As a concrete example, consider a scenario where a user visits the fruit section of a supermarket. While wearing smart glasses, the user looks at a particular fruit for a long time and comments aloud about the fruit's quality. This information is sent to a cloud server in real time and analyzed by a generative AI model, which provides the company with analytical results such as "bananas are attracting the most attention" or "many consumers are commenting on the price of apples." Based on these analytical results, the company can make more effective product placement and pricing decisions.

[0691] An example prompt is:

[0692] "Based on scenarios where users walk through a store and their eyes are drawn to specific products, create prompts that analyze which products and areas are most relevant. Also, analyze comments users make about specific products to gauge their level of interest."

[0693] As described above, this invention solves conventional problems and realizes optimization of marketing and store operations by using an information processing device, communication means, server, generation AI means, data provision means, and reward management means.

[0694] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0695] Step 1:

[0696] A user puts on smart glasses and launches the application. The smart glasses use a camera and microphone to collect visual information (video data) and audio information (audio data). The input data is the user's view of the shelves and store interior, as well as the user's conversations and comments. The output is raw video and audio data.

[0697] Step 2:

[0698] The device (smartphone) receives the visual and audio information collected from the smart glasses and temporarily stores it in storage. The input is the video and audio data sent from the smart glasses, and the output is the temporary data stored on the smartphone.

[0699] Step 3:

[0700] The smartphone transmits the collected visual and audio information via the Internet to a cloud server, where the data is appropriately compressed and uploaded to a cloud storage service. The input is the video and audio data stored on the smartphone, and the output is the data stored in the cloud storage service.

[0701] Step 4:

[0702] The server receives visual and audio information from cloud storage and analyzes it using a generative AI model. Specifically, it analyzes eye movement patterns using deep learning and audio information using natural language processing (NLP). The input is the video and audio data stored in cloud storage, and the output is the analysis results (e.g., eye movement patterns, content of user comments).

[0703] Step 5:

[0704] The server generates a prompt based on the analysis results and provides the analysis results to the company's API endpoint. The input is the analyzed data and the prompt from the generative AI model, and the output is the analysis results sent to the company. Specifically, the server invokes the generative AI model and sends the analysis results to the company's API as a POST request.

[0705] Step 6:

[0706] The server calculates the reward for the user's data provision and assigns points to the user's account through the reward management means. The input is the analysis result and the amount of data provided, and the output is the assigned point information. Specifically, the server accesses the reward management system and updates the points in the user's account.

[0707] Step 7:

[0708] The user uses the points they have earned on their next purchase. The input is the points information earned on the user's account, and the output is the purchase history using the points. Specifically, the user can use the points to receive a discount or exchange them for a specific product at a store.

[0709] Through this series of steps, highly accurate data is collected while users enjoy shopping in stores, and the analysis results are provided to companies, allowing users to receive rewards at the same time.

[0710] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0711] The present invention is a system that uses a terminal equipped with an information processing device and communication means worn by the user, a server, a generating AI means, a data providing means, a reward management means, and an emotion engine to collect, analyze, and provide highly accurate data that is useful for optimizing corporate marketing and store operations. Specific examples are shown below.

[0712] Data collection by terminal

[0713] The user wears the information processing device and launches a dedicated app on their smartphone. The information processing device takes the form of smart glasses, collects visual and audio information, and sends the data to a cloud server via the smartphone using a communication means.

[0714] When a user enters a store, the smart glasses automatically start recording and begin collecting visual and audio information. The user's visual information (eye movements and products they are looking at) and audio information (user comments and ambient sounds) are collected in real time. This visual and audio data is then collected on the smartphone and sent to a cloud server via the Internet.

[0715] Introducing the Emotion Engine

[0716] The emotion engine recognizes the user's emotions based on visual and audio information collected in real time. The emotion engine analyzes eye movements, facial expressions, and tone of voice to identify the user's emotional state. This emotional data, along with the visual and audio information, is sent to a cloud server and incorporated into the analysis by the generative AI method.

[0717] Data analysis by server

[0718] The server receives the uploaded visual, audio, and emotional data and analyzes it using a generative AI method. The generative AI method uses deep learning and natural language processing (NLP) to analyze the data and identify patterns in the user's behavioral tendencies, interests, and newly acquired emotional data. The results of this analysis are stored in a database.

[0719] Providing data to companies

[0720] The analyzed data is provided to companies through data provision means. Based on this data, companies can optimize product display placement, develop promotional strategies, position products to attract consumers' attention, and even implement marketing tactics that respond to user emotions. This will improve the accuracy of marketing strategies and store operations.

[0721] Rewarding

[0722] In addition, users are given rewards through a reward management means. Specifically, users are given points in exchange for providing visual information, audio information, and emotional data. Users can use these points for their next purchase or for other services. This increases users' motivation to participate and enables the acquisition of more abundant data.

[0723] Specific examples

[0724] For example, consider a scenario in which a user wears smart glasses in a store and walks around the grocery section. While shopping as usual, the user may pause at a particular shelf for a long time or comment aloud about a particular product. At this time, the emotion engine analyzes the user's visual and audio information to recognize the user's emotions (e.g., interest, surprise, anticipation, etc.). This data is sent in real time via smartphone to a cloud server. On the server, a generative AI tool analyzes the user's behavior and emotions, providing companies with insights such as which products are likely to attract attention and which locations attract emotional interest. At the same time, the user is awarded points in exchange for providing the data, which can be used on their next purchase.

[0725] The present invention not only utilizes the collected data in this way to improve corporate marketing strategies and store operations, but also promotes improvements in the quality and quantity of data collection by offering rewards to users. The introduction of an emotion engine makes it possible to provide even more advanced analysis and insights.

[0726] The processing flow will be explained below.

[0727] Step 1:

[0728] The user wears the smart glasses and launches a dedicated app on their smartphone, which connects to the smart glasses via Bluetooth or Wi-Fi.

[0729] Step 2:

[0730] When a user enters a store, the smart glasses' camera and microphone automatically activate and begin collecting visual and audio information.

[0731] Step 3:

[0732] The device (smart glasses) collects the user's visual information (eye movements and images of products they are looking at) and audio information (user comments and surrounding environmental sounds) in real time, and the collected data is sent sequentially to a smartphone.

[0733] Step 4:

[0734] When a user pauses in front of a particular product for a long time or reads a product description, the emotion engine processes this information to recognize the user's emotions (interest, surprise, anticipation, etc.) The emotion engine analyzes visual information and changes in voice tone.

[0735] Step 5:

[0736] A user purchases an item at a cash register and uses a cashless payment method (e.g., a smartphone payment app), which links the payment information with visual and audio information.

[0737] Step 6:

[0738] When the user stops recording using the app, the devices (smart glasses and smartphone) upload the collected visual information, audio information, emotional data, and payment information to a cloud server.

[0739] Step 7:

[0740] The server receives the uploaded data and analyzes and patterns the visual, audio, and emotional data using a generative AI. The generative AI uses deep learning and natural language processing (NLP) algorithms to extract the user's behavioral tendencies, interests, and emotions.

[0741] Step 8:

[0742] The server stores the analysis results in a database, organizes the stored data in a format that companies can access, and provides it to them through data provision means.

[0743] Step 9:

[0744] Companies can use the data provided to optimize product displays and develop promotional strategies. In particular, using emotion data can enable more effective marketing measures, such as product sorting and advertising.

[0745] Step 10:

[0746] In return for providing the data, the server uses a reward management means to give the user points, which the user can use for their next purchase or other services.

[0747] These steps enable efficient collection and analysis of users' visual, audio, and emotional data, providing businesses with highly accurate marketing data. Users are rewarded, which strengthens their motivation to provide data and ensures the smooth functioning of the entire system.

[0748] Example 2

[0749] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0750] In modern marketing and store operations, it is important to collect consumer behavioral and emotional data with high accuracy in real time and analyze it efficiently. However, this requires a system that can perform advanced data analysis without requiring users to spend time or money. Conventional systems have difficulty analyzing emotions in real time or providing highly accurate data, which has hindered companies from quickly optimizing their marketing strategies.

[0751] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0752] In this invention, the server includes a terminal equipped with an information processing device and communication means worn by the user, means for receiving visual information and audio information collected by the terminal, generation AI means in the server for analyzing and patterning the collected visual information and audio information, an emotion engine for analyzing the visual information and audio information in real time and recognizing the user's emotions, data provision means for providing companies with the analysis results obtained by the generation AI means, and reward management means for awarding rewards to users. This enables high-precision collection and analysis of consumer behavior data and emotion data in real time, allowing companies to quickly and effectively optimize their marketing strategies and store operations.

[0753] An "information processing device" is a device worn by a user that collects visual and audio information.

[0754] The "communication means" is a means for transmitting data collected from an information processing device to a smartphone or other device.

[0755] "Terminal" refers to a device equipped with an information processing device and a communication means, which collects and transmits visual and audio information of a user.

[0756] "Visual information" refers to information that users perceive visually, such as the movement of their eyes and the products they notice.

[0757] "Audio information" refers to data related to audio, such as the user's speech and surrounding environmental sounds.

[0758] A "server" is a computer system that receives, analyzes, and stores visual and audio information sent from a terminal.

[0759] "Generative AI means" are means of analyzing collected data and creating patterns using technologies such as deep learning and natural language processing.

[0760] The "emotion engine" is an engine that analyzes visual and audio information collected in real time and recognizes the user's emotions.

[0761] "Data provision means" refers to the means for providing companies with the analysis results obtained by the generation AI means.

[0762] The "reward management means" is a means for awarding rewards to users, and specifically includes a method for awarding points.

[0763] "Points" are units of reward that users receive in exchange for providing data, and can be used for their next purchase or other services.

[0764] This invention is a system that uses a terminal equipped with an information processing device and communication means worn by the user, a server, a generating AI means, a data providing means, a reward management means, and an emotion engine to collect, analyze, and provide highly accurate data that is useful for optimizing corporate marketing and store operations. This system can collect and analyze user behavior and emotion data in real time.

[0765] First, the user puts on a smart-glasses-shaped information processing device. This device is designed to collect visual and audio information. Before entering the store, the user launches a dedicated app on their smartphone. At this time, the device collects visual information (e.g., the user's eye movements and products they are looking at) and audio information (e.g., the user's comments and ambient sounds) in real time and sends it to a cloud server via the smartphone. This information is then uploaded to the server via the Internet.

[0766] The server receives and analyzes the uploaded visual, audio, and emotional data. This analysis uses techniques such as deep learning and natural language processing (NLP). The generative AI means analyzes this data and creates patterns based on the user's behavioral tendencies, interests, and newly acquired emotional data. The emotion engine analyzes the visual and audio information in real time to identify the user's emotional state (e.g., interest, surprise, anticipation, etc.). This enables highly accurate emotion recognition.

[0767] The analyzed data is provided to companies through data provision means. This data includes users' behavioral trends, interests, and emotional states, and companies can use this information to optimize their product placement and promotion strategies, thereby improving the accuracy of their marketing strategies and store operations.

[0768] Furthermore, users are given rewards through a reward management means. Specifically, points are awarded in exchange for providing visual information, audio information, and emotional data. Users can use these points for their next purchase or for other services. This increases users' motivation to participate and enables the acquisition of more abundant data.

[0769] Specific examples

[0770] For example, consider a scenario in which a user wears smart glasses in a store and walks around the grocery section. While shopping, the user may pause for a long time on a particular shelf or comment aloud about a certain product. At this time, the emotion engine analyzes the user's visual and audio information to recognize the user's emotions (e.g., interest, surprise, anticipation, etc.). This data is sent in real time to a cloud server via smartphone. On the server, a generative AI tool analyzes the user's behavior and emotions, providing companies with insights such as which products are likely to attract attention and which locations attract emotional interest. At the same time, the user is awarded points in exchange for providing the data, which can be used for their next purchase.

[0771] Example prompts for generative AI models

[0772] Users wear smart glasses when visiting a store and observe product shelves. Based on gaze data and voice data, you need to analyze user sentiment and provide insights that will help optimize retail strategies. Please present how the data should be analyzed, including specific methods and expected results.

[0773] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0774] Step 1:

[0775] The user wears the smart glasses and launches a dedicated app on their smartphone. The smart glasses collect the user's visual information (e.g., eye movements and products they are looking at) and audio information (e.g., what the user says and ambient sounds). The collected visual and audio information is sent to the smartphone via Bluetooth or Wi-Fi.

[0776] Input: User's visual and audio information

[0777] Output: Visual and audio information is sent to a smartphone

[0778] Step 2:

[0779] The smartphone uploads the received visual and audio information to a cloud server via the Internet. The smartphone acts as a communication tool, transmitting data to the server in real time. The smartphone may also format and convert the data.

[0780] Input: Visual and audio information received by the smartphone

[0781] Output: Formatted visual and audio information is sent to a cloud server

[0782] Step 3:

[0783] The server receives visual and audio information uploaded to the cloud. The server first passes this data to the emotion engine, which analyzes the visual and audio information in real time to recognize the user's emotions. The emotion engine analyzes eye movements, facial expressions, and vocal tone to identify the user's emotional state.

[0784] Input: Visual and audio information sent to the cloud server

[0785] Output: User sentiment data

[0786] Step 4:

[0787] The emotional data, visual information, and audio information are returned to the server and analyzed by the Generative AI Means. The Generative AI Means uses deep learning and natural language processing (NLP) to pattern the user's behavior and emotional data. Through this analysis, the Generative AI Means extracts the user's behavioral tendencies and interests.

[0788] Input: User's emotional data, visual information, and audio information

[0789] Output: User behavioral trends and emotional patterns

[0790] Step 5:

[0791] The analyzed data is stored in a database on a server and provided to companies through data provision means. Companies use the data to develop marketing strategies and store operation improvement measures, thereby optimizing product display placement and improving the accuracy of promotion strategies.

[0792] Input: Analysis results obtained by generative AI means

[0793] Output: Data provided to the company

[0794] Step 6:

[0795] Users are rewarded through a reward management system. Points are awarded based on the provision of visual, audio, and emotional data. These points can be used for future purchases or other services. This motivates users to participate in the system and collect more data.

[0796] Input: User-provided visual, audio, and emotional data

[0797] Output: Points awarded to the user

[0798] The above steps will result in a system that collects and analyzes highly accurate consumer data in real time, optimizing corporate marketing and store operations.

[0799] (Application example 2)

[0800] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0801] In conventional marketing and store operations, there was a lack of means to accurately collect and analyze user behavior and emotional data. As a result, it was difficult for companies to develop optimal marketing strategies and store operations based on users' interests and emotions. In addition, there was little incentive for users to actively participate in providing data.

[0802] The identification processing by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a generation AI means for analyzing and patterning collected visual information and audio information, an emotion engine for recognizing a user's emotions in real time based on the user's visual information and audio information, and a data providing means for providing analysis results, including emotion data obtained by the emotion engine, to companies. This enables highly accurate data collection and analysis based on user behavior and emotions, enabling companies to implement more effective marketing strategies and store operations, and providing appropriate incentives for users to participate in data provision.

[0803] An "information processing device" is a device that collects visual and audio information in real time and exchanges data with the outside world through communication means.

[0804] The "communication means" is a means for transmitting data collected from an information processing device to a cloud server, and refers mainly to a device or system that communicates via the Internet.

[0805] "Visual information" refers to video data about the environment and objects that the user sees.

[0806] "Audio information" refers to audio data related to the user's speech and surrounding environmental sounds.

[0807] A "server" is a computer system that receives collected visual and audio information and performs analysis and data management.

[0808] "Generative AI methods" refers to artificial intelligence techniques that use deep learning and natural language processing to analyze and pattern collected visual and audio information.

[0809] An "emotion engine" is a technology or system that recognizes emotions in real time from a user's eye movements, facial expressions, and tone of voice.

[0810] "Data provision means" refers to the means for providing analyzed data to companies, and mainly refers to systems that distribute data via APIs or databases.

[0811] "Reward management means" refers to a means for providing rewards to users in exchange for providing data, and specifically refers to point systems and digital coupons.

[0812] The present invention is a system that uses a terminal equipped with an information processing device and communication means worn by the user, a server, a generating AI means, a data providing means, a reward management means, and an emotion engine, and is useful for optimizing corporate marketing and store operations.

[0813] First, the user wears an information processing device such as smart glasses and launches a dedicated application on their smartphone. The smart glasses collect visual and audio information in real time and send the data to a cloud server via the smartphone. The visual information includes the user's eye movements and information about products they are looking at, while the audio information includes the user's comments and environmental sounds.

[0814] The collected data is first collected on the smartphone and then sent via the internet to a cloud server. The server receives the collected visual and audio information and analyzes the data using a generative AI means and emotion engine. The generative AI means uses deep learning and natural language processing to extract user behavior and emotional patterns, while the emotion engine recognizes the user's emotions in real time from eye movements, facial expressions, and tone of voice.

[0815] The analysis results are stored in a database and provided to companies through data provision means. Companies can use this data to optimize product display placement, develop promotional strategies, and position products to attract consumer interest.

[0816] Furthermore, the reward management means has a function of awarding points to users in exchange for providing data. Users can use these points for their next purchase or for other services, which increases users' motivation to participate and enables the acquisition of higher quality data.

[0817] As a concrete example, consider a scenario in which a user wears smart glasses in a store and walks around the grocery section. While enjoying their usual shopping, the user may pause at a particular shelf for a long time or comment aloud about a particular product. At this time, the emotion engine analyzes the user's visual and audio information and recognizes the user's emotions (interest, surprise, expectation, etc.). This data is sent in real time to a cloud server, where a generative AI tool analyzes the user's behavior and emotions and provides insights useful for store management.

[0818] A specific example of an input prompt sentence using a generative AI model is as follows:

[0819] Example prompt sentence:

[0820] Use user behavioral and emotional data to perform a detailed analysis of the products and emotions that users are interested in within the grocery store. Specifically, report on which shelves users spend the most time looking at and which products they show interest or anticipation for. Provide detailed insights by taking into account users' eye movements, facial expressions, and tone of voice.

[0821] In this way, the present invention enhances corporate marketing strategies and store operations, while at the same time encouraging improvements in the quality and quantity of data collection by providing rewards to users.

[0822] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0823] Step 1:

[0824] The user puts on the smart glasses and launches a dedicated application on their smartphone. At this point, the smart glasses begin collecting visual and audio information. Inputs include the user's eye movements, speech, and ambient sounds, and the output is data stored in the smart glasses. Specifically, the smart glasses capture video and audio using a camera and microphone and transfer this to the smartphone.

[0825] Step 2:

[0826] The device (smartphone) receives visual and audio information from the smart glasses and sends it to a cloud server. The input is the video and audio data transferred from the smart glasses, and the output is that this data is uploaded to the server via the internet. Specifically, the smartphone connects to the internet and sends the collected data to the cloud server in real time.

[0827] Step 3:

[0828] The server analyzes the visual and audio information it receives using a generation AI means and emotion engine. Inputs include video data, audio data, and the user's emotional state uploaded to the cloud server. Outputs include analyzing the user's behavioral patterns and emotional data, and generating patterned data. Specifically, the generation AI means analyzes the video and audio using deep learning, and the emotion engine identifies emotions from eye movements and facial expressions.

[0829] Step 4:

[0830] The server provides the generated analytical data to the company through the data provision means. The input includes data patterned by the generative AI means and the emotion engine, and the output includes insight data received by the company. Specifically, the analytical data is stored in a database, and access rights are granted to the company via an API.

[0831] Step 5:

[0832] The reward management means awards rewards to users. The input is the user's record of providing behavioral data and emotional data, and the output is points awarded to the user. Specifically, points are automatically calculated according to the amount of data provided by the user and added to the user's account.

[0833] This series of processes will enable the realization of a system that collects and analyzes user behavior and emotion data, provides useful insights to companies, and awards appropriate rewards to users. The use of a generative AI model and emotion engine enables highly accurate analysis, which is expected to be extremely useful in optimizing corporate marketing strategies and store operations.

[0834] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0835] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0836] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[0837] [Fourth embodiment]

[0838] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[0839] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0840] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0841] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[0842] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0843] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0844] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0845] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[0846] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0847] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0848] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0849] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0850] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[0851] The present invention is a system that uses a terminal equipped with an information processing device and communication means worn by the user, a server, a generating AI means, a data providing means, and a reward management means, and collects, analyzes, and provides highly accurate data that is useful for optimizing corporate marketing and store operations. Specific examples are shown below.

[0852] Data collection by terminal

[0853] The user wears the information processing device and launches a dedicated app. The information processing device takes the form of smart glasses, and collects visual and audio information through communication means and sends it to a cloud server via a smartphone.

[0854] When a user enters a store, the smart glasses automatically begin recording. The user's visual information (e.g., their behavior as they look at the shelves) and audio information (e.g., their comments about products) are collected in real time. This information is then collected on the smartphone and sent to a cloud server via the Internet.

[0855] Data analysis by server

[0856] The server analyzes the received visual and audio information using a generation AI means. The generation AI means uses advanced algorithms such as deep learning and natural language processing (NLP) to analyze eye movement patterns and speech content, creating patterns. These patterns make it possible to extract the user's behavioral tendencies and interests.

[0857] Providing data to companies

[0858] The analyzed data is provided to companies through data provision means. Based on this data, companies can optimize product display locations, develop promotional strategies, and arrange products to attract consumer interest, thereby improving their marketing strategies and store operations.

[0859] Rewarding

[0860] In addition, users are rewarded through a reward management means. Specifically, points are awarded to users based on the collected and provided data. These points can be used for the next purchase or use of other services. This increases users' motivation to participate and enables the acquisition of more abundant data.

[0861] Specific examples

[0862] For example, consider a scenario in which a user wears smart glasses in a store and walks around the grocery section. While enjoying their usual shopping routine, the user may pause at a particular shelf for a long time or comment aloud about a particular product. This data is collected in real time and sent to a server via smartphone. On the server, a generative AI tool analyzes the user's behavior and provides companies with insights such as which products are likely to attract attention and which areas are often overlooked. At the same time, the user earns points in exchange for providing the data, which can be used on their next purchase.

[0863] The present invention aims to improve the quality and quantity of data collection by not only utilizing the data collected in this way for corporate marketing strategies and store operations, but also by providing direct rewards to users.

[0864] The processing flow will be explained below.

[0865] Step 1:

[0866] The user wears the smart glasses and launches a dedicated app on their smartphone, which connects to the smart glasses via Bluetooth or Wi-Fi.

[0867] Step 2:

[0868] When a user enters a store, a dedicated app automatically activates the smart glasses' camera and microphone to begin collecting visual and audio information.

[0869] Step 3:

[0870] The device (smart glasses) collects the user's visual information (eye movements and products they are looking at) and audio information (user comments and ambient sounds) in real time, and the collected data is sent sequentially to a smartphone.

[0871] Step 4:

[0872] The user purchases an item at the cash register and pays using a cashless payment method such as PayPay. The smartphone app associates the purchase data obtained through the payment with visual and audio information.

[0873] Step 5:

[0874] When a user stops recording using the app, the devices (smart glasses and smartphone) upload visual, audio, and payment information to a cloud server.

[0875] Step 6:

[0876] The server receives the uploaded data and analyzes the visual and audio information using a generative AI, which uses a deep learning algorithm to generate patterns from the data and extract user behavioral trends and interests.

[0877] Step 7:

[0878] The server stores the analysis results in a database, organizes the stored data in a format that companies can access, and provides it to them through data provision means.

[0879] Step 8:

[0880] Based on the data provided, companies can optimize product display locations and develop promotional strategies.

[0881] Step 9:

[0882] In return for providing the data, the server uses a reward management means to give the user points, which the user can use for their next purchase.

[0883] In this way, this system efficiently collects visual and audio information from users, analyzes and patterns it using generative AI, and provides companies with valuable marketing data.In return, users are awarded points in return for providing the data, creating a win-win situation.

[0884] Example 1

[0885] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[0886] Conventional marketing data collection systems have difficulty analyzing user behavior and interests with high accuracy, limiting the quality and quantity of collected data. Furthermore, incentives for users are insufficient, making it difficult to obtain proactive data provision. This makes it difficult for companies to optimize their marketing strategies and store operations.

[0887] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0888] In this invention, the server includes a terminal equipped with an information processing device and communication means worn by the user, a means for receiving visual and audio information collected by the terminal, a data analysis means for analyzing and patterning the visual and audio information collected by the server, an information provision means for providing companies with the analysis results obtained by the data analysis means, a reward management means for rewarding users, a communication means for aggregating data collected from the terminal on a smartphone and transmitting it to a cloud server, a processing means for analyzing the visual and audio data using a generative AI model in the cloud, and a patterning means for extracting user behavioral trends and interests based on the collected data. This enables highly accurate analysis of user behavior and interests and the collection of high-quality data. Furthermore, rewarding users encourages proactive data provision.

[0889] An "information processing device" is a wearable device worn by a user that collects visual and audio information.

[0890] The "communication means" is a means having a function of transferring data from an information processing device to a smartphone and transmitting data from the smartphone to a cloud server.

[0891] A "server" is a computer system that operates on the cloud and has the ability to receive and analyze collected visual and audio information.

[0892] "Data analysis means" refers to algorithms or software that analyzes collected visual and audio information and patterns user behavior and interests.

[0893] "Information provision means" refers to the means for providing analyzed data to companies.

[0894] The "reward management means" is a means for giving rewards to users for providing data, and specifically has a function of giving points.

[0895] "Processing means" refers to a means that has the function of analyzing visual and audio information using a generative AI model on the cloud.

[0896] "Patterning means" refers to a means for extracting and patterning a user's behavioral tendencies and interests.

[0897] A "terminal" is a device that is worn by a user and has an information processing device and communication means.

[0898] A "generative AI model" is an artificial intelligence model that uses deep learning and natural language processing to analyze data and extract user behavior patterns.

[0899] This invention is a system that uses a terminal equipped with an information processing device and communication means worn by the user, a server, a generating AI means, a data providing means, and a reward management means, and collects, analyzes, and provides highly accurate data that is useful for optimizing corporate marketing and store operations.

[0900] Data collection by terminal

[0901] The user wears the information processing device and launches a dedicated app. The information processing device takes the form of smart glasses, which collect visual and audio information and send it to a cloud server via a smartphone. The smart glasses are designed to automatically start recording when the user enters a store. Visual information (e.g., the user's actions as they look at product shelves) and audio information (e.g., comments about products) are collected in real time. This information is first collected on the smartphone and then sent to the cloud server via the Internet.

[0902] Data analysis by server

[0903] The server analyzes the received visual and audio information using a generative AI means. The generative AI means uses advanced algorithms such as deep learning and natural language processing (NLP) to generate patterns of eye movement and speech content. This analysis makes it possible to extract the user's behavioral tendencies and interests.

[0904] Providing data to companies

[0905] The analyzed data is provided to companies through data provision means. Based on this data, companies can optimize product display locations, develop promotional strategies, and position products to attract consumer interest. For example, by positioning specific products in locations that are likely to attract attention, companies can improve their marketing strategies and store operations.

[0906] Rewarding

[0907] Users are rewarded through a reward management system. Specifically, based on the data collected and provided, users are awarded points. These points can be used for their next purchase or for using other services, which increases user participation and allows for the acquisition of more data.

[0908] Specific examples

[0909] For example, consider a scenario in which a user wears smart glasses in a store and walks around the grocery section. While shopping as usual, the user may pause at a particular shelf for a long time or comment aloud about a particular product. This data is collected in real time and sent to a server via smartphone. On the server, a generative AI tool analyzes the user's behavior and provides companies with insights such as which products are likely to attract attention and which areas are often overlooked. At the same time, the user earns points in exchange for providing the data, which can be used on their next purchase.

[0910] Prompt Sentence Examples

[0911] "Users gazed at a specific product shelf for about 10 seconds, then commented out loud on the name of the product. The system analyzed their eye movements and the content of their comments, and provided companies with information on which products were most likely to attract attention."

[0912] This invention aims to improve the quality of collected data by analyzing user behavior and interests with high accuracy, and also to increase motivation for providing data by providing rewards to users.

[0913] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0914] Step 1:

[0915] The user wears the information processing device and launches a dedicated app on their smartphone. The device (smart glasses) synchronizes with the launch of the app and begins collecting data. At this time, the device collects the user's visual information (which shelf they are looking at) and audio information (what they are saying) and converts it into digital format. The input is the visual and audio data captured by the smart glasses, and the output is the visual and audio data converted into digital format.

[0916] Specific examples of behavior:

[0917] When a user enters a store, the smart glasses automatically start recording.

[0918] When the app is launched, the device captures the user's eye movements and voice in real time.

[0919] Step 2:

[0920] The device transfers the collected visual and audio information to a smartphone, which temporarily stores the data and sends it to a cloud server via the Internet. The input is the digital data sent from the device, and the output is the data sent to the cloud server.

[0921] Specific examples of behavior:

[0922] As the user moves around the store, the smart glasses continuously transmit the information they collect to their smartphone.

[0923] The smartphone sends the data to a cloud server at regular intervals.

[0924] Step 3:

[0925] The server runs on the cloud and receives visual and audio information sent from the smartphone. This data is then analyzed by the generative AI model. The input is the visual and audio data sent to the cloud server, and the output is the analyzed behavioral patterns and interest data. The generative AI model performs advanced analysis using deep learning and natural language processing.

[0926] Specific examples of behavior:

[0927] The server analyzes the received visual data and determines which product shelf the user looked at and for how long.

[0928] The voice data is analyzed and information about products that have piqued the user's interest is extracted from the content of the user's comments.

[0929] Step 4:

[0930] The data analyzed on the server is provided to companies through data provision means. Companies use this data to optimize product display locations and promotion strategies. The input is data on behavioral patterns and interests as a result of the analysis, and the output is insight data provided to companies.

[0931] Specific examples of behavior:

[0932] The server generates reports and automatically sends them to the company's marketing system.

[0933] Based on the reports they receive, companies can take action such as changing the location of product displays.

[0934] Step 5:

[0935] The server provides rewards to users through the reward management means. Specifically, points are given to users for the collected and provided data. The input is the behavioral data as the analysis result, and the output is the user's reward point data.

[0936] Specific examples of behavior:

[0937] Points are automatically credited to the user's account based on the data provided by the user.

[0938] A notification regarding the points being awarded will be sent to the user's smartphone.

[0939] In this way, through a series of processing steps, a system is completed that analyzes user behavior and interests with high accuracy, collects high-quality data, and provides it to companies. Users are rewarded, so they are expected to actively provide data.

[0940] (Application example 1)

[0941] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[0942] With conventional methods of collecting marketing data, it was difficult to accurately grasp users' actual purchasing behavior and interests. Furthermore, the analysis of collected data was delayed, making it difficult to change strategies in real time. Furthermore, there was insufficient incentive for users to voluntarily participate in providing data.

[0943] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0944] In this invention, the server includes a means for transmitting visual information to a cloud storage service in real time, a means for analyzing the visual and audio information using a generative AI model, and a means for providing the analysis results to a company's API endpoint. This allows for the collection and analysis of users' visual and audio information in real time and the immediate provision of the analysis results to the company. Furthermore, by awarding users points that can be used toward their next purchase, incentives for data provision are increased, enabling the collection of higher quality data.

[0945] An "information processing device" is a device worn by a user that collects visual and audio information and transmits the data in real time.

[0946] "Communication means" refers to means for transmitting and receiving data between an information processing device and a server, and includes communication via the Internet or cloud storage.

[0947] "Visual information" refers to video data that a user perceives visually, and is information collected using a photographic device such as a camera.

[0948] "Audio information" refers to audio data including the user's speech and surrounding sounds, and is information collected using a microphone or the like.

[0949] "Server" refers to a computer system for receiving and analyzing collected visual and audio information.

[0950] "Generative AI methods" refer to algorithms and systems that use deep learning and natural language processing to analyze collected data and pattern user behavioral trends.

[0951] "Data provision means" refers to the means for providing the analysis results of the generation AI means to the company, and sends the data to the company's API endpoint via the Internet.

[0952] "Reward management means" refers to a system that provides rewards to users in exchange for providing data, such as a mechanism for awarding points.

[0953] A "cloud storage service" is a storage service that enables the storage and retrieval of data via the Internet, and is used to store video data and analysis results.

[0954] "Generative AI model" refers to an advanced machine learning model used to analyze visual and audio information, identify user behavior patterns, and analyze areas of interest, etc.

[0955] "API endpoint" refers to the interface provided by a company to receive data and is used to automatically transmit analysis results.

[0956] This invention is a system that uses a terminal equipped with an information processing device and communication means worn by the user, a server, a generating AI means, a data providing means, and a reward management means. This system collects and analyzes the visual and audio information of the user and provides it to companies to support the optimization of marketing strategies and store operations.

[0957] First, the device includes an information processing device (smart glasses) worn by the user and a communication means (smartphone). The smart glasses are equipped with a camera and microphone to collect visual and audio information from the user. The communication means is responsible for sending the collected information to a cloud server via the Internet.

[0958] The server functions as a cloud server and receives the collected visual and audio information. The server then analyzes the data using a generative AI model that utilizes deep learning and natural language processing to identify patterns of user behavior.

[0959] The data analyzed by the Generative AI Method is provided to the company via the Data Provision Method. Specifically, the analysis results are automatically sent to the API endpoint provided by the company. This allows companies to understand consumer interests in real time and use this information to improve their product displays and promotional strategies.

[0960] Meanwhile, the reward management means acts as an incentive for users. Points are awarded based on the data provided by the user, and these points can be used for future purchases, etc. This increases user participation and allows for the collection of more data.

[0961] As a concrete example, consider a scenario where a user visits the fruit section of a supermarket. While wearing smart glasses, the user looks at a particular fruit for a long time and comments aloud about the fruit's quality. This information is sent to a cloud server in real time and analyzed by a generative AI model, which provides the company with analytical results such as "bananas are attracting the most attention" or "many consumers are commenting on the price of apples." Based on these analytical results, the company can make more effective product placement and pricing decisions.

[0962] An example prompt is:

[0963] "Based on scenarios where users walk through a store and their eyes are drawn to specific products, create prompts that analyze which products and areas are most relevant. Also, analyze comments users make about specific products to gauge their level of interest."

[0964] As described above, this invention solves conventional problems and realizes optimization of marketing and store operations by using an information processing device, communication means, server, generation AI means, data provision means, and reward management means.

[0965] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0966] Step 1:

[0967] A user puts on smart glasses and launches the application. The smart glasses use a camera and microphone to collect visual information (video data) and audio information (audio data). The input data is the user's view of the shelves and store interior, as well as the user's conversations and comments. The output is raw video and audio data.

[0968] Step 2:

[0969] The device (smartphone) receives the visual and audio information collected from the smart glasses and temporarily stores it in storage. The input is the video and audio data sent from the smart glasses, and the output is the temporary data stored on the smartphone.

[0970] Step 3:

[0971] The smartphone transmits the collected visual and audio information via the Internet to a cloud server, where the data is appropriately compressed and uploaded to a cloud storage service. The input is the video and audio data stored on the smartphone, and the output is the data stored in the cloud storage service.

[0972] Step 4:

[0973] The server receives visual and audio information from cloud storage and analyzes it using a generative AI model. Specifically, it analyzes eye movement patterns using deep learning and audio information using natural language processing (NLP). The input is the video and audio data stored in cloud storage, and the output is the analysis results (e.g., eye movement patterns, content of user comments).

[0974] Step 5:

[0975] The server generates a prompt based on the analysis results and provides the analysis results to the company's API endpoint. The input is the analyzed data and the prompt from the generative AI model, and the output is the analysis results sent to the company. Specifically, the server invokes the generative AI model and sends the analysis results to the company's API as a POST request.

[0976] Step 6:

[0977] The server calculates the reward for the user's data provision and assigns points to the user's account through the reward management means. The input is the analysis result and the amount of data provided, and the output is the assigned point information. Specifically, the server accesses the reward management system and updates the points in the user's account.

[0978] Step 7:

[0979] The user uses the points they have earned on their next purchase. The input is the points information earned on the user's account, and the output is the purchase history using the points. Specifically, the user can use the points to receive a discount or exchange them for a specific product at a store.

[0980] Through this series of steps, highly accurate data is collected while users enjoy shopping in stores, and the analysis results are provided to companies, allowing users to receive rewards at the same time.

[0981] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0982] The present invention is a system that uses a terminal equipped with an information processing device and communication means worn by the user, a server, a generating AI means, a data providing means, a reward management means, and an emotion engine to collect, analyze, and provide highly accurate data that is useful for optimizing corporate marketing and store operations. Specific examples are shown below.

[0983] Data collection by terminal

[0984] The user wears the information processing device and launches a dedicated app on their smartphone. The information processing device takes the form of smart glasses, collects visual and audio information, and sends the data to a cloud server via the smartphone using a communication means.

[0985] When a user enters a store, the smart glasses automatically start recording and begin collecting visual and audio information. The user's visual information (eye movements and products they are looking at) and audio information (user comments and ambient sounds) are collected in real time. This visual and audio data is then collected on the smartphone and sent to a cloud server via the Internet.

[0986] Introducing the Emotion Engine

[0987] The emotion engine recognizes the user's emotions based on visual and audio information collected in real time. The emotion engine analyzes eye movements, facial expressions, and tone of voice to identify the user's emotional state. This emotional data, along with the visual and audio information, is sent to a cloud server and incorporated into the analysis by the generative AI method.

[0988] Data analysis by server

[0989] The server receives the uploaded visual, audio, and emotional data and analyzes it using a generative AI method. The generative AI method uses deep learning and natural language processing (NLP) to analyze the data and identify patterns in the user's behavioral tendencies, interests, and newly acquired emotional data. The results of this analysis are stored in a database.

[0990] Providing data to companies

[0991] The analyzed data is provided to companies through data provision means. Based on this data, companies can optimize product display placement, develop promotional strategies, position products to attract consumers' attention, and even implement marketing tactics that respond to user emotions. This will improve the accuracy of marketing strategies and store operations.

[0992] Rewarding

[0993] In addition, users are given rewards through a reward management means. Specifically, users are given points in exchange for providing visual information, audio information, and emotional data. Users can use these points for their next purchase or for other services. This increases users' motivation to participate and enables the acquisition of more abundant data.

[0994] Specific examples

[0995] For example, consider a scenario in which a user wears smart glasses in a store and walks around the grocery section. While shopping as usual, the user may pause at a particular shelf for a long time or comment aloud about a particular product. At this time, the emotion engine analyzes the user's visual and audio information to recognize the user's emotions (e.g., interest, surprise, anticipation, etc.). This data is sent in real time via smartphone to a cloud server. On the server, a generative AI tool analyzes the user's behavior and emotions, providing companies with insights such as which products are likely to attract attention and which locations attract emotional interest. At the same time, the user is awarded points in exchange for providing the data, which can be used on their next purchase.

[0996] The present invention not only utilizes the collected data in this way to improve corporate marketing strategies and store operations, but also promotes improvements in the quality and quantity of data collection by offering rewards to users. The introduction of an emotion engine makes it possible to provide even more advanced analysis and insights.

[0997] The processing flow will be explained below.

[0998] Step 1:

[0999] The user wears the smart glasses and launches a dedicated app on their smartphone, which connects to the smart glasses via Bluetooth or Wi-Fi.

[1000] Step 2:

[1001] When a user enters a store, the smart glasses' camera and microphone automatically activate and begin collecting visual and audio information.

[1002] Step 3:

[1003] The device (smart glasses) collects the user's visual information (eye movements and images of products they are looking at) and audio information (user comments and surrounding environmental sounds) in real time, and the collected data is sent sequentially to a smartphone.

[1004] Step 4:

[1005] When a user pauses in front of a particular product for a long time or reads a product description, the emotion engine processes this information to recognize the user's emotions (interest, surprise, anticipation, etc.) The emotion engine analyzes visual information and changes in voice tone.

[1006] Step 5:

[1007] A user purchases an item at a cash register and uses a cashless payment method (e.g., a smartphone payment app), which links the payment information with visual and audio information.

[1008] Step 6:

[1009] When the user stops recording using the app, the devices (smart glasses and smartphone) upload the collected visual information, audio information, emotional data, and payment information to a cloud server.

[1010] Step 7:

[1011] The server receives the uploaded data and analyzes and patterns the visual, audio, and emotional data using a generative AI. The generative AI uses deep learning and natural language processing (NLP) algorithms to extract the user's behavioral tendencies, interests, and emotions.

[1012] Step 8:

[1013] The server stores the analysis results in a database, organizes the stored data in a format that companies can access, and provides it to them through data provision means.

[1014] Step 9:

[1015] Companies can use the data provided to optimize product displays and develop promotional strategies. In particular, using emotion data can enable more effective marketing measures, such as product sorting and advertising.

[1016] Step 10:

[1017] In return for providing the data, the server uses a reward management means to give the user points, which the user can use for their next purchase or other services.

[1018] These steps enable efficient collection and analysis of users' visual, audio, and emotional data, providing businesses with highly accurate marketing data. Users are rewarded, which strengthens their motivation to provide data and ensures the smooth functioning of the entire system.

[1019] Example 2

[1020] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1021] In modern marketing and store operations, it is important to collect consumer behavioral and emotional data with high accuracy in real time and analyze it efficiently. However, this requires a system that can perform advanced data analysis without requiring users to spend time or money. Conventional systems have difficulty analyzing emotions in real time or providing highly accurate data, which has hindered companies from quickly optimizing their marketing strategies.

[1022] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1023] In this invention, the server includes a terminal equipped with an information processing device and communication means worn by the user, means for receiving visual information and audio information collected by the terminal, generation AI means in the server for analyzing and patterning the collected visual information and audio information, an emotion engine for analyzing the visual information and audio information in real time and recognizing the user's emotions, data provision means for providing companies with the analysis results obtained by the generation AI means, and reward management means for awarding rewards to users. This enables high-precision collection and analysis of consumer behavior data and emotion data in real time, allowing companies to quickly and effectively optimize their marketing strategies and store operations.

[1024] An "information processing device" is a device worn by a user that collects visual and audio information.

[1025] The "communication means" is a means for transmitting data collected from an information processing device to a smartphone or other device.

[1026] "Terminal" refers to a device equipped with an information processing device and a communication means, which collects and transmits visual and audio information of a user.

[1027] "Visual information" refers to information that users perceive visually, such as the movement of their eyes and the products they notice.

[1028] "Audio information" refers to data related to audio, such as the user's speech and surrounding environmental sounds.

[1029] A "server" is a computer system that receives, analyzes, and stores visual and audio information sent from a terminal.

[1030] "Generative AI means" are means of analyzing collected data and creating patterns using technologies such as deep learning and natural language processing.

[1031] The "emotion engine" is an engine that analyzes visual and audio information collected in real time and recognizes the user's emotions.

[1032] "Data provision means" refers to the means for providing companies with the analysis results obtained by the generation AI means.

[1033] The "reward management means" is a means for awarding rewards to users, and specifically includes a method for awarding points.

[1034] "Points" are units of reward that users receive in exchange for providing data, and can be used for their next purchase or other services.

[1035] This invention is a system that uses a terminal equipped with an information processing device and communication means worn by the user, a server, a generating AI means, a data providing means, a reward management means, and an emotion engine to collect, analyze, and provide highly accurate data that is useful for optimizing corporate marketing and store operations. This system can collect and analyze user behavior and emotion data in real time.

[1036] First, the user puts on a smart-glasses-shaped information processing device. This device is designed to collect visual and audio information. Before entering the store, the user launches a dedicated app on their smartphone. At this time, the device collects visual information (e.g., the user's eye movements and products they are looking at) and audio information (e.g., the user's comments and ambient sounds) in real time and sends it to a cloud server via the smartphone. This information is then uploaded to the server via the Internet.

[1037] The server receives and analyzes the uploaded visual, audio, and emotional data. This analysis uses techniques such as deep learning and natural language processing (NLP). The generative AI means analyzes this data and creates patterns based on the user's behavioral tendencies, interests, and newly acquired emotional data. The emotion engine analyzes the visual and audio information in real time to identify the user's emotional state (e.g., interest, surprise, anticipation, etc.). This enables highly accurate emotion recognition.

[1038] The analyzed data is provided to companies through data provision means. This data includes users' behavioral trends, interests, and emotional states, and companies can use this information to optimize their product placement and promotion strategies, thereby improving the accuracy of their marketing strategies and store operations.

[1039] Furthermore, users are given rewards through a reward management means. Specifically, points are awarded in exchange for providing visual information, audio information, and emotional data. Users can use these points for their next purchase or for other services. This increases users' motivation to participate and enables the acquisition of more abundant data.

[1040] Specific examples

[1041] For example, consider a scenario in which a user wears smart glasses in a store and walks around the grocery section. While shopping, the user may pause for a long time on a particular shelf or comment aloud about a certain product. At this time, the emotion engine analyzes the user's visual and audio information to recognize the user's emotions (e.g., interest, surprise, anticipation, etc.). This data is sent in real time to a cloud server via smartphone. On the server, a generative AI tool analyzes the user's behavior and emotions, providing companies with insights such as which products are likely to attract attention and which locations attract emotional interest. At the same time, the user is awarded points in exchange for providing the data, which can be used for their next purchase.

[1042] Example prompts for generative AI models

[1043] Users wear smart glasses when visiting a store and observe product shelves. Based on gaze data and voice data, you need to analyze user sentiment and provide insights that will help optimize retail strategies. Please present how the data should be analyzed, including specific methods and expected results.

[1044] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1045] Step 1:

[1046] The user wears the smart glasses and launches a dedicated app on their smartphone. The smart glasses collect the user's visual information (e.g., eye movements and products they are looking at) and audio information (e.g., what the user says and ambient sounds). The collected visual and audio information is sent to the smartphone via Bluetooth or Wi-Fi.

[1047] Input: User's visual and audio information

[1048] Output: Visual and audio information is sent to a smartphone

[1049] Step 2:

[1050] The smartphone uploads the received visual and audio information to a cloud server via the Internet. The smartphone acts as a communication tool, transmitting data to the server in real time. The smartphone may also format and convert the data.

[1051] Input: Visual and audio information received by the smartphone

[1052] Output: Formatted visual and audio information is sent to a cloud server

[1053] Step 3:

[1054] The server receives visual and audio information uploaded to the cloud. The server first passes this data to the emotion engine, which analyzes the visual and audio information in real time to recognize the user's emotions. The emotion engine analyzes eye movements, facial expressions, and vocal tone to identify the user's emotional state.

[1055] Input: Visual and audio information sent to the cloud server

[1056] Output: User sentiment data

[1057] Step 4:

[1058] The emotional data, visual information, and audio information are returned to the server and analyzed by the Generative AI Means. The Generative AI Means uses deep learning and natural language processing (NLP) to pattern the user's behavior and emotional data. Through this analysis, the Generative AI Means extracts the user's behavioral tendencies and interests.

[1059] Input: User's emotional data, visual information, and audio information

[1060] Output: User behavioral trends and emotional patterns

[1061] Step 5:

[1062] The analyzed data is stored in a database on a server and provided to companies through data provision means. Companies use the data to develop marketing strategies and store operation improvement measures, thereby optimizing product display placement and improving the accuracy of promotion strategies.

[1063] Input: Analysis results obtained by generative AI means

[1064] Output: Data provided to the company

[1065] Step 6:

[1066] Users are rewarded through a reward management system. Points are awarded based on the provision of visual, audio, and emotional data. These points can be used for future purchases or other services. This motivates users to participate in the system and collect more data.

[1067] Input: User-provided visual, audio, and emotional data

[1068] Output: Points awarded to the user

[1069] The above steps will result in a system that collects and analyzes highly accurate consumer data in real time, optimizing corporate marketing and store operations.

[1070] (Application example 2)

[1071] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1072] In conventional marketing and store operations, there was a lack of means to accurately collect and analyze user behavior and emotional data. As a result, it was difficult for companies to develop optimal marketing strategies and store operations based on users' interests and emotions. In addition, there was little incentive for users to actively participate in providing data.

[1073] The identification processing by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a generation AI means for analyzing and patterning collected visual information and audio information, an emotion engine for recognizing a user's emotions in real time based on the user's visual information and audio information, and a data providing means for providing analysis results, including emotion data obtained by the emotion engine, to companies. This enables highly accurate data collection and analysis based on user behavior and emotions, enabling companies to implement more effective marketing strategies and store operations, and providing appropriate incentives for users to participate in data provision.

[1074] An "information processing device" is a device that collects visual and audio information in real time and exchanges data with the outside world through communication means.

[1075] The "communication means" is a means for transmitting data collected from an information processing device to a cloud server, and refers mainly to a device or system that communicates via the Internet.

[1076] "Visual information" refers to video data about the environment and objects that the user sees.

[1077] "Audio information" refers to audio data related to the user's speech and surrounding environmental sounds.

[1078] A "server" is a computer system that receives collected visual and audio information and performs analysis and data management.

[1079] "Generative AI methods" refers to artificial intelligence techniques that use deep learning and natural language processing to analyze and pattern collected visual and audio information.

[1080] An "emotion engine" is a technology or system that recognizes emotions in real time from a user's eye movements, facial expressions, and tone of voice.

[1081] "Data provision means" refers to the means for providing analyzed data to companies, and mainly refers to systems that distribute data via APIs or databases.

[1082] "Reward management means" refers to a means for providing rewards to users in exchange for providing data, and specifically refers to point systems and digital coupons.

[1083] The present invention is a system that uses a terminal equipped with an information processing device and communication means worn by the user, a server, a generating AI means, a data providing means, a reward management means, and an emotion engine, and is useful for optimizing corporate marketing and store operations.

[1084] First, the user wears an information processing device such as smart glasses and launches a dedicated application on their smartphone. The smart glasses collect visual and audio information in real time and send the data to a cloud server via the smartphone. The visual information includes the user's eye movements and information about products they are looking at, while the audio information includes the user's comments and environmental sounds.

[1085] The collected data is first collected on the smartphone and then sent via the internet to a cloud server. The server receives the collected visual and audio information and analyzes the data using a generative AI means and emotion engine. The generative AI means uses deep learning and natural language processing to extract user behavior and emotional patterns, while the emotion engine recognizes the user's emotions in real time from eye movements, facial expressions, and tone of voice.

[1086] The analysis results are stored in a database and provided to companies through data provision means. Companies can use this data to optimize product display placement, develop promotional strategies, and position products to attract consumer interest.

[1087] Furthermore, the reward management means has a function of awarding points to users in exchange for providing data. Users can use these points for their next purchase or for other services, which increases users' motivation to participate and enables the acquisition of higher quality data.

[1088] As a concrete example, consider a scenario in which a user wears smart glasses in a store and walks around the grocery section. While enjoying their usual shopping, the user may pause at a particular shelf for a long time or comment aloud about a particular product. At this time, the emotion engine analyzes the user's visual and audio information and recognizes the user's emotions (interest, surprise, expectation, etc.). This data is sent in real time to a cloud server, where a generative AI tool analyzes the user's behavior and emotions and provides insights useful for store management.

[1089] A specific example of an input prompt sentence using a generative AI model is as follows:

[1090] Example prompt sentence:

[1091] Use user behavioral and emotional data to perform a detailed analysis of the products and emotions that users are interested in within the grocery store. Specifically, report on which shelves users spend the most time looking at and which products they show interest or anticipation for. Provide detailed insights by taking into account users' eye movements, facial expressions, and tone of voice.

[1092] In this way, the present invention enhances corporate marketing strategies and store operations, while at the same time encouraging improvements in the quality and quantity of data collection by providing rewards to users.

[1093] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1094] Step 1:

[1095] The user puts on the smart glasses and launches a dedicated application on their smartphone. At this point, the smart glasses begin collecting visual and audio information. Inputs include the user's eye movements, speech, and ambient sounds, and the output is data stored in the smart glasses. Specifically, the smart glasses capture video and audio using a camera and microphone and transfer this to the smartphone.

[1096] Step 2:

[1097] The device (smartphone) receives visual and audio information from the smart glasses and sends it to a cloud server. The input is the video and audio data transferred from the smart glasses, and the output is that this data is uploaded to the server via the internet. Specifically, the smartphone connects to the internet and sends the collected data to the cloud server in real time.

[1098] Step 3:

[1099] The server analyzes the visual and audio information it receives using a generation AI means and emotion engine. Inputs include video data, audio data, and the user's emotional state uploaded to the cloud server. Outputs include analyzing the user's behavioral patterns and emotional data, and generating patterned data. Specifically, the generation AI means analyzes the video and audio using deep learning, and the emotion engine identifies emotions from eye movements and facial expressions.

[1100] Step 4:

[1101] The server provides the generated analytical data to the company through the data provision means. The input includes data patterned by the generative AI means and the emotion engine, and the output includes insight data received by the company. Specifically, the analytical data is stored in a database, and access rights are granted to the company via an API.

[1102] Step 5:

[1103] The reward management means awards rewards to users. The input is the user's record of providing behavioral data and emotional data, and the output is points awarded to the user. Specifically, points are automatically calculated according to the amount of data provided by the user and added to the user's account.

[1104] This series of processes will enable the realization of a system that collects and analyzes user behavior and emotion data, provides useful insights to companies, and awards appropriate rewards to users. The use of a generative AI model and emotion engine enables highly accurate analysis, which is expected to be extremely useful in optimizing corporate marketing strategies and store operations.

[1105] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1106] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1107] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1108] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1109] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1110] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1111] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1112] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1113] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1114] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1115] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1116] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1117] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1118] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1119] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1120] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1121] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1122] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1123] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1124] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1125] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1126] The following is further disclosed regarding the above embodiment.

[1127] (Claim 1)

[1128] a terminal equipped with an information processing device and communication means worn by a user;

[1129] a server that receives the visual and audio information collected by the terminal;

[1130] A generating AI means in the server that analyzes the collected visual information and audio information and creates patterns;

[1131] a data providing means for providing the analysis results obtained by the generating AI means to a company;

[1132] a reward management means for granting rewards to users;

[1133] A system including:

[1134] (Claim 2)

[1135] 2. The system of claim 1, wherein the terminal is a smart glass worn by a user.

[1136] (Claim 3)

[1137] 2. The system according to claim 1, wherein the reward management means is a means for awarding points based on data provided by a user.

[1138] "Example 1"

[1139] (Claim 1)

[1140] a terminal equipped with an information processing device and communication means worn by a user;

[1141] a server that receives the visual and audio information collected by the terminal;

[1142] a data analysis means in the server for analyzing and patterning the collected visual and audio information;

[1143] an information providing means for providing the analysis results obtained by the data analysis means to companies;

[1144] a reward management means for granting rewards to users;

[1145] A communication means for aggregating data collected from the terminals on a smartphone and transmitting it to a cloud server;

[1146] processing means in the cloud for analyzing the visual and audio data using a generative AI model;

[1147] A patterning method for extracting user behavioral trends and interests based on collected data;

[1148] A system including:

[1149] (Claim 2)

[1150] 2. The system of claim 1, wherein the terminal is a smart glass worn by a user.

[1151] (Claim 3)

[1152] 2. The system according to claim 1, wherein the reward management means is a means for awarding points based on data provided by a user.

[1153] "Application Example 1"

[1154] (Claim 1)

[1155] a terminal equipped with an information processing device and communication means worn by a user;

[1156] a server that receives the visual and audio information collected by the terminal;

[1157] A generating AI means in the server that analyzes the collected visual information and audio information and creates patterns;

[1158] a data providing means for providing the analysis results obtained by the generating AI means to a company;

[1159] a reward management means for granting rewards to users;

[1160] means for transmitting the visual information to a cloud storage service in real time;

[1161] a means for analyzing visual and audio information using a generative AI model;

[1162] a means for providing the analysis results to an enterprise API endpoint;

[1163] A system including:

[1164] (Claim 2)

[1165] 10. The system of claim 1, comprising smart glasses worn by a user and including means for uploading video information to a cloud storage service.

[1166] (Claim 3)

[1167] The system of claim 1, further comprising a means for granting points that can be used by the user for their next purchase, and a means for providing insights to companies based on the analysis results.

[1168] "Example 2: Combining Emotion Engines"

[1169] (Claim 1)

[1170] a terminal equipped with an information processing device and communication means worn by a user;

[1171] a server that receives the visual and audio information collected by the terminal;

[1172] A generating AI means in the server that analyzes the collected visual information and audio information and creates patterns;

[1173] An emotion engine that analyzes visual and audio information in real time to recognize the user's emotions;

[1174] a data providing means for providing the analysis results obtained by the generating AI means to a company;

[1175] a reward management means for granting rewards to users;

[1176] A system including:

[1177] (Claim 2)

[1178] 2. The system of claim 1, wherein the terminal is a smart glass worn by a user.

[1179] (Claim 3)

[1180] 2. The system according to claim 1, wherein the reward management means is a means for awarding points based on data provided by a user.

[1181] "Application example 2 when combining emotion engines"

[1182] (Claim 1)

[1183] a terminal equipped with an information processing device and communication means worn by a user;

[1184] a server that receives the visual and audio information collected by the terminal;

[1185] A generating AI means in the server that analyzes the collected visual information and audio information and creates patterns;

[1186] an emotion engine that recognizes a user's emotions in real time based on the user's visual and audio information;

[1187] a data providing means for providing companies with analysis results including emotion data obtained by the emotion engine;

[1188] a reward management means for granting rewards to users;

[1189] A system including:

[1190] (Claim 2)

[1191] 2. The system of claim 1, wherein the terminal is a smart glass worn by a user.

[1192] (Claim 3)

[1193] 2. The system according to claim 1, wherein the reward management means is a means for awarding points based on data provided by a user. [Explanation of symbols]

[1194] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. a terminal equipped with an information processing device and communication means worn by a user; a server that receives the visual and audio information collected by the terminal; In the server, a generating AI means for analyzing the collected visual information and audio information and creating a pattern; a data providing means for providing the analysis results obtained by the generating AI means to a company; a reward management means for granting rewards to users; A system including:

2. The system according to claim 1 , wherein the terminal is a smart glass worn by a user.

3. 2. The system according to claim 1, wherein the reward management means is a means for awarding points based on data provided by a user.

Images