System

Abstract

A system is provided.SOLUTION: A system comprising: a vision data storage device worn by a leading athlete or an instructor; a server that receives and stores line-of-sight data recorded by the vision data storage device; a data analysis unit that analyzes the line-of-sight data and specifies a focus point or a movement pattern of a line of sight; and an advice generation unit that provides advice to a user based on a result of the analysis.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In general, in sports and other technical fields, the perspectives and movements of consciousness of top athletes and instructors are difficult for beginners and intermediate players to understand. Conventional methods lack a mechanism for effectively collecting and analyzing this visual information and providing individualized advice. This makes it difficult to obtain feedback for effective training and improvement. The present invention aims to solve this problem and provide an effective system that allows users to learn the perspectives of top athletes and instructors and improve their skills. [Means for solving the problem]

[0005] The present invention first provides a visual data storage device worn by top athletes and coaches, which records gaze data in real time. The visual data storage device transmits the recorded gaze data to a server. The server then stores the received gaze data, and data analysis means analyzes the data. Specifically, the analysis means identifies focus points and eye movement patterns and visualizes them as a gaze map or heat map. The present invention also provides an advice generation means that provides personalized advice to users based on the analysis results. In this way, the present invention enables users to learn top-class gaze and awareness and use this knowledge to improve their skills through a system including a visual data storage device, data analysis means, and advice generation means.

[0006] A "visual data storage device" is a device worn by top athletes and coaches that records their eye movements and focus points in real time.

[0007] The "server" is a computer system that receives, stores, and manages gaze data transmitted from the visual data storage device.

[0008] The "data analysis means" is a combination of software and hardware for analyzing the received gaze data and identifying the focus point and the movement pattern of the gaze.

[0009] The "advice generating means" is a combination of software and hardware for providing personalized advice to the user based on the analysis results obtained by the data analyzing means.

[0010] "Gaze data" refers to information about eye movements and gaze positions recorded by a visual data storage device.

[0011] The "analysis results" are information such as focus points, eye movement patterns, eye gaze maps, and heat maps extracted from the gaze data processed by the data analysis means.

[0012] "Focus points" are positions or points that top athletes and coaches pay particular attention to, identified from analyzed gaze data.

[0013] "Eye movement patterns" are data relating to the path and speed of eye movement recorded by a visual data storage device.

[0014] A "gaze map" is a map-like graphical representation that visualizes gaze data recorded by a visual data storage device and shows eye movements and gaze points.

[0015] A "heat map" is a graphical representation that uses shades of color to indicate the frequency and intensity of focus points based on gaze data recorded by a visual data storage device.

[0016] "User" refers to an individual or group who uses the visual data storage device and related systems to learn the gaze data of top athletes and coaches and to improve their own skills. [Brief explanation of the drawings]

[0017] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7]FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0018] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0019] First, the terms used in the following description will be explained.

[0020] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0021] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0022] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0023] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0024] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0025] [First embodiment]

[0026] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0027] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0028] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0029] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0030] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0031] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0032] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0033] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0034] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0035] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0036] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0037] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0038] The present invention is embodied as a system including a visual data storage device worn by top athletes and coaches, data analysis means, and advice generation means. This allows users to learn the perspectives and awareness of top athletes and use this knowledge to improve their skills. Specific details of the embodiments of the present invention are described below.

[0039] Key elements of the system

[0040] 1. Visual Data Storage Device

[0041] The visual data storage device is a device worn by top athletes and coaches that records eye movements and focus points in real time. The device is equipped with a camera and sensors, and the acquired gaze data is sent to a server via wireless communication.

[0042] 2. Server

[0043] The server receives, stores, and manages the gaze data sent from the visual data storage device. The server has a database and has the function of individually storing data for each user. Furthermore, the server provides the computing resources to execute the analysis means and the advice generation means.

[0044] 3. Data Analysis Methods

[0045] The data analysis tool is software and algorithms that run on a server. It analyzes gaze data to identify focus points, eye movement patterns, etc. This data is visualized as gaze maps and heat maps and is used to generate advice later.

[0046] 4. Advice Generation Methods

[0047] The advice generator is software that provides personalized advice to users based on the analysis results. It suggests specific areas for improvement and practice methods based on the analyzed gaze data.

[0048] Program processing flow

[0049] 1. Data Collection Phase

[0050] server:

[0051] The server works with the visual data storage device to prepare for data collection. Specifically, the server registers the device ID and user information of the visual data storage device in a database and sets up data reception.

[0052] Device:

[0053] Visual data storage devices are worn by top athletes and coaches to record their eye movements and focus points in real time. The wearer's gaze data is stored on the device and periodically sent to a server.

[0054] User:

[0055] Before practice or a game, the visual data storage device is attached and checked for proper fit. After confirming that the device is attached correctly, data collection begins.

[0056] 2. Data analysis phase

[0057] server:

[0058] The server stores the received gaze data and performs pre-processing, which includes cleaning and formatting the data to convert it into a format that can be analyzed by the data analysis tool.

[0059] Data analysis methods:

[0060] Gaze data is analyzed to identify focus points and eye movement patterns, and the analysis results are visualized as gaze maps and heat maps.

[0061] User:

[0062] Check the analysis results to understand where top athletes and coaches focus their attention. Visually check specific eye movements as gaze maps and heat maps.

[0063] 3. Advice Phase

[0064] server:

[0065] Based on the results of the data analysis, the AI ​​generates personalized advice, which is then sent from the server to the device.

[0066] Device:

[0067] The received advice is displayed to the user, who can then interactively review the advice within the application and receive further information.

[0068] User:

[0069] The player then practices or plays a match based on the advice provided. The visual data storage device is then worn again to collect gaze data and check the progress of the player's improvement. This creates a feedback loop, enabling gradual improvement of skills.

[0070] Specific examples

[0071] For example, if you were to wear a visual data storage device to collect gaze data during a soccer game:

[0072] Data Collection Phase

[0073] Server: Registers the device ID of the visual data storage device and configures data reception.

[0074] Device: The glasses record eye movements in real time during the game and periodically send the data to a server.

[0075] User: Put on the glasses before the match, check the fit, and then record gaze data during the match.

[0076] Data analysis phase

[0077] Server: Cleans the received gaze data and passes it to the data analysis means.

[0078] Data analysis method: AI analyzes gaze data and displays it as a gaze map or heat map.

[0079] Users: Check the analytics results to understand what they focused on during the match.

[0080] Advice Providing Phase

[0081] Server: The AI ​​generates advice based on the analysis results and sends it to the device.

[0082] Device: Display advice within your app so users can find out more information.

[0083] User: Follow the advice and then perform your next practice or match, collecting gaze data again to see any improvements.

[0084] In this way, the present invention provides an effective system for users to study the gaze data of top athletes and coaches and improve their skills.

[0085] The processing flow will be explained below.

[0086] Step 1:

[0087] Server: The server starts the system that links with the visual data storage device and prepares for data transmission. The server registers the device ID and user information of the visual data storage device in the database and sets up data reception.

[0088] Step 2:

[0089] Device: Power on the visual data storage device, check the network connection, pair it with the corresponding mobile app, and verify that the device is working properly.

[0090] Step 3:

[0091] User: Wears the visual data storage device before training or a match. Checks the fit and secures it in place. Presses the Start Session button to begin data collection.

[0092] Step 4:

[0093] Device: The glasses record eye movements in real time and capture gaze data, which is temporarily stored in a buffer on the device and sent to the server at set intervals.

[0094] Step 5:

[0095] Server: Receives gaze data sent from the device periodically and accumulates it in a database. The data is saved with a timestamp for later analysis.

[0096] Step 6:

[0097] Server: Preprocesses the received gaze data, cleaning and denoising the data, and optionally formatting the data into an analyzable format.

[0098] Step 7:

[0099] Data analysis means (server): Analyzes gaze data using an AI model. Specifically, it analyzes gaze movement patterns, detection of attention points, reaction times, etc. The analysis results are visualized as gaze maps and heat maps.

[0100] Step 8:

[0101] Server: Analyzed data is stored on the server and prepared for transmission to the device as needed.

[0102] Step 9:

[0103] Device: Once the analysis results are sent from the server, they are presented to the user. They are displayed as gaze maps or heat maps within the application, allowing the user to visually confirm the results.

[0104] Step 10:

[0105] User: Check the analysis results through the application, specifically understanding where top coaches and athletes are looking and when they move their eyes.

[0106] Step 11:

[0107] Advice generator (server): The AI ​​model generates personalized advice based on the analysis results. This advice includes specific areas for improvement and practice.

[0108] Step 12:

[0109] Server: Sends generated advice to the device, formatted in a user-friendly format.

[0110] Step 13:

[0111] Terminal: Display the received advice to the user, interactively within the application, allowing the user to view more information if needed.

[0112] Step 14:

[0113] User: Based on the advice provided, the user will train and compete, and in the process will use the visual data storage device again to collect data for further improvement.

[0114] Step 15:

[0115] Server: Re-analyzes the data collected and evaluates the user's progress. Always provides feedback based on the latest data.

[0116] Through this series of processes, users can learn in detail about the gaze and awareness of top athletes and coaches, and use this knowledge to improve their own skills.

[0117] Example 1

[0118] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0119] In conventional sports instruction and training, there is a lack of means to collect gaze data from top athletes and coaches in real time and provide specific feedback based on that data. This makes it difficult for individual athletes to easily obtain specific advice on how to improve their skills. In addition, there is a lack of methods to visually represent and present the analysis results in an easy-to-understand manner.

[0120] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0121] In this invention, the server includes a visual data storage device worn by top athletes and coaches, an information processing device that receives and stores gaze data recorded by the visual data storage device, a preprocessing device that cleans the gaze data and passes it to data analysis means, a data analysis means that analyzes the gaze data and identifies focus points and eye movement patterns, a means for visualizing the analysis results as a gaze map or heat map, an advice generation means that provides advice to users based on the analysis results, and a display means that displays the advice on the user's terminal. This allows users to learn the perspectives and awareness of top athletes and use it to improve their own skills.

[0122] A "visual data storage device" is a device worn by top athletes and coaches to record eye movements and focus points in real time.

[0123] The "information processing device" is a device that receives, stores, and manages the gaze data transmitted from the visual data storage device.

[0124] The "pre-processing device" is a device that cleans the received gaze data, removes noise, complements missing data, and passes the data to the data analysis means.

[0125] "Data analysis means" refers to software and algorithms for analyzing gaze data and identifying focus points and eye movement patterns.

[0126] A "gaze map" is a map that visually represents gaze movements based on gaze data.

[0127] A "heat map" is a visual representation that shows gaze points using shades of color.

[0128] An "advice generator" is software and algorithms for providing personalized advice to a user based on the analysis results.

[0129] The "display means" is a device or function for displaying the generated advice and analysis results on the user's terminal.

[0130] A "generative AI model" is an artificial intelligence model that analyzes gaze data and generates gaze maps or heat maps.

[0131] The present invention is embodied as a system including a visual data storage device worn by top athletes and coaches, data analysis means, and advice generation means. This allows users to learn the perspectives and awareness of top athletes and use this knowledge to improve their skills. Specific details of the embodiments of the present invention are described below.

[0132] Key elements of the system

[0133] 1. Visual Data Storage Device

[0134] The visual data storage device is a device worn by top athletes and coaches. It is equipped with cameras and sensors that record eye movements and focus points in real time. The collected gaze data is transmitted to a server via wireless communication.

[0135] 2. Server

[0136] The server receives, stores, and manages gaze data transmitted from the visual data storage device. In particular, the server incorporates a pre-processing unit that cleans the gaze data and passes it to the data analysis unit. The server also provides the computing resources to run the data analysis unit and the advice generation unit.

[0137] 3. Data Analysis Methods

[0138] The data analysis tool is software and algorithms that run on a server. It analyzes gaze data to identify focus points and eye movement patterns. This data is visualized as gaze maps and heat maps and used to generate advice later.

[0139] 4. Advice Generation Methods

[0140] The advice generator is software that provides personalized advice to users based on the analysis results of the data analysis means. It uses a generative AI model to suggest specific areas for improvement and practice methods.

[0141] 5. Display means

[0142] The display means is a device or function for displaying the generated advice and analysis results on the user's terminal. It has an interactive display function, allowing the user to check detailed information.

[0143] Example of operation flow

[0144] For example, consider the case where a visual data storage device is worn by a player during a soccer match to collect gaze data. In this case, the server registers the device ID of the visual data storage device and configures data reception. During the match, the device's camera records the player's eye movements in real time and periodically sends this to the server. The server cleans the received gaze data and passes it to the data analysis means. This means analyzes the gaze data using a generative AI model and generates a gaze map or heat map. The server sends advice generated by the AI ​​to the device, which displays the advice within the application. The player then performs the next practice or match based on the advice provided.

[0145] Prompt Sentence Examples

[0146] "Please explain the algorithm that analyzes focus points and eye movement patterns using gaze data from athletes wearing visual data storage devices."

[0147] With this structure, the system of the present invention allows users to learn the gaze data of top athletes and coaches and use it to improve their own skills.

[0148] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0149] Step 1: Prepare for data collection

[0150] Server Action:

[0151] The server registers the device ID and user information of the visual data storage device in a database. The input is the device ID (e.g., ID12345) of the visual data storage device and the user information, and outputs a database entry for linking and managing the device ID and user information. The server then performs operations to prepare for communication with the visual data storage device.

[0152] Step 2: Start collecting data

[0153] Terminal handling:

[0154] The device is worn by top athletes and coaches. The device's built-in camera and sensors record eye movements and focus points in real time. The input is the athlete's eye movements, and the output is the recorded gaze data. The device operates to collect gaze data in real time.

[0155] Step 3: Send data

[0156] Terminal handling:

[0157] The terminal periodically collects and transmits the gaze data to the server via wireless communication. The input is the gaze data recorded on the terminal, and the output is the gaze data transmitted to the server. The terminal controls the timing of data transmission and performs operations to maintain data consistency.

[0158] Step 4: Save Data

[0159] Server Action:

[0160] The server stores the received gaze data in a database. The input is the gaze data received from the device, and the output is the gaze data stored in the database. It performs operations to verify the accuracy of the data (for example, checking data reception and verifying it before saving).

[0161] Step 5: Data Cleaning

[0162] Server Action:

[0163] The server cleans the stored gaze data, removes noise, and fills in missing data. The input is the stored gaze data, and the output is the cleaned gaze data. It applies noise filtering and data filling algorithms.

[0164] Step 6: Data analysis

[0165] Processing data analysis methods:

[0166] The data analysis means analyzes the cleaned gaze data and identifies focus points and eye movement patterns. The input is the cleaned gaze data and the output is the analysis results. The data analysis means performs operations to execute a focus point detection algorithm and a movement pattern analysis algorithm.

[0167] Step 7: Generate gaze maps and heat maps

[0168] Processing data analysis methods:

[0169] The data analysis means visualizes the analysis results as gaze maps and heat maps. The input is the analysis results, and the output is gaze maps and heat maps. A visualization algorithm is used to graphically display the gaze data.

[0170] Step 8: Advice Generation

[0171] Advice generator processing:

[0172] The advice generation means generates personalized advice for the user based on the analysis results of the gaze map and heat map. The input is the analysis results of the gaze map and heat map, and the output is the generated advice. The system operates to generate personalized feedback using a generative AI model.

[0173] Step 9: Submitting Advice

[0174] Server Action:

[0175] The server sends the generated advice to the terminal. The input is the generated advice and the output is the advice sent to the terminal. It performs actions to ensure the accuracy of the data transmission (e.g., checking the data package and confirming transmission).

[0176] Step 10: Viewing Advice

[0177] Terminal handling:

[0178] The terminal displays the advice received within the application to the user. The input is the advice received from the server, and the output is the advice visually displayed to the user. It provides an interactive display function and performs actions that allow the user to check detailed information.

[0179] Step 11: Act on the advice

[0180] User Action:

[0181] The user then practices or plays a game based on the advice provided. The input is the advice displayed on the device, and the output is the action the user performs. The user then wears the visual data storage device again, collects gaze data during the next game or practice, and performs actions to check for improvement.

[0182] (Application example 1)

[0183] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0184] Currently, there is no real-time advice system to improve the driving skills of autonomous vehicles, so there is a need for autonomous driving systems to provide appropriate feedback on the driving situation by utilizing gaze data from professional drivers. Conventional systems do not sufficiently link the collection and analysis of driving data, making it difficult to improve technology and safety in real time.

[0185] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0186] In this invention, the server includes a means for receiving and storing gaze data, a means for analyzing the gaze data to identify focus points and eye movement patterns, and a means for providing feedback to the autonomous driving AI system based on the analysis results. This makes it possible to improve driving skills in real time using gaze data from professional drivers and provide advice according to driving conditions.

[0187] A "visual data storage device" is a device worn by the pilot to record eye movements and gaze direction.

[0188] A "server" is a central device for receiving, storing, and processing gaze data transmitted from a visual data repository.

[0189] The "data analysis means" is software or algorithm that runs on the server and analyzes the received gaze data to identify focus points and gaze movement patterns.

[0190] The "advice generation means" is software or an algorithm that provides the user with advice for improving their skills based on the analysis results and the gaze data of top pilots.

[0191] "Means for improving autonomous driving technology" refers to a means for supplying analyzed gaze data to the driving AI system of an autonomous vehicle in order to improve driving technology.

[0192] The "real-time advice providing means" is a means for providing appropriate advice for the driving situation in real time.

[0193] A "gaze map" is a map that visually displays gaze movement patterns based on gaze data from professional drivers.

[0194] A "heat map" is a map that visually displays points of focus and the degree of gaze concentration.

[0195] A "driving AI system" is an artificial intelligence system that controls the driving of autonomous vehicles and optimizes driving operations based on external data.

[0196] This invention is implemented as a system that provides a visual data storage device worn by a professional driver, a real-time data analysis means, and appropriate driving advice to improve automated driving technology. The system includes the following main elements:

[0197] Key elements of the system

[0198] 1. Visual Data Storage Device

[0199] These smart glasses are worn by professional drivers and record eye movements and gaze direction in real time, wirelessly transmitting the data to a server.

[0200] 2. Server

[0201] The server receives and stores the gaze data sent from the visual data storage device. It analyzes the data and provides the analysis results to the driving AI system. As a specific example, an Amazon Web Services (AWS) EC2 instance is used.

[0202] 3. Data Analysis Methods

[0203] This is software and algorithms that run on a server to analyze gaze data, identify focus points and gaze movement patterns, and visualize them as gaze maps and heat maps using Python and OpenCV.

[0204] 4. Advice Generation Methods

[0205] It is software or algorithm that provides feedback to autonomous driving AI systems based on analysis results, generating personalized advice using TensorFlow.

[0206] 5. Real-time advice delivery methods

[0207] It is a means of providing appropriate advice in real time on driving situations during automated driving, which will improve automated driving technology and increase driving safety.

[0208] Specific processing explanation of the system

[0209] The server receives gaze data from the visual data storage device and stores it in a database. This data first undergoes a data cleaning process and is formatted for analysis. The gaze data is then analyzed using Python and OpenCV and visualized as gaze maps and heat maps. The analysis results are input into an AI model using TensorFlow to generate personalized advice for the driving AI system.

[0210] Specific examples of processing

[0211] Once the server receives the visual data, it performs data cleaning to remove noise. It is then analyzed using OpenCV to generate a gaze map. For example, it can identify the gaze points of a professional driver when approaching an intersection and provide this to the driving AI system. Based on this data, the generative AI model can provide appropriate driving advice in real time.

[0212] Examples of prompt statements

[0213] Here are some example prompts to input to a generative AI model:

[0214] Based on the gaze data analysis results, identify the points that professional drivers focus on while driving and provide optimal advice and suggestions for improvement for safe driving.

[0215] By using this prompt, the AI ​​model can generate specific advice for improving driving skills based on the analysis results.

[0216] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0217] Step 1:

[0218] Visual data collection

[0219] The user has a professional driver wear smart glasses, which collect real-time gaze data while driving and transmit it to a server via Wi-Fi.

[0220] Input: Professional driver's gaze data

[0221] Output: Gaze data sent to the server

[0222] Step 2:

[0223] Receiving and storing data

[0224] The server receives the gaze data sent from the smart glasses and stores it securely in a database.

[0225] Input: Gaze data sent from smart glasses

[0226] Output: Gaze data stored in a database

[0227] Step 3:

[0228] Data Preprocessing

[0229] The server performs a cleaning process on the received gaze data, which involves removing noise and reformatting the data format.

[0230] Input: Gaze data stored in a database

[0231] Output: Cleaned gaze data

[0232] Step 4:

[0233] Data analysis

[0234] The server analyzes the cleaned gaze data using data analysis tools. It uses OpenCV to generate gaze maps and heat maps. This analysis identifies focus points and gaze movement patterns while driving.

[0235] Input: Cleaned gaze data

[0236] Output: gaze maps and heat maps

[0237] Step 5:

[0238] Advice Generation

[0239] Based on the analysis results, the server uses TensorFlow to generate AI model advice for the driving AI system. It uses prompt sentences to suggest appropriate driving advice and improvements.

[0240] Input: gaze maps and heat maps

[0241] Output: Advice for driving AI systems

[0242] Step 6:

[0243] Real-time advice provided

[0244] The server sends the generated advice to the driving AI system in real time, which then optimizes driving operations based on the advice received, ensuring safe driving.

[0245] Input: Advice for driving AI systems

[0246] Output: Optimized driving operation by the driving AI system

[0247] Through this series of processes, the gaze data of professional drivers is used to improve autonomous driving technology and provide appropriate advice according to the driving situation.

[0248] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0249] The present invention is embodied as a system that combines a visual data storage device worn by top athletes and coaches, data analysis means, and advice generation means with an emotion engine that recognizes the user's emotions. This allows users to learn not only the perspectives and awareness of top athletes, but also their own emotional state, which can be used to improve their skills. Specific details of the embodiments of the present invention are described below.

[0250] Key elements of the system

[0251] 1. Visual Data Storage Device

[0252] The visual data storage device is a device worn by top athletes and coaches that records eye movements and focus points in real time. The device is equipped with a camera and sensors, and the acquired gaze data is sent to a server via wireless communication.

[0253] 2. Emotion Engine

[0254] The emotion engine is a combination of software and hardware that analyzes data such as the user's facial expressions, voice tone, and heart rate to recognize emotions in real time. It is built into the visual data storage device and transmits emotion data along with gaze data to the server.

[0255] 3. Server

[0256] The server receives, stores, and manages the gaze data and emotion data sent from the visual data storage device and emotion engine. The server has a database and has the function of individually storing data for each user. Furthermore, the server provides the computational resources to execute the analysis means and advice generation means.

[0257] 4. Data Analysis Methods

[0258] The data analysis tool is software and algorithms running on a server. It analyzes gaze data and emotion data to identify focus points, eye movement patterns, and emotional states. This data is visualized as gaze maps and heat maps and used to generate advice.

[0259] 5. Advice Generation Methods

[0260] The advice generator is software that provides personalized advice to users based on the analysis results. It suggests specific areas for improvement and practice methods based on the analyzed gaze data and emotion data.

[0261] Program processing flow

[0262] 1. Data Collection Phase

[0263] server:

[0264] The server works with the visual data storage device and emotion engine to prepare for data collection. Specifically, the server registers the device IDs and user information of the visual data storage device and emotion engine in a database and sets up data reception.

[0265] Device:

[0266] Visual data storage devices and emotion engines are worn by top athletes and coaches to record their eye movements, focus points, and emotional states in real time. The wearer's gaze data and emotion data are stored on the device and periodically transmitted to a server.

[0267] User:

[0268] Before practice or a game, the visual data storage device is attached and checked for proper fit. After confirming that the device is attached correctly, data collection begins.

[0269] 2. Data analysis phase

[0270] server:

[0271] The server stores and pre-processes the received gaze and emotion data, including cleaning and denoising the data, and converts it into a format that can be analyzed by the data analysis means.

[0272] Data analysis methods:

[0273] Gaze data and emotional data are analyzed to identify focus points, eye movement patterns, and emotional states. The analysis results are visualized as gaze maps and heat maps.

[0274] User:

[0275] By reviewing the analysis results, you can understand where top athletes and coaches focused and what their emotional state was. Visually check specific eye movements and emotional states as gaze maps and heat maps.

[0276] 3. Advice Phase

[0277] server:

[0278] Based on the results of the data analysis, the AI ​​generates personalized advice, which is then sent from the server to the device.

[0279] Device:

[0280] The received advice is displayed to the user, who can then interactively review the advice within the application and receive further information.

[0281] User:

[0282] The player then practices and plays matches based on the advice provided. The visual data storage device is then worn again to collect gaze and emotional data, and the player's progress is monitored. This creates a feedback loop, enabling gradual improvement in technique and emotional control.

[0283] Specific examples

[0284] For example, if you wear a visual data storage device and an emotion engine to collect gaze data and emotion data during a soccer game:

[0285] Data Collection Phase

[0286] Server: Registers the device IDs of the visual data storage device and emotion engine, and configures data reception.

[0287] Device: The glasses record eye movements and emotional state in real time during the match and periodically transmit the data to a server.

[0288] User: The user puts on the glasses before the match, checks their fit, and then records their gaze and emotion data during the match.

[0289] Data analysis phase

[0290] Server: Cleans the received gaze data and emotion data and passes them to the data analysis means.

[0291] Data analysis method: AI analyzes gaze data and emotion data and displays them as gaze maps and heat maps.

[0292] Users: Review analytics to understand what they focused on and what their emotional state was during a match.

[0293] Advice Providing Phase

[0294] Server: The AI ​​generates advice based on the analysis results and sends it to the device.

[0295] Device: Display advice within your app so users can find out more information.

[0296] User: Based on the advice, conduct the next practice or match, and collect gaze and emotion data again to confirm improvement.

[0297] In this way, the present invention allows users to learn the gaze data and emotion data of top athletes and coaches and use it to improve their own skills and control their emotions.

[0298] The processing flow will be explained below.

[0299] Step 1:

[0300] Server: Registers the device IDs and user information of the visual data storage device and emotion engine in the database, and sets up data reception. Installs the server software on the server and prepares it for operation.

[0301] Step 2:

[0302] Device: Power on the visual data storage device and emotion engine, check the network connection, pair with the corresponding mobile app, and verify that the device is working properly. Calibrate the emotion engine so that it can properly capture data such as the user's facial expressions, tone of voice, and heart rate.

[0303] Step 3:

[0304] User: Before training or a match, wear the visual data storage device and emotion engine and check the fit. After making sure the device is fixed in the correct position, press the start button to begin data collection.

[0305] Step 4:

[0306] Device: The glasses record eye movements and the emotion engine records emotion data in real time, capturing them as gaze data and emotion data. The captured data is temporarily stored in a buffer on the device and sent to the server at set intervals.

[0307] Step 5:

[0308] Server: Receives gaze data and emotion data periodically sent from the device and stores them in a database. The data is saved with a timestamp for later analysis.

[0309] Step 6:

[0310] Server: Preprocesses the received gaze and emotion data, cleaning and denoising the data, and optionally formatting the data to make it more analyzable.

[0311] Step 7:

[0312] Data analysis means (server): Analyzes gaze data and emotional data using an AI model. Specifically, it analyzes gaze movement patterns, detection of attention points, reaction times, and emotional states. The analysis results are visualized as gaze maps and heat maps.

[0313] Step 8:

[0314] Server: The analyzed data is stored on the server and prepared for transmission to the device as needed. The analysis results are formatted in a user-friendly format.

[0315] Step 9:

[0316] Terminal: When the analysis results are sent from the server, they are presented to the user. They are displayed as gaze maps or heat maps, allowing the user to visually confirm the results.

[0317] Step 10:

[0318] Users: View analytics results through the application, specifically to understand where elite athletes and coaches focus their efforts and their emotional state.

[0319] Step 11:

[0320] Advice generation means (server): The AI ​​model generates personalized advice based on the analysis results. This advice includes specific improvements and practice methods related to both gaze data and emotion data.

[0321] Step 12:

[0322] Server: Sends generated advice to the device, formatted in a user-friendly format.

[0323] Step 13:

[0324] Terminal: Display the received advice to the user, interactively within the application, allowing the user to view more information if needed.

[0325] Step 14:

[0326] User: Based on the advice provided, the user trains and competes, again using the visual data storage device and emotion engine in the process to collect data for further improvement.

[0327] Step 15:

[0328] Server: Re-analyzes the newly collected data and evaluates the user's progress. Always provides feedback based on the latest data, and checks the effectiveness of the user's technical improvement and emotional control.

[0329] Through this series of processes, users can learn in detail about the gaze and emotional states of top athletes and coaches, and use this knowledge to improve their own skills and manage their emotions.

[0330] Example 2

[0331] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0332] While conventional sports training systems focus on analyzing the user's visual information and focus points, they lack the technology to provide comprehensive training advice that takes the user's emotional state into account. As a result, feedback on the user's technical improvement is limited, making it difficult to adequately support emotional control and mental growth. Furthermore, real-time data collection and analysis are insufficient, making it difficult for users to receive immediate advice.

[0333] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0334] In this invention, the server includes a visual data storage device worn by top athletes and coaches, an emotion engine that recognizes the user's emotions in real time, means for receiving and saving data recorded by the visual data storage device and emotion engine, means for analyzing the gaze data and emotion data received from the server and identifying focus points, eye movement patterns, and emotional states, and means for providing personalized advice to the user based on the results of the analysis. This enables the user to receive comprehensive advice that takes into account not only visual information but also emotional states, thereby enhancing both technical improvement and emotional control.

[0335] A "visual data storage device" is a device worn by top athletes and coaches that records eye movements and focus points in real time.

[0336] An "emotion engine" is a combination of software and hardware that analyzes data such as a user's facial expressions, voice tone, and heart rate to recognize emotions in real time.

[0337] "Server" refers to a system that receives, stores, and manages data sent from the visual data storage device and emotion engine, and provides the computational resources necessary for analyzing the data and generating advice.

[0338] "Data analysis means" refers to a means for analyzing gaze data and emotion data by software and algorithms executed on a server to identify focus points, eye movement patterns, and emotional states.

[0339] The "advice generation means" is software for providing personalized advice to the user based on the analysis results.

[0340] "Gaze data" refers to data relating to eye movements and focus points recorded by a visual data storage device.

[0341] "Emotion data" refers to data relating to the user's emotional state, such as facial expressions, tone of voice, and heart rate, recognized by the emotion engine.

[0342] A "gaze map" is a visual display of analyzed gaze data, showing focus points and eye movement patterns.

[0343] A "heat map" is a visual representation of the distribution and concentration of analyzed data, showing the density and frequency of data within a specific range.

[0344] "Personalized advice" refers to suggestions for technical instruction and practice methods that are individually generated based on each user's gaze data and emotional data.

[0345] This invention combines a visual data storage device used by top athletes and coaches with an emotion engine that recognizes users' emotions in real time, a server, data analysis means, and advice generation means. This allows users to improve their skills as well as recognize and improve their own emotional state.

[0346] To implement the present invention, the following elements are included:

[0347] 1. Visual Data Storage Device

[0348] The visual data storage device is a device worn by top athletes and coaches that records eye movements and focus points in real time. The device is equipped with a camera and sensors, and the acquired gaze data is transmitted to a server via wireless communication.

[0349] 2. Emotion Engine

[0350] The emotion engine is a combination of software and hardware that analyzes data such as the user's facial expression, voice tone, and heart rate to recognize emotions in real time. This engine works in conjunction with a visual data storage device and transmits emotion data along with gaze data to a server.

[0351] 3. Server

[0352] The server receives, stores, and manages the gaze data and emotion data sent from the visual data storage device and emotion engine. The server has a database and has the function of individually storing data for each user. Furthermore, the server provides computing resources for executing the data analysis means and advice generation means.

[0353] 4. Data Analysis Methods

[0354] The data analysis tool is software and algorithms running on a server that analyzes gaze data and emotion data to identify focus points, eye movement patterns, and emotional states. This data is visualized as gaze maps and heat maps, which are then used to generate advice.

[0355] 5. Advice Generation Methods

[0356] The advice generator is software that provides personalized advice to users based on the analysis results. It suggests specific areas for improvement and practice methods based on the analyzed gaze data and emotion data.

[0357] Specific hardware and software used in the present invention include the following:

[0358] Visual data storage devices (e.g., cameras, sensors)

[0359] Emotion engine (e.g., facial expression analysis software, heart rate sensor)

[0360] Server (database, analysis algorithms, computational resources)

[0361] Data analysis methods (analysis algorithms using AI technology)

[0362] Advice generation means (personalized advice software)

[0363] Specific examples

[0364] For example, when gaze data and emotion data are collected by wearing a visual data storage device and an emotion engine during a soccer match, the system operates as follows.

[0365] Data Collection Phase

[0366] Server: Registers the device IDs of the visual data storage device and emotion engine, and configures data reception.

[0367] Terminal: The visual data storage device and emotion engine record eye movements and emotional states during the match in real time and periodically transmit the data to the server.

[0368] User: The user wears the device before the match, checks the fit, and then records gaze data and emotion data during the match.

[0369] Data analysis phase

[0370] Server: Cleans the received gaze data and emotion data and passes them to the data analysis means.

[0371] Data analysis method: AI analyzes gaze data and emotion data and displays them as gaze maps and heat maps.

[0372] Users: Review analytics to understand what they focused on and what their emotional state was during a match.

[0373] Advice Providing Phase

[0374] Server: The AI ​​generates advice based on the analysis results and sends it to the device.

[0375] Device: Display advice within your app so users can find out more information.

[0376] User: Based on the advice, conduct the next practice or match, and collect gaze and emotion data again to confirm improvement.

[0377] Prompt Sentence Examples

[0378] "Please collect the focus points and emotional state of top soccer players during a match and visualize them as gaze maps and heat maps. Also, please provide specific advice for improving skills based on that data."

[0379] In the above-described manner, the present invention enables a user to learn the gaze data and emotion data of top athletes and coaches, and to use this data to improve their own skills and control their emotions.

[0380] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0381] Step 1:

[0382] Initial Device Setup

[0383] Device:

[0384] The visual data storage device and emotion engine are powered on and initial calibration is performed, which involves adjusting the camera position and checking the accuracy of the sensors.

[0385] Input: Initial data from deployed cameras and sensors.

[0386] Output: Adjusted device state data after calibration.

[0387] User:

[0388] The wearer checks that the device is fitted correctly and checks that there are no problems with comfort or operation when wearing it.

[0389] Input: Physical location of the device.

[0390] Output: Confirmation data for successfully attached devices.

[0391] Step 2:

[0392] Data collection

[0393] Device:

[0394] Data collection begins. The visual data storage device records eye movements and focus points in real time, and the emotion engine monitors the user's facial expressions and heart rate, generating gaze data and emotion data.

[0395] Input: Real-time eye movement data and heart rate data.

[0396] Output: Collected gaze and emotion data.

[0397] User:

[0398] Users go about their normal practice or game routines and are aware that the device is collecting data.

[0399] Input: Practice and game action data.

[0400] Output: Data recorded by the Visual Data Storage Device and Emotion Engine.

[0401] Step 3:

[0402] Data Transfer

[0403] Device:

[0404] The collected data is sent to a server via wireless communication at regular intervals. For example, it can be set to send data at every half-time of a match.

[0405] Input: Collected data (gaze data and emotion data).

[0406] Output: The data sent to the server.

[0407] server:

[0408] Prepare to receive data and receive the data sent.

[0409] Input: Submitted gaze and emotion data.

[0410] Output: Stores the received data.

[0411] Step 4:

[0412] Data storage and preprocessing

[0413] server:

[0414] The received gaze and emotion data is stored in a database, after which the data is cleaned and denoised. During this pre-processing phase, the data is converted into an analyzable format.

[0415] Input: Incoming data (gaze data and emotion data).

[0416] Output: Preprocessed data.

[0417] Step 5:

[0418] Data analysis

[0419] server:

[0420] The data analysis means analyzes the preprocessed data, identifying focus points and eye movement patterns from the gaze data and emotional states from the emotion data, and generates gaze maps and heat maps.

[0421] Input: Preprocessed data.

[0422] Output: gaze maps and heat maps.

[0423] Data analysis methods:

[0424] Using AI technology, the data is analysed to identify focus points, eye movement patterns and emotional states.

[0425] Input: Preprocessed data.

[0426] Output: Analysis results (gaze maps, heat maps).

[0427] Step 6:

[0428] Advice Generation

[0429] server:

[0430] Based on the analysis results, the advice generation means generates personalized advice, proposing technical guidance and practice methods suited to each user.

[0431] Input: Analysis results (gaze map, heat map).

[0432] Output: Personalized advice.

[0433] Step 7:

[0434] Providing advice

[0435] server:

[0436] The generated advice is sent to the device.

[0437] Enter: personalized advice.

[0438] Output: Advice sent.

[0439] Device:

[0440] Display the advice to the user, allowing them to view more information through an interactive application.

[0441] Input: Advice sent by the server.

[0442] Output: The displayed advice.

[0443] User:

[0444] Use the advice provided to prepare for the next practice or game, and then wear the device again to prepare for the next data collection phase.

[0445] Input: The advice shown.

[0446] Output: Improvement actions based on the advice.

[0447] Through these steps, the system uses gaze data and emotion data to provide users with appropriate advice, enhancing both technical improvement and emotional control.

[0448] (Application example 2)

[0449] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0450] Conventional methods for analyzing customer behavior in brick-and-mortar stores do not adequately collect and analyze gaze data and emotional data, making it difficult to accurately grasp customers' true interests and emotional states. This makes it difficult to find specific improvement measures for optimizing store layout and maximizing the effectiveness of promotions. Furthermore, insufficient real-time data collection and analysis makes it difficult to respond immediately.

[0451] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0452] In this invention, the server includes means for receiving and storing gaze data and emotion data, means for analyzing the gaze data and emotion data to identify focus points and emotional states, and means for providing advice to users and visualizing customer behavior and emotions, thereby enabling real-time analysis of customer behavior and emotional states and provision of specific advice for effective store management.

[0453] A "visual data storage device" is a device worn by the customer that records the direction of their gaze and movements of their viewpoint in real time.

[0454] "Gaze data" refers to information about the direction of a customer's gaze and the position of the point of gaze recorded by a visual data storage device.

[0455] "Emotional data" refers to information about a customer's emotional state, such as facial expressions, tone of voice, and heart rate, collected by a visual data storage device.

[0456] A "server" is a computer system that receives, stores, and manages gaze data and emotion data.

[0457] "Data analysis means" refers to software and algorithms that run on a server and have the function of analyzing gaze data and emotion data to identify focus points and emotional states.

[0458] The "advice generating means" is software for providing specific advice to the user based on the analysis results obtained by the data analyzing means.

[0459] A "gaze map" is a visual map that shows which parts of a product customers are focusing on based on analyzed gaze data.

[0460] An "emotion map" is a visual map that shows a customer's emotional state based on analyzed emotional data.

[0461] A "heat map" is a map that visually represents gaze data and emotion data, showing gaze time, emotion intensity, etc., using shades of color.

[0462] The present invention relates to a system for analyzing customer behavior and emotions in a physical store. The system includes a visual data storage device worn by a customer, a server that receives and stores the data, a data analysis means that analyzes the data, and an advice generation means that provides advice to a user.

[0463] Key elements of the system

[0464] 1. Visual Data Storage Device

[0465] The visual data storage device is a device worn by the customer that records the direction of gaze and movement of the viewpoint in real time. This device has a built-in camera and sensors and collects gaze data and emotional data. Emotional data can be obtained from facial expressions, voice tone, heart rate, etc.

[0466] 2. Server

[0467] The server is a computer system that receives, stores, and manages gaze data and emotion data. The server uses multiple databases to store individual customer data and provides computing resources to run the analysis means and advice generation means.

[0468] 3. Data Analysis Methods

[0469] The data analysis means is software and algorithms running on a server that analyzes gaze data and emotion data to identify focus points and emotional states. The analysis results are visualized as gaze maps, emotion maps, or heat maps.

[0470] 4. Advice Generation Methods

[0471] The advice generator is software that provides specific advice to store staff and marketing teams based on the analyzed gaze data and emotion data, enabling them to improve store layout and optimize promotions.

[0472] Explanation of program processing

[0473] The gaze data and emotion data collected by the visual data storage device are transmitted to a server via wireless communication. The server stores the received data and performs preprocessing, which includes data cleaning and noise removal. The preprocessed data is then analyzed by a data analysis means to identify focus points and emotional states. The analysis results, visualized as gaze maps and emotion maps, are passed to an advice generation means.

[0474] Based on the analysis results, the advice generator identifies which areas and products customers are most interested in, and in which situations their emotions are stirred, and suggests specific areas for improvement. This advice can be viewed in real time by store staff and marketing teams, enabling prompt action.

[0475] Specific examples

[0476] For example, in a department store, staff wearing smart glasses observe customer behavior. They collect gaze and emotion data when customers enter a specific area and look at specific products. The server analyzes this data and displays gaze and emotion maps showing which products customers are interested in and which areas they feel stressed or excited in. Based on the analysis results, the system provides specific advice for improving the next promotion or product placement.

[0477] Prompt Sentence Examples

[0478] "Detect customer gaze points and emotional states from input images and generate gaze maps and emotion maps in real time. The gaze map should identify the products and areas customers are focusing on, and the emotion map should display their emotional states, such as smile, surprise, or excitement. Furthermore, provide specific advice to improve the customer experience based on the data."

[0479] As described above, the present invention aims to optimize store operations and improve customer experience by analyzing customer behavior and emotions in detail in brick-and-mortar stores, enabling quick responses on-site and strategic decisions based on data.

[0480] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0481] Step 1:

[0482] Data Collection Phase

[0483] The terminal (visual data storage device) records the customer's gaze data and emotional data (facial expressions, voice tone, heart rate, etc.) in real time. The input is real-time visual and physiological data, and the output is collected gaze data and emotional data. This data is periodically transmitted to the server via wireless communication. In other words, the terminal collects data and prepares it to be transmitted to the server.

[0484] Step 2:

[0485] Data Receiving Phase

[0486] The server receives and stores the gaze data and emotion data sent from the device. The input is the gaze data and emotion data sent from the device, and the output is the data stored on the server. Specifically, the server records the data sent in a database and prepares for the next analysis phase.

[0487] Step 3:

[0488] Data Preprocessing Phase

[0489] The server preprocesses the received gaze and emotion data. The input is the stored gaze and emotion data, and the output is cleaned and denoised data. This preprocessing includes data cleaning and denoising, such as imputing missing values ​​and correcting anomalous data, and converting the data into an analyzable format.

[0490] Step 4:

[0491] Data analysis phase

[0492] The server analyzes the preprocessed gaze data and emotion data using data analysis means. The input is the preprocessed gaze data and emotion data, and the output is the analysis results visualized as a gaze map, emotion map, or heat map. Specifically, the AI ​​identifies gaze focus points and emotional states and processes them into a format that can be displayed visually.

[0493] Step 5:

[0494] Advice Generation Phase

[0495] The server generates specific advice using the advice generation means based on the analysis results obtained by the data analysis means. The input is the visualized analysis results, and the output is personalized advice. Specifically, it generates improvement suggestions for store layout and promotions based on which areas and products customers are interested in and in which situations they are emotionally moved.

[0496] Step 6:

[0497] Advice Providing Phase

[0498] The terminal receives the advice sent from the server and displays it to the user. The input is the generated advice, and the output is the advice presented to the user. Specifically, the terminal interactively displays the advice content and provides an interface that allows the user to check detailed information.

[0499] Step 7:

[0500] Feedback gathering phase

[0501] Based on the provided advice, the user makes changes to the store layout or improves customer service methods, and evaluates the effectiveness. The input is the specific actions taken based on the advice, and the output is feedback data on the effectiveness of the implemented improvements. Specifically, the visual data storage device is used again to collect gaze data and emotion data, and the areas for improvement are confirmed and evaluated based on the new data.

[0502] Through the above processing steps, it is possible to perform detailed analysis of customer behavior and emotional states in physical stores, and based on that, it is possible to effectively manage stores and improve customer experience.

[0503] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0504] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0505] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0506] [Second embodiment]

[0507] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0508] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0509] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0510] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0511] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0512] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0513] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0514] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0515] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0516] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0517] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0518] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0519] The present invention is embodied as a system including a visual data storage device worn by top athletes and coaches, data analysis means, and advice generation means. This allows users to learn the perspectives and awareness of top athletes and use this knowledge to improve their skills. Specific details of the embodiments of the present invention are described below.

[0520] Key elements of the system

[0521] 1. Visual Data Storage Device

[0522] The visual data storage device is a device worn by top athletes and coaches that records eye movements and focus points in real time. The device is equipped with a camera and sensors, and the acquired gaze data is sent to a server via wireless communication.

[0523] 2. Server

[0524] The server receives, stores, and manages the gaze data sent from the visual data storage device. The server has a database and has the function of individually storing data for each user. Furthermore, the server provides the computing resources to execute the analysis means and the advice generation means.

[0525] 3. Data Analysis Methods

[0526] The data analysis tool is software and algorithms that run on a server. It analyzes gaze data to identify focus points, eye movement patterns, etc. This data is visualized as gaze maps and heat maps and is used to generate advice later.

[0527] 4. Advice Generation Methods

[0528] The advice generator is software that provides personalized advice to users based on the analysis results. It suggests specific areas for improvement and practice methods based on the analyzed gaze data.

[0529] Program processing flow

[0530] 1. Data Collection Phase

[0531] server:

[0532] The server works with the visual data storage device to prepare for data collection. Specifically, the server registers the device ID and user information of the visual data storage device in a database and sets up data reception.

[0533] Device:

[0534] Visual data storage devices are worn by top athletes and coaches to record their eye movements and focus points in real time. The wearer's gaze data is stored on the device and periodically sent to a server.

[0535] User:

[0536] Before practice or a game, the visual data storage device is attached and checked for proper fit. After confirming that the device is attached correctly, data collection begins.

[0537] 2. Data analysis phase

[0538] server:

[0539] The server stores the received gaze data and performs pre-processing, which includes cleaning and formatting the data to convert it into a format that can be analyzed by the data analysis tool.

[0540] Data analysis methods:

[0541] Gaze data is analyzed to identify focus points and eye movement patterns, and the analysis results are visualized as gaze maps and heat maps.

[0542] User:

[0543] Check the analysis results to understand where top athletes and coaches focus their attention. Visually check specific eye movements as gaze maps and heat maps.

[0544] 3. Advice Phase

[0545] server:

[0546] Based on the results of the data analysis, the AI ​​generates personalized advice, which is then sent from the server to the device.

[0547] Device:

[0548] The received advice is displayed to the user, who can then interactively review the advice within the application and receive further information.

[0549] User:

[0550] The player then practices or plays a match based on the advice provided. The visual data storage device is then worn again to collect gaze data and check the progress of the player's improvement. This creates a feedback loop, enabling gradual improvement of skills.

[0551] Specific examples

[0552] For example, if you were to wear a visual data storage device to collect gaze data during a soccer game:

[0553] Data Collection Phase

[0554] Server: Registers the device ID of the visual data storage device and configures data reception.

[0555] Device: The glasses record eye movements in real time during the game and periodically send the data to a server.

[0556] User: Put on the glasses before the match, check the fit, and then record gaze data during the match.

[0557] Data analysis phase

[0558] Server: Cleans the received gaze data and passes it to the data analysis means.

[0559] Data analysis method: AI analyzes gaze data and displays it as a gaze map or heat map.

[0560] Users: Check the analytics results to understand what they focused on during the match.

[0561] Advice Providing Phase

[0562] Server: The AI ​​generates advice based on the analysis results and sends it to the device.

[0563] Device: Display advice within your app so users can find out more information.

[0564] User: Follow the advice and then perform your next practice or match, collecting gaze data again to see any improvements.

[0565] In this way, the present invention provides an effective system for users to study the gaze data of top athletes and coaches and improve their skills.

[0566] The processing flow will be explained below.

[0567] Step 1:

[0568] Server: The server starts the system that links with the visual data storage device and prepares for data transmission. The server registers the device ID and user information of the visual data storage device in the database and sets up data reception.

[0569] Step 2:

[0570] Device: Power on the visual data storage device, check the network connection, pair it with the corresponding mobile app, and verify that the device is working properly.

[0571] Step 3:

[0572] User: Wears the visual data storage device before training or a match. Checks the fit and secures it in place. Presses the Start Session button to begin data collection.

[0573] Step 4:

[0574] Device: The glasses record eye movements in real time and capture gaze data, which is temporarily stored in a buffer on the device and sent to the server at set intervals.

[0575] Step 5:

[0576] Server: Receives gaze data sent from the device periodically and accumulates it in a database. The data is saved with a timestamp for later analysis.

[0577] Step 6:

[0578] Server: Preprocesses the received gaze data, cleaning and denoising the data, and optionally formatting the data into an analyzable format.

[0579] Step 7:

[0580] Data analysis means (server): Analyzes gaze data using an AI model. Specifically, it analyzes gaze movement patterns, detection of attention points, reaction times, etc. The analysis results are visualized as gaze maps and heat maps.

[0581] Step 8:

[0582] Server: Analyzed data is stored on the server and prepared for transmission to the device as needed.

[0583] Step 9:

[0584] Device: Once the analysis results are sent from the server, they are presented to the user. They are displayed as gaze maps or heat maps within the application, allowing the user to visually confirm the results.

[0585] Step 10:

[0586] User: Check the analysis results through the application, specifically understanding where top coaches and athletes are looking and when they move their eyes.

[0587] Step 11:

[0588] Advice generator (server): The AI ​​model generates personalized advice based on the analysis results. This advice includes specific areas for improvement and practice.

[0589] Step 12:

[0590] Server: Sends generated advice to the device, formatted in a user-friendly format.

[0591] Step 13:

[0592] Terminal: Display the received advice to the user, interactively within the application, allowing the user to view more information if needed.

[0593] Step 14:

[0594] User: Based on the advice provided, the user will train and compete, and in the process will use the visual data storage device again to collect data for further improvement.

[0595] Step 15:

[0596] Server: Re-analyzes the data collected and evaluates the user's progress. Always provides feedback based on the latest data.

[0597] Through this series of processes, users can learn in detail about the gaze and awareness of top athletes and coaches, and use this knowledge to improve their own skills.

[0598] Example 1

[0599] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0600] In conventional sports instruction and training, there is a lack of means to collect gaze data from top athletes and coaches in real time and provide specific feedback based on that data. This makes it difficult for individual athletes to easily obtain specific advice on how to improve their skills. In addition, there is a lack of methods to visually represent and present the analysis results in an easy-to-understand manner.

[0601] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0602] In this invention, the server includes a visual data storage device worn by top athletes and coaches, an information processing device that receives and stores gaze data recorded by the visual data storage device, a preprocessing device that cleans the gaze data and passes it to data analysis means, a data analysis means that analyzes the gaze data and identifies focus points and eye movement patterns, a means for visualizing the analysis results as a gaze map or heat map, an advice generation means that provides advice to users based on the analysis results, and a display means that displays the advice on the user's terminal. This allows users to learn the perspectives and awareness of top athletes and use it to improve their own skills.

[0603] A "visual data storage device" is a device worn by top athletes and coaches to record eye movements and focus points in real time.

[0604] The "information processing device" is a device that receives, stores, and manages the gaze data transmitted from the visual data storage device.

[0605] The "pre-processing device" is a device that cleans the received gaze data, removes noise, complements missing data, and passes the data to the data analysis means.

[0606] "Data analysis means" refers to software and algorithms for analyzing gaze data and identifying focus points and eye movement patterns.

[0607] A "gaze map" is a map that visually represents gaze movements based on gaze data.

[0608] A "heat map" is a visual representation that shows gaze points using shades of color.

[0609] An "advice generator" is software and algorithms for providing personalized advice to a user based on the analysis results.

[0610] The "display means" is a device or function for displaying the generated advice and analysis results on the user's terminal.

[0611] A "generative AI model" is an artificial intelligence model that analyzes gaze data and generates gaze maps or heat maps.

[0612] The present invention is embodied as a system including a visual data storage device worn by top athletes and coaches, data analysis means, and advice generation means. This allows users to learn the perspectives and awareness of top athletes and use this knowledge to improve their skills. Specific details of the embodiments of the present invention are described below.

[0613] Key elements of the system

[0614] 1. Visual Data Storage Device

[0615] The visual data storage device is a device worn by top athletes and coaches. It is equipped with cameras and sensors that record eye movements and focus points in real time. The collected gaze data is transmitted to a server via wireless communication.

[0616] 2. Server

[0617] The server receives, stores, and manages gaze data transmitted from the visual data storage device. In particular, the server incorporates a pre-processing unit that cleans the gaze data and passes it to the data analysis unit. The server also provides the computing resources to run the data analysis unit and the advice generation unit.

[0618] 3. Data Analysis Methods

[0619] The data analysis tool is software and algorithms that run on a server. It analyzes gaze data to identify focus points and eye movement patterns. This data is visualized as gaze maps and heat maps and used to generate advice later.

[0620] 4. Advice Generation Methods

[0621] The advice generator is software that provides personalized advice to users based on the analysis results of the data analysis means. It uses a generative AI model to suggest specific areas for improvement and practice methods.

[0622] 5. Display means

[0623] The display means is a device or function for displaying the generated advice and analysis results on the user's terminal. It has an interactive display function, allowing the user to check detailed information.

[0624] Example of operation flow

[0625] For example, consider the case where a visual data storage device is worn by a player during a soccer match to collect gaze data. In this case, the server registers the device ID of the visual data storage device and configures data reception. During the match, the device's camera records the player's eye movements in real time and periodically sends this to the server. The server cleans the received gaze data and passes it to the data analysis means. This means analyzes the gaze data using a generative AI model and generates a gaze map or heat map. The server sends advice generated by the AI ​​to the device, which displays the advice within the application. The player then performs the next practice or match based on the advice provided.

[0626] Prompt Sentence Examples

[0627] "Please explain the algorithm that analyzes focus points and eye movement patterns using gaze data from athletes wearing visual data storage devices."

[0628] With this structure, the system of the present invention allows users to learn the gaze data of top athletes and coaches and use it to improve their own skills.

[0629] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0630] Step 1: Prepare for data collection

[0631] Server Action:

[0632] The server registers the device ID and user information of the visual data storage device in a database. The input is the device ID (e.g., ID12345) of the visual data storage device and the user information, and outputs a database entry for linking and managing the device ID and user information. The server then performs operations to prepare for communication with the visual data storage device.

[0633] Step 2: Start collecting data

[0634] Terminal handling:

[0635] The device is worn by top athletes and coaches. The device's built-in camera and sensors record eye movements and focus points in real time. The input is the athlete's eye movements, and the output is the recorded gaze data. The device operates to collect gaze data in real time.

[0636] Step 3: Send data

[0637] Terminal handling:

[0638] The terminal periodically collects and transmits the gaze data to the server via wireless communication. The input is the gaze data recorded on the terminal, and the output is the gaze data transmitted to the server. The terminal controls the timing of data transmission and performs operations to maintain data consistency.

[0639] Step 4: Save Data

[0640] Server Action:

[0641] The server stores the received gaze data in a database. The input is the gaze data received from the device, and the output is the gaze data stored in the database. It performs operations to verify the accuracy of the data (for example, checking data reception and verifying it before saving).

[0642] Step 5: Data Cleaning

[0643] Server Action:

[0644] The server cleans the stored gaze data, removes noise, and fills in missing data. The input is the stored gaze data, and the output is the cleaned gaze data. It applies noise filtering and data filling algorithms.

[0645] Step 6: Data analysis

[0646] Processing data analysis methods:

[0647] The data analysis means analyzes the cleaned gaze data and identifies focus points and eye movement patterns. The input is the cleaned gaze data and the output is the analysis results. The data analysis means performs operations to execute a focus point detection algorithm and a movement pattern analysis algorithm.

[0648] Step 7: Generate gaze maps and heat maps

[0649] Processing data analysis methods:

[0650] The data analysis means visualizes the analysis results as gaze maps and heat maps. The input is the analysis results, and the output is gaze maps and heat maps. A visualization algorithm is used to graphically display the gaze data.

[0651] Step 8: Advice Generation

[0652] Advice generator processing:

[0653] The advice generation means generates personalized advice for the user based on the analysis results of the gaze map and heat map. The input is the analysis results of the gaze map and heat map, and the output is the generated advice. The system operates to generate personalized feedback using a generative AI model.

[0654] Step 9: Submitting Advice

[0655] Server Action:

[0656] The server sends the generated advice to the terminal. The input is the generated advice and the output is the advice sent to the terminal. It performs actions to ensure the accuracy of the data transmission (e.g., checking the data package and confirming transmission).

[0657] Step 10: Viewing Advice

[0658] Terminal handling:

[0659] The terminal displays the advice received within the application to the user. The input is the advice received from the server, and the output is the advice visually displayed to the user. It provides an interactive display function and performs actions that allow the user to check detailed information.

[0660] Step 11: Act on the advice

[0661] User Action:

[0662] The user then practices or plays a game based on the advice provided. The input is the advice displayed on the device, and the output is the action the user performs. The user then wears the visual data storage device again, collects gaze data during the next game or practice, and performs actions to check for improvement.

[0663] (Application example 1)

[0664] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0665] Currently, there is no real-time advice system to improve the driving skills of autonomous vehicles, so there is a need for autonomous driving systems to provide appropriate feedback on the driving situation by utilizing gaze data from professional drivers. Conventional systems do not sufficiently link the collection and analysis of driving data, making it difficult to improve technology and safety in real time.

[0666] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0667] In this invention, the server includes a means for receiving and storing gaze data, a means for analyzing the gaze data to identify focus points and eye movement patterns, and a means for providing feedback to the autonomous driving AI system based on the analysis results. This makes it possible to improve driving skills in real time using gaze data from professional drivers and provide advice according to driving conditions.

[0668] A "visual data storage device" is a device worn by the pilot to record eye movements and gaze direction.

[0669] A "server" is a central device for receiving, storing, and processing gaze data transmitted from a visual data repository.

[0670] The "data analysis means" is software or algorithm that runs on the server and analyzes the received gaze data to identify focus points and gaze movement patterns.

[0671] The "advice generation means" is software or an algorithm that provides the user with advice for improving their skills based on the analysis results and the gaze data of top pilots.

[0672] "Means for improving autonomous driving technology" refers to a means for supplying analyzed gaze data to the driving AI system of an autonomous vehicle in order to improve driving technology.

[0673] The "real-time advice providing means" is a means for providing appropriate advice for the driving situation in real time.

[0674] A "gaze map" is a map that visually displays gaze movement patterns based on gaze data from professional drivers.

[0675] A "heat map" is a map that visually displays points of focus and the degree of gaze concentration.

[0676] A "driving AI system" is an artificial intelligence system that controls the driving of autonomous vehicles and optimizes driving operations based on external data.

[0677] This invention is implemented as a system that provides a visual data storage device worn by a professional driver, a real-time data analysis means, and appropriate driving advice to improve automated driving technology. The system includes the following main elements:

[0678] Key elements of the system

[0679] 1. Visual Data Storage Device

[0680] These smart glasses are worn by professional drivers and record eye movements and gaze direction in real time, wirelessly transmitting the data to a server.

[0681] 2. Server

[0682] The server receives and stores the gaze data sent from the visual data storage device. It analyzes the data and provides the analysis results to the driving AI system. As a specific example, an Amazon Web Services (AWS) EC2 instance is used.

[0683] 3. Data Analysis Methods

[0684] This is software and algorithms that run on a server to analyze gaze data, identify focus points and gaze movement patterns, and visualize them as gaze maps and heat maps using Python and OpenCV.

[0685] 4. Advice Generation Methods

[0686] It is software or algorithm that provides feedback to autonomous driving AI systems based on analysis results, generating personalized advice using TensorFlow.

[0687] 5. Real-time advice delivery methods

[0688] It is a means of providing appropriate advice in real time on driving situations during automated driving, which will improve automated driving technology and increase driving safety.

[0689] Specific processing explanation of the system

[0690] The server receives gaze data from the visual data storage device and stores it in a database. This data first undergoes a data cleaning process and is formatted for analysis. The gaze data is then analyzed using Python and OpenCV and visualized as gaze maps and heat maps. The analysis results are input into an AI model using TensorFlow to generate personalized advice for the driving AI system.

[0691] Specific examples of processing

[0692] Once the server receives the visual data, it performs data cleaning to remove noise. It is then analyzed using OpenCV to generate a gaze map. For example, it can identify the gaze points of a professional driver when approaching an intersection and provide this to the driving AI system. Based on this data, the generative AI model can provide appropriate driving advice in real time.

[0693] Examples of prompt statements

[0694] Here are some example prompts to input to a generative AI model:

[0695] Based on the gaze data analysis results, identify the points that professional drivers focus on while driving and provide optimal advice and suggestions for improvement for safe driving.

[0696] By using this prompt, the AI ​​model can generate specific advice for improving driving skills based on the analysis results.

[0697] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0698] Step 1:

[0699] Visual data collection

[0700] The user has a professional driver wear smart glasses, which collect real-time gaze data while driving and transmit it to a server via Wi-Fi.

[0701] Input: Professional driver's gaze data

[0702] Output: Gaze data sent to the server

[0703] Step 2:

[0704] Receiving and storing data

[0705] The server receives the gaze data sent from the smart glasses and stores it securely in a database.

[0706] Input: Gaze data sent from smart glasses

[0707] Output: Gaze data stored in a database

[0708] Step 3:

[0709] Data Preprocessing

[0710] The server performs a cleaning process on the received gaze data, which involves removing noise and reformatting the data format.

[0711] Input: Gaze data stored in a database

[0712] Output: Cleaned gaze data

[0713] Step 4:

[0714] Data analysis

[0715] The server analyzes the cleaned gaze data using data analysis tools. It uses OpenCV to generate gaze maps and heat maps. This analysis identifies focus points and gaze movement patterns while driving.

[0716] Input: Cleaned gaze data

[0717] Output: gaze maps and heat maps

[0718] Step 5:

[0719] Advice Generation

[0720] Based on the analysis results, the server uses TensorFlow to generate AI model advice for the driving AI system. It uses prompt sentences to suggest appropriate driving advice and improvements.

[0721] Input: gaze maps and heat maps

[0722] Output: Advice for driving AI systems

[0723] Step 6:

[0724] Real-time advice provided

[0725] The server sends the generated advice to the driving AI system in real time, which then optimizes driving operations based on the advice received, ensuring safe driving.

[0726] Input: Advice for driving AI systems

[0727] Output: Optimized driving operation by the driving AI system

[0728] Through this series of processes, the gaze data of professional drivers is used to improve autonomous driving technology and provide appropriate advice according to the driving situation.

[0729] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0730] The present invention is embodied as a system that combines a visual data storage device worn by top athletes and coaches, data analysis means, and advice generation means with an emotion engine that recognizes the user's emotions. This allows users to learn not only the perspectives and awareness of top athletes, but also their own emotional state, which can be used to improve their skills. Specific details of the embodiments of the present invention are described below.

[0731] Key elements of the system

[0732] 1. Visual Data Storage Device

[0733] The visual data storage device is a device worn by top athletes and coaches that records eye movements and focus points in real time. The device is equipped with a camera and sensors, and the acquired gaze data is sent to a server via wireless communication.

[0734] 2. Emotion Engine

[0735] The emotion engine is a combination of software and hardware that analyzes data such as the user's facial expressions, voice tone, and heart rate to recognize emotions in real time. It is built into the visual data storage device and transmits emotion data along with gaze data to the server.

[0736] 3. Server

[0737] The server receives, stores, and manages the gaze data and emotion data sent from the visual data storage device and emotion engine. The server has a database and has the function of individually storing data for each user. Furthermore, the server provides the computational resources to execute the analysis means and advice generation means.

[0738] 4. Data Analysis Methods

[0739] The data analysis tool is software and algorithms running on a server. It analyzes gaze data and emotion data to identify focus points, eye movement patterns, and emotional states. This data is visualized as gaze maps and heat maps and used to generate advice.

[0740] 5. Advice Generation Methods

[0741] The advice generator is software that provides personalized advice to users based on the analysis results. It suggests specific areas for improvement and practice methods based on the analyzed gaze data and emotion data.

[0742] Program processing flow

[0743] 1. Data Collection Phase

[0744] server:

[0745] The server works with the visual data storage device and emotion engine to prepare for data collection. Specifically, the server registers the device IDs and user information of the visual data storage device and emotion engine in a database and sets up data reception.

[0746] Device:

[0747] Visual data storage devices and emotion engines are worn by top athletes and coaches to record their eye movements, focus points, and emotional states in real time. The wearer's gaze data and emotion data are stored on the device and periodically transmitted to a server.

[0748] User:

[0749] Before practice or a game, the visual data storage device is attached and checked for proper fit. After confirming that the device is attached correctly, data collection begins.

[0750] 2. Data analysis phase

[0751] server:

[0752] The server stores and pre-processes the received gaze and emotion data, including cleaning and denoising the data, and converts it into a format that can be analyzed by the data analysis means.

[0753] Data analysis methods:

[0754] Gaze data and emotional data are analyzed to identify focus points, eye movement patterns, and emotional states. The analysis results are visualized as gaze maps and heat maps.

[0755] User:

[0756] By reviewing the analysis results, you can understand where top athletes and coaches focused and what their emotional state was. Visually check specific eye movements and emotional states as gaze maps and heat maps.

[0757] 3. Advice Phase

[0758] server:

[0759] Based on the results of the data analysis, the AI ​​generates personalized advice, which is then sent from the server to the device.

[0760] Device:

[0761] The received advice is displayed to the user, who can then interactively review the advice within the application and receive further information.

[0762] User:

[0763] The player then practices and plays matches based on the advice provided. The visual data storage device is then worn again to collect gaze and emotional data, and the player's progress is monitored. This creates a feedback loop, enabling gradual improvement in technique and emotional control.

[0764] Specific examples

[0765] For example, if you wear a visual data storage device and an emotion engine to collect gaze data and emotion data during a soccer game:

[0766] Data Collection Phase

[0767] Server: Registers the device IDs of the visual data storage device and emotion engine, and configures data reception.

[0768] Device: The glasses record eye movements and emotional state in real time during the match and periodically transmit the data to a server.

[0769] User: The user puts on the glasses before the match, checks their fit, and then records their gaze and emotion data during the match.

[0770] Data analysis phase

[0771] Server: Cleans the received gaze data and emotion data and passes them to the data analysis means.

[0772] Data analysis method: AI analyzes gaze data and emotion data and displays them as gaze maps and heat maps.

[0773] Users: Review analytics to understand what they focused on and what their emotional state was during a match.

[0774] Advice Providing Phase

[0775] Server: The AI ​​generates advice based on the analysis results and sends it to the device.

[0776] Device: Display advice within your app so users can find out more information.

[0777] User: Based on the advice, conduct the next practice or match, and collect gaze and emotion data again to confirm improvement.

[0778] In this way, the present invention allows users to learn the gaze data and emotion data of top athletes and coaches and use it to improve their own skills and control their emotions.

[0779] The processing flow will be explained below.

[0780] Step 1:

[0781] Server: Registers the device IDs and user information of the visual data storage device and emotion engine in the database, and sets up data reception. Installs the server software on the server and prepares it for operation.

[0782] Step 2:

[0783] Device: Power on the visual data storage device and emotion engine, check the network connection, pair with the corresponding mobile app, and verify that the device is working properly. Calibrate the emotion engine so that it can properly capture data such as the user's facial expressions, tone of voice, and heart rate.

[0784] Step 3:

[0785] User: Before training or a match, wear the visual data storage device and emotion engine and check the fit. After making sure the device is fixed in the correct position, press the start button to begin data collection.

[0786] Step 4:

[0787] Device: The glasses record eye movements and the emotion engine records emotion data in real time, capturing them as gaze data and emotion data. The captured data is temporarily stored in a buffer on the device and sent to the server at set intervals.

[0788] Step 5:

[0789] Server: Receives gaze data and emotion data periodically sent from the device and stores them in a database. The data is saved with a timestamp for later analysis.

[0790] Step 6:

[0791] Server: Preprocesses the received gaze and emotion data, cleaning and denoising the data, and optionally formatting the data to make it more analyzable.

[0792] Step 7:

[0793] Data analysis means (server): Analyzes gaze data and emotional data using an AI model. Specifically, it analyzes gaze movement patterns, detection of attention points, reaction times, and emotional states. The analysis results are visualized as gaze maps and heat maps.

[0794] Step 8:

[0795] Server: The analyzed data is stored on the server and prepared for transmission to the device as needed. The analysis results are formatted in a user-friendly format.

[0796] Step 9:

[0797] Terminal: When the analysis results are sent from the server, they are presented to the user. They are displayed as gaze maps or heat maps, allowing the user to visually confirm the results.

[0798] Step 10:

[0799] Users: View analytics results through the application, specifically to understand where elite athletes and coaches focus their efforts and their emotional state.

[0800] Step 11:

[0801] Advice generation means (server): The AI ​​model generates personalized advice based on the analysis results. This advice includes specific improvements and practice methods related to both gaze data and emotion data.

[0802] Step 12:

[0803] Server: Sends generated advice to the device, formatted in a user-friendly format.

[0804] Step 13:

[0805] Terminal: Display the received advice to the user, interactively within the application, allowing the user to view more information if needed.

[0806] Step 14:

[0807] User: Based on the advice provided, the user trains and competes, again using the visual data storage device and emotion engine in the process to collect data for further improvement.

[0808] Step 15:

[0809] Server: Re-analyzes the newly collected data and evaluates the user's progress. Always provides feedback based on the latest data, and checks the effectiveness of the user's technical improvement and emotional control.

[0810] Through this series of processes, users can learn in detail about the gaze and emotional states of top athletes and coaches, and use this knowledge to improve their own skills and manage their emotions.

[0811] Example 2

[0812] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0813] While conventional sports training systems focus on analyzing the user's visual information and focus points, they lack the technology to provide comprehensive training advice that takes the user's emotional state into account. As a result, feedback on the user's technical improvement is limited, making it difficult to adequately support emotional control and mental growth. Furthermore, real-time data collection and analysis are insufficient, making it difficult for users to receive immediate advice.

[0814] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0815] In this invention, the server includes a visual data storage device worn by top athletes and coaches, an emotion engine that recognizes the user's emotions in real time, means for receiving and saving data recorded by the visual data storage device and emotion engine, means for analyzing the gaze data and emotion data received from the server and identifying focus points, eye movement patterns, and emotional states, and means for providing personalized advice to the user based on the results of the analysis. This enables the user to receive comprehensive advice that takes into account not only visual information but also emotional states, thereby enhancing both technical improvement and emotional control.

[0816] A "visual data storage device" is a device worn by top athletes and coaches that records eye movements and focus points in real time.

[0817] An "emotion engine" is a combination of software and hardware that analyzes data such as a user's facial expressions, voice tone, and heart rate to recognize emotions in real time.

[0818] "Server" refers to a system that receives, stores, and manages data sent from the visual data storage device and emotion engine, and provides the computational resources necessary for analyzing the data and generating advice.

[0819] "Data analysis means" refers to a means for analyzing gaze data and emotion data by software and algorithms executed on a server to identify focus points, eye movement patterns, and emotional states.

[0820] The "advice generation means" is software for providing personalized advice to the user based on the analysis results.

[0821] "Gaze data" refers to data relating to eye movements and focus points recorded by a visual data storage device.

[0822] "Emotion data" refers to data relating to the user's emotional state, such as facial expressions, tone of voice, and heart rate, recognized by the emotion engine.

[0823] A "gaze map" is a visual display of analyzed gaze data, showing focus points and eye movement patterns.

[0824] A "heat map" is a visual representation of the distribution and concentration of analyzed data, showing the density and frequency of data within a specific range.

[0825] "Personalized advice" refers to suggestions for technical instruction and practice methods that are individually generated based on each user's gaze data and emotional data.

[0826] This invention combines a visual data storage device used by top athletes and coaches with an emotion engine that recognizes users' emotions in real time, a server, data analysis means, and advice generation means. This allows users to improve their skills as well as recognize and improve their own emotional state.

[0827] To implement the present invention, the following elements are included:

[0828] 1. Visual Data Storage Device

[0829] The visual data storage device is a device worn by top athletes and coaches that records eye movements and focus points in real time. The device is equipped with a camera and sensors, and the acquired gaze data is transmitted to a server via wireless communication.

[0830] 2. Emotion Engine

[0831] The emotion engine is a combination of software and hardware that analyzes data such as the user's facial expression, voice tone, and heart rate to recognize emotions in real time. This engine works in conjunction with a visual data storage device and transmits emotion data along with gaze data to a server.

[0832] 3. Server

[0833] The server receives, stores, and manages the gaze data and emotion data sent from the visual data storage device and emotion engine. The server has a database and has the function of individually storing data for each user. Furthermore, the server provides computing resources for executing the data analysis means and advice generation means.

[0834] 4. Data Analysis Methods

[0835] The data analysis tool is software and algorithms running on a server that analyzes gaze data and emotion data to identify focus points, eye movement patterns, and emotional states. This data is visualized as gaze maps and heat maps, which are then used to generate advice.

[0836] 5. Advice Generation Methods

[0837] The advice generator is software that provides personalized advice to users based on the analysis results. It suggests specific areas for improvement and practice methods based on the analyzed gaze data and emotion data.

[0838] Specific hardware and software used in the present invention include the following:

[0839] Visual data storage devices (e.g., cameras, sensors)

[0840] Emotion engine (e.g., facial expression analysis software, heart rate sensor)

[0841] Server (database, analysis algorithms, computational resources)

[0842] Data analysis methods (analysis algorithms using AI technology)

[0843] Advice generation means (personalized advice software)

[0844] Specific examples

[0845] For example, when gaze data and emotion data are collected by wearing a visual data storage device and an emotion engine during a soccer match, the system operates as follows.

[0846] Data Collection Phase

[0847] Server: Registers the device IDs of the visual data storage device and emotion engine, and configures data reception.

[0848] Terminal: The visual data storage device and emotion engine record eye movements and emotional states during the match in real time and periodically transmit the data to the server.

[0849] User: The user wears the device before the match, checks the fit, and then records gaze data and emotion data during the match.

[0850] Data analysis phase

[0851] Server: Cleans the received gaze data and emotion data and passes them to the data analysis means.

[0852] Data analysis method: AI analyzes gaze data and emotion data and displays them as gaze maps and heat maps.

[0853] Users: Review analytics to understand what they focused on and what their emotional state was during a match.

[0854] Advice Providing Phase

[0855] Server: The AI ​​generates advice based on the analysis results and sends it to the device.

[0856] Device: Display advice within your app so users can find out more information.

[0857] User: Based on the advice, conduct the next practice or match, and collect gaze and emotion data again to confirm improvement.

[0858] Prompt Sentence Examples

[0859] "Please collect the focus points and emotional state of top soccer players during a match and visualize them as gaze maps and heat maps. Also, please provide specific advice for improving skills based on that data."

[0860] In the above-described manner, the present invention enables a user to learn the gaze data and emotion data of top athletes and coaches, and to use this data to improve their own skills and control their emotions.

[0861] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0862] Step 1:

[0863] Initial Device Setup

[0864] Device:

[0865] The visual data storage device and emotion engine are powered on and initial calibration is performed, which involves adjusting the camera position and checking the accuracy of the sensors.

[0866] Input: Initial data from deployed cameras and sensors.

[0867] Output: Adjusted device state data after calibration.

[0868] User:

[0869] The wearer checks that the device is fitted correctly and checks that there are no problems with comfort or operation when wearing it.

[0870] Input: Physical location of the device.

[0871] Output: Confirmation data for successfully attached devices.

[0872] Step 2:

[0873] Data collection

[0874] Device:

[0875] Data collection begins. The visual data storage device records eye movements and focus points in real time, and the emotion engine monitors the user's facial expressions and heart rate, generating gaze data and emotion data.

[0876] Input: Real-time eye movement data and heart rate data.

[0877] Output: Collected gaze and emotion data.

[0878] User:

[0879] Users go about their normal practice or game routines and are aware that the device is collecting data.

[0880] Input: Practice and game action data.

[0881] Output: Data recorded by the Visual Data Storage Device and Emotion Engine.

[0882] Step 3:

[0883] Data Transfer

[0884] Device:

[0885] The collected data is sent to a server via wireless communication at regular intervals. For example, it can be set to send data at every half-time of a match.

[0886] Input: Collected data (gaze data and emotion data).

[0887] Output: The data sent to the server.

[0888] server:

[0889] Prepare to receive data and receive the data sent.

[0890] Input: Submitted gaze and emotion data.

[0891] Output: Stores the received data.

[0892] Step 4:

[0893] Data storage and preprocessing

[0894] server:

[0895] The received gaze and emotion data is stored in a database, after which the data is cleaned and denoised. During this pre-processing phase, the data is converted into an analyzable format.

[0896] Input: Incoming data (gaze data and emotion data).

[0897] Output: Preprocessed data.

[0898] Step 5:

[0899] Data analysis

[0900] server:

[0901] The data analysis means analyzes the preprocessed data, identifying focus points and eye movement patterns from the gaze data and emotional states from the emotion data, and generates gaze maps and heat maps.

[0902] Input: Preprocessed data.

[0903] Output: gaze maps and heat maps.

[0904] Data analysis methods:

[0905] Using AI technology, the data is analysed to identify focus points, eye movement patterns and emotional states.

[0906] Input: Preprocessed data.

[0907] Output: Analysis results (gaze maps, heat maps).

[0908] Step 6:

[0909] Advice Generation

[0910] server:

[0911] Based on the analysis results, the advice generation means generates personalized advice, proposing technical guidance and practice methods suited to each user.

[0912] Input: Analysis results (gaze map, heat map).

[0913] Output: Personalized advice.

[0914] Step 7:

[0915] Providing advice

[0916] server:

[0917] The generated advice is sent to the device.

[0918] Enter: personalized advice.

[0919] Output: Advice sent.

[0920] Device:

[0921] Display the advice to the user, allowing them to view more information through an interactive application.

[0922] Input: Advice sent by the server.

[0923] Output: The displayed advice.

[0924] User:

[0925] Use the advice provided to prepare for the next practice or game, and then wear the device again to prepare for the next data collection phase.

[0926] Input: The advice shown.

[0927] Output: Improvement actions based on the advice.

[0928] Through these steps, the system uses gaze data and emotion data to provide users with appropriate advice, enhancing both technical improvement and emotional control.

[0929] (Application example 2)

[0930] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0931] Conventional methods for analyzing customer behavior in brick-and-mortar stores do not adequately collect and analyze gaze data and emotional data, making it difficult to accurately grasp customers' true interests and emotional states. This makes it difficult to find specific improvement measures for optimizing store layout and maximizing the effectiveness of promotions. Furthermore, insufficient real-time data collection and analysis makes it difficult to respond immediately.

[0932] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0933] In this invention, the server includes means for receiving and storing gaze data and emotion data, means for analyzing the gaze data and emotion data to identify focus points and emotional states, and means for providing advice to users and visualizing customer behavior and emotions, thereby enabling real-time analysis of customer behavior and emotional states and provision of specific advice for effective store management.

[0934] A "visual data storage device" is a device worn by the customer that records the direction of their gaze and movements of their viewpoint in real time.

[0935] "Gaze data" refers to information about the direction of a customer's gaze and the position of the point of gaze recorded by a visual data storage device.

[0936] "Emotional data" refers to information about a customer's emotional state, such as facial expressions, tone of voice, and heart rate, collected by a visual data storage device.

[0937] A "server" is a computer system that receives, stores, and manages gaze data and emotion data.

[0938] "Data analysis means" refers to software and algorithms that run on a server and have the function of analyzing gaze data and emotion data to identify focus points and emotional states.

[0939] The "advice generating means" is software for providing specific advice to the user based on the analysis results obtained by the data analyzing means.

[0940] A "gaze map" is a visual map that shows which parts of a product customers are focusing on based on analyzed gaze data.

[0941] An "emotion map" is a visual map that shows a customer's emotional state based on analyzed emotional data.

[0942] A "heat map" is a map that visually represents gaze data and emotion data, showing gaze time, emotion intensity, etc., using shades of color.

[0943] The present invention relates to a system for analyzing customer behavior and emotions in a physical store. The system includes a visual data storage device worn by a customer, a server that receives and stores the data, a data analysis means that analyzes the data, and an advice generation means that provides advice to a user.

[0944] Key elements of the system

[0945] 1. Visual Data Storage Device

[0946] The visual data storage device is a device worn by the customer that records the direction of gaze and movement of the viewpoint in real time. This device has a built-in camera and sensors and collects gaze data and emotional data. Emotional data can be obtained from facial expressions, voice tone, heart rate, etc.

[0947] 2. Server

[0948] The server is a computer system that receives, stores, and manages gaze data and emotion data. The server uses multiple databases to store individual customer data and provides computing resources to run the analysis means and advice generation means.

[0949] 3. Data Analysis Methods

[0950] The data analysis means is software and algorithms running on a server that analyzes gaze data and emotion data to identify focus points and emotional states. The analysis results are visualized as gaze maps, emotion maps, or heat maps.

[0951] 4. Advice Generation Methods

[0952] The advice generator is software that provides specific advice to store staff and marketing teams based on the analyzed gaze data and emotion data, enabling them to improve store layout and optimize promotions.

[0953] Explanation of program processing

[0954] The gaze data and emotion data collected by the visual data storage device are transmitted to a server via wireless communication. The server stores the received data and performs preprocessing, which includes data cleaning and noise removal. The preprocessed data is then analyzed by a data analysis means to identify focus points and emotional states. The analysis results, visualized as gaze maps and emotion maps, are passed to an advice generation means.

[0955] Based on the analysis results, the advice generator identifies which areas and products customers are most interested in, and in which situations their emotions are stirred, and suggests specific areas for improvement. This advice can be viewed in real time by store staff and marketing teams, enabling prompt action.

[0956] Specific examples

[0957] For example, in a department store, staff wearing smart glasses observe customer behavior. They collect gaze and emotion data when customers enter a specific area and look at specific products. The server analyzes this data and displays gaze and emotion maps showing which products customers are interested in and which areas they feel stressed or excited in. Based on the analysis results, the system provides specific advice for improving the next promotion or product placement.

[0958] Prompt Sentence Examples

[0959] "Detect customer gaze points and emotional states from input images and generate gaze maps and emotion maps in real time. The gaze map should identify the products and areas customers are focusing on, and the emotion map should display their emotional states, such as smile, surprise, or excitement. Furthermore, provide specific advice to improve the customer experience based on the data."

[0960] As described above, the present invention aims to optimize store operations and improve customer experience by analyzing customer behavior and emotions in detail in brick-and-mortar stores, enabling quick responses on-site and strategic decisions based on data.

[0961] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0962] Step 1:

[0963] Data Collection Phase

[0964] The terminal (visual data storage device) records the customer's gaze data and emotional data (facial expressions, voice tone, heart rate, etc.) in real time. The input is real-time visual and physiological data, and the output is collected gaze data and emotional data. This data is periodically transmitted to the server via wireless communication. In other words, the terminal collects data and prepares it to be transmitted to the server.

[0965] Step 2:

[0966] Data Receiving Phase

[0967] The server receives and stores the gaze data and emotion data sent from the device. The input is the gaze data and emotion data sent from the device, and the output is the data stored on the server. Specifically, the server records the data sent in a database and prepares for the next analysis phase.

[0968] Step 3:

[0969] Data Preprocessing Phase

[0970] The server preprocesses the received gaze and emotion data. The input is the stored gaze and emotion data, and the output is cleaned and denoised data. This preprocessing includes data cleaning and denoising, such as imputing missing values ​​and correcting anomalous data, and converting the data into an analyzable format.

[0971] Step 4:

[0972] Data analysis phase

[0973] The server analyzes the preprocessed gaze data and emotion data using data analysis means. The input is the preprocessed gaze data and emotion data, and the output is the analysis results visualized as a gaze map, emotion map, or heat map. Specifically, the AI ​​identifies gaze focus points and emotional states and processes them into a format that can be displayed visually.

[0974] Step 5:

[0975] Advice Generation Phase

[0976] The server generates specific advice using the advice generation means based on the analysis results obtained by the data analysis means. The input is the visualized analysis results, and the output is personalized advice. Specifically, it generates improvement suggestions for store layout and promotions based on which areas and products customers are interested in and in which situations they are emotionally moved.

[0977] Step 6:

[0978] Advice Providing Phase

[0979] The terminal receives the advice sent from the server and displays it to the user. The input is the generated advice, and the output is the advice presented to the user. Specifically, the terminal interactively displays the advice content and provides an interface that allows the user to check detailed information.

[0980] Step 7:

[0981] Feedback gathering phase

[0982] Based on the provided advice, the user makes changes to the store layout or improves customer service methods, and evaluates the effectiveness. The input is the specific actions taken based on the advice, and the output is feedback data on the effectiveness of the implemented improvements. Specifically, the visual data storage device is used again to collect gaze data and emotion data, and the areas for improvement are confirmed and evaluated based on the new data.

[0983] Through the above processing steps, it is possible to perform detailed analysis of customer behavior and emotional states in physical stores, and based on that, it is possible to effectively manage stores and improve customer experience.

[0984] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0985] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0986] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0987] [Third embodiment]

[0988] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0989] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0990] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0991] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0992] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0993] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0994] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0995] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0996] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0997] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0998] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0999] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1000] The present invention is embodied as a system including a visual data storage device worn by top athletes and coaches, data analysis means, and advice generation means. This allows users to learn the perspectives and awareness of top athletes and use this knowledge to improve their skills. Specific details of the embodiments of the present invention are described below.

[1001] Key elements of the system

[1002] 1. Visual Data Storage Device

[1003] The visual data storage device is a device worn by top athletes and coaches that records eye movements and focus points in real time. The device is equipped with a camera and sensors, and the acquired gaze data is sent to a server via wireless communication.

[1004] 2. Server

[1005] The server receives, stores, and manages the gaze data sent from the visual data storage device. The server has a database and has the function of individually storing data for each user. Furthermore, the server provides the computing resources to execute the analysis means and the advice generation means.

[1006] 3. Data Analysis Methods

[1007] The data analysis tool is software and algorithms that run on a server. It analyzes gaze data to identify focus points, eye movement patterns, etc. This data is visualized as gaze maps and heat maps and is used to generate advice later.

[1008] 4. Advice Generation Methods

[1009] The advice generator is software that provides personalized advice to users based on the analysis results. It suggests specific areas for improvement and practice methods based on the analyzed gaze data.

[1010] Program processing flow

[1011] 1. Data Collection Phase

[1012] server:

[1013] The server works with the visual data storage device to prepare for data collection. Specifically, the server registers the device ID and user information of the visual data storage device in a database and sets up data reception.

[1014] Device:

[1015] Visual data storage devices are worn by top athletes and coaches to record their eye movements and focus points in real time. The wearer's gaze data is stored on the device and periodically sent to a server.

[1016] User:

[1017] Before practice or a game, the visual data storage device is attached and checked for proper fit. After confirming that the device is attached correctly, data collection begins.

[1018] 2. Data analysis phase

[1019] server:

[1020] The server stores the received gaze data and performs pre-processing, which includes cleaning and formatting the data to convert it into a format that can be analyzed by the data analysis tool.

[1021] Data analysis methods:

[1022] Gaze data is analyzed to identify focus points and eye movement patterns, and the analysis results are visualized as gaze maps and heat maps.

[1023] User:

[1024] Check the analysis results to understand where top athletes and coaches focus their attention. Visually check specific eye movements as gaze maps and heat maps.

[1025] 3. Advice Phase

[1026] server:

[1027] Based on the results of the data analysis, the AI ​​generates personalized advice, which is then sent from the server to the device.

[1028] Device:

[1029] The received advice is displayed to the user, who can then interactively review the advice within the application and receive further information.

[1030] User:

[1031] The player then practices or plays a match based on the advice provided. The visual data storage device is then worn again to collect gaze data and check the progress of the player's improvement. This creates a feedback loop, enabling gradual improvement of skills.

[1032] Specific examples

[1033] For example, if you were to wear a visual data storage device to collect gaze data during a soccer game:

[1034] Data Collection Phase

[1035] Server: Registers the device ID of the visual data storage device and configures data reception.

[1036] Device: The glasses record eye movements in real time during the game and periodically send the data to a server.

[1037] User: Put on the glasses before the match, check the fit, and then record gaze data during the match.

[1038] Data analysis phase

[1039] Server: Cleans the received gaze data and passes it to the data analysis means.

[1040] Data analysis method: AI analyzes gaze data and displays it as a gaze map or heat map.

[1041] Users: Check the analytics results to understand what they focused on during the match.

[1042] Advice Providing Phase

[1043] Server: The AI ​​generates advice based on the analysis results and sends it to the device.

[1044] Device: Display advice within your app so users can find out more information.

[1045] User: Follow the advice and then perform your next practice or match, collecting gaze data again to see any improvements.

[1046] In this way, the present invention provides an effective system for users to study the gaze data of top athletes and coaches and improve their skills.

[1047] The processing flow will be explained below.

[1048] Step 1:

[1049] Server: The server starts the system that links with the visual data storage device and prepares for data transmission. The server registers the device ID and user information of the visual data storage device in the database and sets up data reception.

[1050] Step 2:

[1051] Device: Power on the visual data storage device, check the network connection, pair it with the corresponding mobile app, and verify that the device is working properly.

[1052] Step 3:

[1053] User: Wears the visual data storage device before training or a match. Checks the fit and secures it in place. Presses the Start Session button to begin data collection.

[1054] Step 4:

[1055] Device: The glasses record eye movements in real time and capture gaze data, which is temporarily stored in a buffer on the device and sent to the server at set intervals.

[1056] Step 5:

[1057] Server: Receives gaze data sent from the device periodically and accumulates it in a database. The data is saved with a timestamp for later analysis.

[1058] Step 6:

[1059] Server: Preprocesses the received gaze data, cleaning and denoising the data, and optionally formatting the data into an analyzable format.

[1060] Step 7:

[1061] Data analysis means (server): Analyzes gaze data using an AI model. Specifically, it analyzes gaze movement patterns, detection of attention points, reaction times, etc. The analysis results are visualized as gaze maps and heat maps.

[1062] Step 8:

[1063] Server: Analyzed data is stored on the server and prepared for transmission to the device as needed.

[1064] Step 9:

[1065] Device: Once the analysis results are sent from the server, they are presented to the user. They are displayed as gaze maps or heat maps within the application, allowing the user to visually confirm the results.

[1066] Step 10:

[1067] User: Check the analysis results through the application, specifically understanding where top coaches and athletes are looking and when they move their eyes.

[1068] Step 11:

[1069] Advice generator (server): The AI ​​model generates personalized advice based on the analysis results. This advice includes specific areas for improvement and practice.

[1070] Step 12:

[1071] Server: Sends generated advice to the device, formatted in a user-friendly format.

[1072] Step 13:

[1073] Terminal: Display the received advice to the user, interactively within the application, allowing the user to view more information if needed.

[1074] Step 14:

[1075] User: Based on the advice provided, the user will train and compete, and in the process will use the visual data storage device again to collect data for further improvement.

[1076] Step 15:

[1077] Server: Re-analyzes the data collected and evaluates the user's progress. Always provides feedback based on the latest data.

[1078] Through this series of processes, users can learn in detail about the gaze and awareness of top athletes and coaches, and use this knowledge to improve their own skills.

[1079] Example 1

[1080] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1081] In conventional sports instruction and training, there is a lack of means to collect gaze data from top athletes and coaches in real time and provide specific feedback based on that data. This makes it difficult for individual athletes to easily obtain specific advice on how to improve their skills. In addition, there is a lack of methods to visually represent and present the analysis results in an easy-to-understand manner.

[1082] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1083] In this invention, the server includes a visual data storage device worn by top athletes and coaches, an information processing device that receives and stores gaze data recorded by the visual data storage device, a preprocessing device that cleans the gaze data and passes it to data analysis means, a data analysis means that analyzes the gaze data and identifies focus points and eye movement patterns, a means for visualizing the analysis results as a gaze map or heat map, an advice generation means that provides advice to users based on the analysis results, and a display means that displays the advice on the user's terminal. This allows users to learn the perspectives and awareness of top athletes and use it to improve their own skills.

[1084] A "visual data storage device" is a device worn by top athletes and coaches to record eye movements and focus points in real time.

[1085] The "information processing device" is a device that receives, stores, and manages the gaze data transmitted from the visual data storage device.

[1086] The "pre-processing device" is a device that cleans the received gaze data, removes noise, complements missing data, and passes the data to the data analysis means.

[1087] "Data analysis means" refers to software and algorithms for analyzing gaze data and identifying focus points and eye movement patterns.

[1088] A "gaze map" is a map that visually represents gaze movements based on gaze data.

[1089] A "heat map" is a visual representation that shows gaze points using shades of color.

[1090] An "advice generator" is software and algorithms for providing personalized advice to a user based on the analysis results.

[1091] The "display means" is a device or function for displaying the generated advice and analysis results on the user's terminal.

[1092] A "generative AI model" is an artificial intelligence model that analyzes gaze data and generates gaze maps or heat maps.

[1093] The present invention is embodied as a system including a visual data storage device worn by top athletes and coaches, data analysis means, and advice generation means. This allows users to learn the perspectives and awareness of top athletes and use this knowledge to improve their skills. Specific details of the embodiments of the present invention are described below.

[1094] Key elements of the system

[1095] 1. Visual Data Storage Device

[1096] The visual data storage device is a device worn by top athletes and coaches. It is equipped with cameras and sensors that record eye movements and focus points in real time. The collected gaze data is transmitted to a server via wireless communication.

[1097] 2. Server

[1098] The server receives, stores, and manages gaze data transmitted from the visual data storage device. In particular, the server incorporates a pre-processing unit that cleans the gaze data and passes it to the data analysis unit. The server also provides the computing resources to run the data analysis unit and the advice generation unit.

[1099] 3. Data Analysis Methods

[1100] The data analysis tool is software and algorithms that run on a server. It analyzes gaze data to identify focus points and eye movement patterns. This data is visualized as gaze maps and heat maps and used to generate advice later.

[1101] 4. Advice Generation Methods

[1102] The advice generator is software that provides personalized advice to users based on the analysis results of the data analysis means. It uses a generative AI model to suggest specific areas for improvement and practice methods.

[1103] 5. Display means

[1104] The display means is a device or function for displaying the generated advice and analysis results on the user's terminal. It has an interactive display function, allowing the user to check detailed information.

[1105] Example of operation flow

[1106] For example, consider the case where a visual data storage device is worn by a player during a soccer match to collect gaze data. In this case, the server registers the device ID of the visual data storage device and configures data reception. During the match, the device's camera records the player's eye movements in real time and periodically sends this to the server. The server cleans the received gaze data and passes it to the data analysis means. This means analyzes the gaze data using a generative AI model and generates a gaze map or heat map. The server sends advice generated by the AI ​​to the device, which displays the advice within the application. The player then performs the next practice or match based on the advice provided.

[1107] Prompt Sentence Examples

[1108] "Please explain the algorithm that analyzes focus points and eye movement patterns using gaze data from athletes wearing visual data storage devices."

[1109] With this structure, the system of the present invention allows users to learn the gaze data of top athletes and coaches and use it to improve their own skills.

[1110] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1111] Step 1: Prepare for data collection

[1112] Server Action:

[1113] The server registers the device ID and user information of the visual data storage device in a database. The input is the device ID (e.g., ID12345) of the visual data storage device and the user information, and outputs a database entry for linking and managing the device ID and user information. The server then performs operations to prepare for communication with the visual data storage device.

[1114] Step 2: Start collecting data

[1115] Terminal handling:

[1116] The device is worn by top athletes and coaches. The device's built-in camera and sensors record eye movements and focus points in real time. The input is the athlete's eye movements, and the output is the recorded gaze data. The device operates to collect gaze data in real time.

[1117] Step 3: Send data

[1118] Terminal handling:

[1119] The terminal periodically collects and transmits the gaze data to the server via wireless communication. The input is the gaze data recorded on the terminal, and the output is the gaze data transmitted to the server. The terminal controls the timing of data transmission and performs operations to maintain data consistency.

[1120] Step 4: Save Data

[1121] Server Action:

[1122] The server stores the received gaze data in a database. The input is the gaze data received from the device, and the output is the gaze data stored in the database. It performs operations to verify the accuracy of the data (for example, checking data reception and verifying it before saving).

[1123] Step 5: Data Cleaning

[1124] Server Action:

[1125] The server cleans the stored gaze data, removes noise, and fills in missing data. The input is the stored gaze data, and the output is the cleaned gaze data. It applies noise filtering and data filling algorithms.

[1126] Step 6: Data analysis

[1127] Processing data analysis methods:

[1128] The data analysis means analyzes the cleaned gaze data and identifies focus points and eye movement patterns. The input is the cleaned gaze data and the output is the analysis results. The data analysis means performs operations to execute a focus point detection algorithm and a movement pattern analysis algorithm.

[1129] Step 7: Generate gaze maps and heat maps

[1130] Processing data analysis methods:

[1131] The data analysis means visualizes the analysis results as gaze maps and heat maps. The input is the analysis results, and the output is gaze maps and heat maps. A visualization algorithm is used to graphically display the gaze data.

[1132] Step 8: Advice Generation

[1133] Advice generator processing:

[1134] The advice generation means generates personalized advice for the user based on the analysis results of the gaze map and heat map. The input is the analysis results of the gaze map and heat map, and the output is the generated advice. The system operates to generate personalized feedback using a generative AI model.

[1135] Step 9: Submitting Advice

[1136] Server Action:

[1137] The server sends the generated advice to the terminal. The input is the generated advice and the output is the advice sent to the terminal. It performs actions to ensure the accuracy of the data transmission (e.g., checking the data package and confirming transmission).

[1138] Step 10: Viewing Advice

[1139] Terminal handling:

[1140] The terminal displays the advice received within the application to the user. The input is the advice received from the server, and the output is the advice visually displayed to the user. It provides an interactive display function and performs actions that allow the user to check detailed information.

[1141] Step 11: Act on the advice

[1142] User Action:

[1143] The user then practices or plays a game based on the advice provided. The input is the advice displayed on the device, and the output is the action the user performs. The user then wears the visual data storage device again, collects gaze data during the next game or practice, and performs actions to check for improvement.

[1144] (Application example 1)

[1145] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1146] Currently, there is no real-time advice system to improve the driving skills of autonomous vehicles, so there is a need for autonomous driving systems to provide appropriate feedback on the driving situation by utilizing gaze data from professional drivers. Conventional systems do not sufficiently link the collection and analysis of driving data, making it difficult to improve technology and safety in real time.

[1147] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1148] In this invention, the server includes a means for receiving and storing gaze data, a means for analyzing the gaze data to identify focus points and eye movement patterns, and a means for providing feedback to the autonomous driving AI system based on the analysis results. This makes it possible to improve driving skills in real time using gaze data from professional drivers and provide advice according to driving conditions.

[1149] A "visual data storage device" is a device worn by the pilot to record eye movements and gaze direction.

[1150] A "server" is a central device for receiving, storing, and processing gaze data transmitted from a visual data repository.

[1151] The "data analysis means" is software or algorithm that runs on the server and analyzes the received gaze data to identify focus points and gaze movement patterns.

[1152] The "advice generation means" is software or an algorithm that provides the user with advice for improving their skills based on the analysis results and the gaze data of top pilots.

[1153] "Means for improving autonomous driving technology" refers to a means for supplying analyzed gaze data to the driving AI system of an autonomous vehicle in order to improve driving technology.

[1154] The "real-time advice providing means" is a means for providing appropriate advice for the driving situation in real time.

[1155] A "gaze map" is a map that visually displays gaze movement patterns based on gaze data from professional drivers.

[1156] A "heat map" is a map that visually displays points of focus and the degree of gaze concentration.

[1157] A "driving AI system" is an artificial intelligence system that controls the driving of autonomous vehicles and optimizes driving operations based on external data.

[1158] This invention is implemented as a system that provides a visual data storage device worn by a professional driver, a real-time data analysis means, and appropriate driving advice to improve automated driving technology. The system includes the following main elements:

[1159] Key elements of the system

[1160] 1. Visual Data Storage Device

[1161] These smart glasses are worn by professional drivers and record eye movements and gaze direction in real time, wirelessly transmitting the data to a server.

[1162] 2. Server

[1163] The server receives and stores the gaze data sent from the visual data storage device. It analyzes the data and provides the analysis results to the driving AI system. As a specific example, an Amazon Web Services (AWS) EC2 instance is used.

[1164] 3. Data Analysis Methods

[1165] This is software and algorithms that run on a server to analyze gaze data, identify focus points and gaze movement patterns, and visualize them as gaze maps and heat maps using Python and OpenCV.

[1166] 4. Advice Generation Methods

[1167] It is software or algorithm that provides feedback to autonomous driving AI systems based on analysis results, generating personalized advice using TensorFlow.

[1168] 5. Real-time advice delivery methods

[1169] It is a means of providing appropriate advice in real time on driving situations during automated driving, which will improve automated driving technology and increase driving safety.

[1170] Specific processing explanation of the system

[1171] The server receives gaze data from the visual data storage device and stores it in a database. This data first undergoes a data cleaning process and is formatted for analysis. The gaze data is then analyzed using Python and OpenCV and visualized as gaze maps and heat maps. The analysis results are input into an AI model using TensorFlow to generate personalized advice for the driving AI system.

[1172] Specific examples of processing

[1173] Once the server receives the visual data, it performs data cleaning to remove noise. It is then analyzed using OpenCV to generate a gaze map. For example, it can identify the gaze points of a professional driver when approaching an intersection and provide this to the driving AI system. Based on this data, the generative AI model can provide appropriate driving advice in real time.

[1174] Examples of prompt statements

[1175] Here are some example prompts to input to a generative AI model:

[1176] Based on the gaze data analysis results, identify the points that professional drivers focus on while driving and provide optimal advice and suggestions for improvement for safe driving.

[1177] By using this prompt, the AI ​​model can generate specific advice for improving driving skills based on the analysis results.

[1178] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1179] Step 1:

[1180] Visual data collection

[1181] The user has a professional driver wear smart glasses, which collect real-time gaze data while driving and transmit it to a server via Wi-Fi.

[1182] Input: Professional driver's gaze data

[1183] Output: Gaze data sent to the server

[1184] Step 2:

[1185] Receiving and storing data

[1186] The server receives the gaze data sent from the smart glasses and stores it securely in a database.

[1187] Input: Gaze data sent from smart glasses

[1188] Output: Gaze data stored in a database

[1189] Step 3:

[1190] Data Preprocessing

[1191] The server performs a cleaning process on the received gaze data, which involves removing noise and reformatting the data format.

[1192] Input: Gaze data stored in a database

[1193] Output: Cleaned gaze data

[1194] Step 4:

[1195] Data analysis

[1196] The server analyzes the cleaned gaze data using data analysis tools. It uses OpenCV to generate gaze maps and heat maps. This analysis identifies focus points and gaze movement patterns while driving.

[1197] Input: Cleaned gaze data

[1198] Output: gaze maps and heat maps

[1199] Step 5:

[1200] Advice Generation

[1201] Based on the analysis results, the server uses TensorFlow to generate AI model advice for the driving AI system. It uses prompt sentences to suggest appropriate driving advice and improvements.

[1202] Input: gaze maps and heat maps

[1203] Output: Advice for driving AI systems

[1204] Step 6:

[1205] Real-time advice provided

[1206] The server sends the generated advice to the driving AI system in real time, which then optimizes driving operations based on the advice received, ensuring safe driving.

[1207] Input: Advice for driving AI systems

[1208] Output: Optimized driving operation by the driving AI system

[1209] Through this series of processes, the gaze data of professional drivers is used to improve autonomous driving technology and provide appropriate advice according to the driving situation.

[1210] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1211] The present invention is embodied as a system that combines a visual data storage device worn by top athletes and coaches, data analysis means, and advice generation means with an emotion engine that recognizes the user's emotions. This allows users to learn not only the perspectives and awareness of top athletes, but also their own emotional state, which can be used to improve their skills. Specific details of the embodiments of the present invention are described below.

[1212] Key elements of the system

[1213] 1. Visual Data Storage Device

[1214] The visual data storage device is a device worn by top athletes and coaches that records eye movements and focus points in real time. The device is equipped with a camera and sensors, and the acquired gaze data is sent to a server via wireless communication.

[1215] 2. Emotion Engine

[1216] The emotion engine is a combination of software and hardware that analyzes data such as the user's facial expressions, voice tone, and heart rate to recognize emotions in real time. It is built into the visual data storage device and transmits emotion data along with gaze data to the server.

[1217] 3. Server

[1218] The server receives, stores, and manages the gaze data and emotion data sent from the visual data storage device and emotion engine. The server has a database and has the function of individually storing data for each user. Furthermore, the server provides the computational resources to execute the analysis means and advice generation means.

[1219] 4. Data Analysis Methods

[1220] The data analysis tool is software and algorithms running on a server. It analyzes gaze data and emotion data to identify focus points, eye movement patterns, and emotional states. This data is visualized as gaze maps and heat maps and used to generate advice.

[1221] 5. Advice Generation Methods

[1222] The advice generator is software that provides personalized advice to users based on the analysis results. It suggests specific areas for improvement and practice methods based on the analyzed gaze data and emotion data.

[1223] Program processing flow

[1224] 1. Data Collection Phase

[1225] server:

[1226] The server works with the visual data storage device and emotion engine to prepare for data collection. Specifically, the server registers the device IDs and user information of the visual data storage device and emotion engine in a database and sets up data reception.

[1227] Device:

[1228] Visual data storage devices and emotion engines are worn by top athletes and coaches to record their eye movements, focus points, and emotional states in real time. The wearer's gaze data and emotion data are stored on the device and periodically transmitted to a server.

[1229] User:

[1230] Before practice or a game, the visual data storage device is attached and checked for proper fit. After confirming that the device is attached correctly, data collection begins.

[1231] 2. Data analysis phase

[1232] server:

[1233] The server stores and pre-processes the received gaze and emotion data, including cleaning and denoising the data, and converts it into a format that can be analyzed by the data analysis means.

[1234] Data analysis methods:

[1235] Gaze data and emotional data are analyzed to identify focus points, eye movement patterns, and emotional states. The analysis results are visualized as gaze maps and heat maps.

[1236] User:

[1237] By reviewing the analysis results, you can understand where top athletes and coaches focused and what their emotional state was. Visually check specific eye movements and emotional states as gaze maps and heat maps.

[1238] 3. Advice Phase

[1239] server:

[1240] Based on the results of the data analysis, the AI ​​generates personalized advice, which is then sent from the server to the device.

[1241] Device:

[1242] The received advice is displayed to the user, who can then interactively review the advice within the application and receive further information.

[1243] User:

[1244] The player then practices and plays matches based on the advice provided. The visual data storage device is then worn again to collect gaze and emotional data, and the player's progress is monitored. This creates a feedback loop, enabling gradual improvement in technique and emotional control.

[1245] Specific examples

[1246] For example, if you wear a visual data storage device and an emotion engine to collect gaze data and emotion data during a soccer game:

[1247] Data Collection Phase

[1248] Server: Registers the device IDs of the visual data storage device and emotion engine, and configures data reception.

[1249] Device: The glasses record eye movements and emotional state in real time during the match and periodically transmit the data to a server.

[1250] User: The user puts on the glasses before the match, checks their fit, and then records their gaze and emotion data during the match.

[1251] Data analysis phase

[1252] Server: Cleans the received gaze data and emotion data and passes them to the data analysis means.

[1253] Data analysis method: AI analyzes gaze data and emotion data and displays them as gaze maps and heat maps.

[1254] Users: Review analytics to understand what they focused on and what their emotional state was during a match.

[1255] Advice Providing Phase

[1256] Server: The AI ​​generates advice based on the analysis results and sends it to the device.

[1257] Device: Display advice within your app so users can find out more information.

[1258] User: Based on the advice, conduct the next practice or match, and collect gaze and emotion data again to confirm improvement.

[1259] In this way, the present invention allows users to learn the gaze data and emotion data of top athletes and coaches and use it to improve their own skills and control their emotions.

[1260] The processing flow will be explained below.

[1261] Step 1:

[1262] Server: Registers the device IDs and user information of the visual data storage device and emotion engine in the database, and sets up data reception. Installs the server software on the server and prepares it for operation.

[1263] Step 2:

[1264] Device: Power on the visual data storage device and emotion engine, check the network connection, pair with the corresponding mobile app, and verify that the device is working properly. Calibrate the emotion engine so that it can properly capture data such as the user's facial expressions, tone of voice, and heart rate.

[1265] Step 3:

[1266] User: Before training or a match, wear the visual data storage device and emotion engine and check the fit. After making sure the device is fixed in the correct position, press the start button to begin data collection.

[1267] Step 4:

[1268] Device: The glasses record eye movements and the emotion engine records emotion data in real time, capturing them as gaze data and emotion data. The captured data is temporarily stored in a buffer on the device and sent to the server at set intervals.

[1269] Step 5:

[1270] Server: Receives gaze data and emotion data periodically sent from the device and stores them in a database. The data is saved with a timestamp for later analysis.

[1271] Step 6:

[1272] Server: Preprocesses the received gaze and emotion data, cleaning and denoising the data, and optionally formatting the data to make it more analyzable.

[1273] Step 7:

[1274] Data analysis means (server): Analyzes gaze data and emotional data using an AI model. Specifically, it analyzes gaze movement patterns, detection of attention points, reaction times, and emotional states. The analysis results are visualized as gaze maps and heat maps.

[1275] Step 8:

[1276] Server: The analyzed data is stored on the server and prepared for transmission to the device as needed. The analysis results are formatted in a user-friendly format.

[1277] Step 9:

[1278] Terminal: When the analysis results are sent from the server, they are presented to the user. They are displayed as gaze maps or heat maps, allowing the user to visually confirm the results.

[1279] Step 10:

[1280] Users: View analytics results through the application, specifically to understand where elite athletes and coaches focus their efforts and their emotional state.

[1281] Step 11:

[1282] Advice generation means (server): The AI ​​model generates personalized advice based on the analysis results. This advice includes specific improvements and practice methods related to both gaze data and emotion data.

[1283] Step 12:

[1284] Server: Sends generated advice to the device, formatted in a user-friendly format.

[1285] Step 13:

[1286] Terminal: Display the received advice to the user, interactively within the application, allowing the user to view more information if needed.

[1287] Step 14:

[1288] User: Based on the advice provided, the user trains and competes, again using the visual data storage device and emotion engine in the process to collect data for further improvement.

[1289] Step 15:

[1290] Server: Re-analyzes the newly collected data and evaluates the user's progress. Always provides feedback based on the latest data, and checks the effectiveness of the user's technical improvement and emotional control.

[1291] Through this series of processes, users can learn in detail about the gaze and emotional states of top athletes and coaches, and use this knowledge to improve their own skills and manage their emotions.

[1292] Example 2

[1293] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1294] While conventional sports training systems focus on analyzing the user's visual information and focus points, they lack the technology to provide comprehensive training advice that takes the user's emotional state into account. As a result, feedback on the user's technical improvement is limited, making it difficult to adequately support emotional control and mental growth. Furthermore, real-time data collection and analysis are insufficient, making it difficult for users to receive immediate advice.

[1295] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1296] In this invention, the server includes a visual data storage device worn by top athletes and coaches, an emotion engine that recognizes the user's emotions in real time, means for receiving and saving data recorded by the visual data storage device and emotion engine, means for analyzing the gaze data and emotion data received from the server and identifying focus points, eye movement patterns, and emotional states, and means for providing personalized advice to the user based on the results of the analysis. This enables the user to receive comprehensive advice that takes into account not only visual information but also emotional states, thereby enhancing both technical improvement and emotional control.

[1297] A "visual data storage device" is a device worn by top athletes and coaches that records eye movements and focus points in real time.

[1298] An "emotion engine" is a combination of software and hardware that analyzes data such as a user's facial expressions, voice tone, and heart rate to recognize emotions in real time.

[1299] "Server" refers to a system that receives, stores, and manages data sent from the visual data storage device and emotion engine, and provides the computational resources necessary for analyzing the data and generating advice.

[1300] "Data analysis means" refers to a means for analyzing gaze data and emotion data by software and algorithms executed on a server to identify focus points, eye movement patterns, and emotional states.

[1301] The "advice generation means" is software for providing personalized advice to the user based on the analysis results.

[1302] "Gaze data" refers to data relating to eye movements and focus points recorded by a visual data storage device.

[1303] "Emotion data" refers to data relating to the user's emotional state, such as facial expressions, tone of voice, and heart rate, recognized by the emotion engine.

[1304] A "gaze map" is a visual display of analyzed gaze data, showing focus points and eye movement patterns.

[1305] A "heat map" is a visual representation of the distribution and concentration of analyzed data, showing the density and frequency of data within a specific range.

[1306] "Personalized advice" refers to suggestions for technical instruction and practice methods that are individually generated based on each user's gaze data and emotional data.

[1307] This invention combines a visual data storage device used by top athletes and coaches with an emotion engine that recognizes users' emotions in real time, a server, data analysis means, and advice generation means. This allows users to improve their skills as well as recognize and improve their own emotional state.

[1308] To implement the present invention, the following elements are included:

[1309] 1. Visual Data Storage Device

[1310] The visual data storage device is a device worn by top athletes and coaches that records eye movements and focus points in real time. The device is equipped with a camera and sensors, and the acquired gaze data is transmitted to a server via wireless communication.

[1311] 2. Emotion Engine

[1312] The emotion engine is a combination of software and hardware that analyzes data such as the user's facial expression, voice tone, and heart rate to recognize emotions in real time. This engine works in conjunction with a visual data storage device and transmits emotion data along with gaze data to a server.

[1313] 3. Server

[1314] The server receives, stores, and manages the gaze data and emotion data sent from the visual data storage device and emotion engine. The server has a database and has the function of individually storing data for each user. Furthermore, the server provides computing resources for executing the data analysis means and advice generation means.

[1315] 4. Data Analysis Methods

[1316] The data analysis tool is software and algorithms running on a server that analyzes gaze data and emotion data to identify focus points, eye movement patterns, and emotional states. This data is visualized as gaze maps and heat maps, which are then used to generate advice.

[1317] 5. Advice Generation Methods

[1318] The advice generator is software that provides personalized advice to users based on the analysis results. It suggests specific areas for improvement and practice methods based on the analyzed gaze data and emotion data.

[1319] Specific hardware and software used in the present invention include the following:

[1320] Visual data storage devices (e.g., cameras, sensors)

[1321] Emotion engine (e.g., facial expression analysis software, heart rate sensor)

[1322] Server (database, analysis algorithms, computational resources)

[1323] Data analysis methods (analysis algorithms using AI technology)

[1324] Advice generation means (personalized advice software)

[1325] Specific examples

[1326] For example, when gaze data and emotion data are collected by wearing a visual data storage device and an emotion engine during a soccer match, the system operates as follows.

[1327] Data Collection Phase

[1328] Server: Registers the device IDs of the visual data storage device and emotion engine, and configures data reception.

[1329] Terminal: The visual data storage device and emotion engine record eye movements and emotional states during the match in real time and periodically transmit the data to the server.

[1330] User: The user wears the device before the match, checks the fit, and then records gaze data and emotion data during the match.

[1331] Data analysis phase

[1332] Server: Cleans the received gaze data and emotion data and passes them to the data analysis means.

[1333] Data analysis method: AI analyzes gaze data and emotion data and displays them as gaze maps and heat maps.

[1334] Users: Review analytics to understand what they focused on and what their emotional state was during a match.

[1335] Advice Providing Phase

[1336] Server: The AI ​​generates advice based on the analysis results and sends it to the device.

[1337] Device: Display advice within your app so users can find out more information.

[1338] User: Based on the advice, conduct the next practice or match, and collect gaze and emotion data again to confirm improvement.

[1339] Prompt Sentence Examples

[1340] "Please collect the focus points and emotional state of top soccer players during a match and visualize them as gaze maps and heat maps. Also, please provide specific advice for improving skills based on that data."

[1341] In the above-described manner, the present invention enables a user to learn the gaze data and emotion data of top athletes and coaches, and to use this data to improve their own skills and control their emotions.

[1342] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1343] Step 1:

[1344] Initial Device Setup

[1345] Device:

[1346] The visual data storage device and emotion engine are powered on and initial calibration is performed, which involves adjusting the camera position and checking the accuracy of the sensors.

[1347] Input: Initial data from deployed cameras and sensors.

[1348] Output: Adjusted device state data after calibration.

[1349] User:

[1350] The wearer checks that the device is fitted correctly and checks that there are no problems with comfort or operation when wearing it.

[1351] Input: Physical location of the device.

[1352] Output: Confirmation data for successfully attached devices.

[1353] Step 2:

[1354] Data collection

[1355] Device:

[1356] Data collection begins. The visual data storage device records eye movements and focus points in real time, and the emotion engine monitors the user's facial expressions and heart rate, generating gaze data and emotion data.

[1357] Input: Real-time eye movement data and heart rate data.

[1358] Output: Collected gaze and emotion data.

[1359] User:

[1360] Users go about their normal practice or game routines and are aware that the device is collecting data.

[1361] Input: Practice and game action data.

[1362] Output: Data recorded by the Visual Data Storage Device and Emotion Engine.

[1363] Step 3:

[1364] Data Transfer

[1365] Device:

[1366] The collected data is sent to a server via wireless communication at regular intervals. For example, it can be set to send data at every half-time of a match.

[1367] Input: Collected data (gaze data and emotion data).

[1368] Output: The data sent to the server.

[1369] server:

[1370] Prepare to receive data and receive the data sent.

[1371] Input: Submitted gaze and emotion data.

[1372] Output: Stores the received data.

[1373] Step 4:

[1374] Data storage and preprocessing

[1375] server:

[1376] The received gaze and emotion data is stored in a database, after which the data is cleaned and denoised. During this pre-processing phase, the data is converted into an analyzable format.

[1377] Input: Incoming data (gaze data and emotion data).

[1378] Output: Preprocessed data.

[1379] Step 5:

[1380] Data analysis

[1381] server:

[1382] The data analysis means analyzes the preprocessed data, identifying focus points and eye movement patterns from the gaze data and emotional states from the emotion data, and generates gaze maps and heat maps.

[1383] Input: Preprocessed data.

[1384] Output: gaze maps and heat maps.

[1385] Data analysis methods:

[1386] Using AI technology, the data is analysed to identify focus points, eye movement patterns and emotional states.

[1387] Input: Preprocessed data.

[1388] Output: Analysis results (gaze maps, heat maps).

[1389] Step 6:

[1390] Advice Generation

[1391] server:

[1392] Based on the analysis results, the advice generation means generates personalized advice, proposing technical guidance and practice methods suited to each user.

[1393] Input: Analysis results (gaze map, heat map).

[1394] Output: Personalized advice.

[1395] Step 7:

[1396] Providing advice

[1397] server:

[1398] The generated advice is sent to the device.

[1399] Enter: personalized advice.

[1400] Output: Advice sent.

[1401] Device:

[1402] Display the advice to the user, allowing them to view more information through an interactive application.

[1403] Input: Advice sent by the server.

[1404] Output: The displayed advice.

[1405] User:

[1406] Use the advice provided to prepare for the next practice or game, and then wear the device again to prepare for the next data collection phase.

[1407] Input: The advice shown.

[1408] Output: Improvement actions based on the advice.

[1409] Through these steps, the system uses gaze data and emotion data to provide users with appropriate advice, enhancing both technical improvement and emotional control.

[1410] (Application example 2)

[1411] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1412] Conventional methods for analyzing customer behavior in brick-and-mortar stores do not adequately collect and analyze gaze data and emotional data, making it difficult to accurately grasp customers' true interests and emotional states. This makes it difficult to find specific improvement measures for optimizing store layout and maximizing the effectiveness of promotions. Furthermore, insufficient real-time data collection and analysis makes it difficult to respond immediately.

[1413] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1414] In this invention, the server includes means for receiving and storing gaze data and emotion data, means for analyzing the gaze data and emotion data to identify focus points and emotional states, and means for providing advice to users and visualizing customer behavior and emotions, thereby enabling real-time analysis of customer behavior and emotional states and provision of specific advice for effective store management.

[1415] A "visual data storage device" is a device worn by the customer that records the direction of their gaze and movements of their viewpoint in real time.

[1416] "Gaze data" refers to information about the direction of a customer's gaze and the position of the point of gaze recorded by a visual data storage device.

[1417] "Emotional data" refers to information about a customer's emotional state, such as facial expressions, tone of voice, and heart rate, collected by a visual data storage device.

[1418] A "server" is a computer system that receives, stores, and manages gaze data and emotion data.

[1419] "Data analysis means" refers to software and algorithms that run on a server and have the function of analyzing gaze data and emotion data to identify focus points and emotional states.

[1420] The "advice generating means" is software for providing specific advice to the user based on the analysis results obtained by the data analyzing means.

[1421] A "gaze map" is a visual map that shows which parts of a product customers are focusing on based on analyzed gaze data.

[1422] An "emotion map" is a visual map that shows a customer's emotional state based on analyzed emotional data.

[1423] A "heat map" is a map that visually represents gaze data and emotion data, showing gaze time, emotion intensity, etc., using shades of color.

[1424] The present invention relates to a system for analyzing customer behavior and emotions in a physical store. The system includes a visual data storage device worn by a customer, a server that receives and stores the data, a data analysis means that analyzes the data, and an advice generation means that provides advice to a user.

[1425] Key elements of the system

[1426] 1. Visual Data Storage Device

[1427] The visual data storage device is a device worn by the customer that records the direction of gaze and movement of the viewpoint in real time. This device has a built-in camera and sensors and collects gaze data and emotional data. Emotional data can be obtained from facial expressions, voice tone, heart rate, etc.

[1428] 2. Server

[1429] The server is a computer system that receives, stores, and manages gaze data and emotion data. The server uses multiple databases to store individual customer data and provides computing resources to run the analysis means and advice generation means.

[1430] 3. Data Analysis Methods

[1431] The data analysis means is software and algorithms running on a server that analyzes gaze data and emotion data to identify focus points and emotional states. The analysis results are visualized as gaze maps, emotion maps, or heat maps.

[1432] 4. Advice Generation Methods

[1433] The advice generator is software that provides specific advice to store staff and marketing teams based on the analyzed gaze data and emotion data, enabling them to improve store layout and optimize promotions.

[1434] Explanation of program processing

[1435] The gaze data and emotion data collected by the visual data storage device are transmitted to a server via wireless communication. The server stores the received data and performs preprocessing, which includes data cleaning and noise removal. The preprocessed data is then analyzed by a data analysis means to identify focus points and emotional states. The analysis results, visualized as gaze maps and emotion maps, are passed to an advice generation means.

[1436] Based on the analysis results, the advice generator identifies which areas and products customers are most interested in, and in which situations their emotions are stirred, and suggests specific areas for improvement. This advice can be viewed in real time by store staff and marketing teams, enabling prompt action.

[1437] Specific examples

[1438] For example, in a department store, staff wearing smart glasses observe customer behavior. They collect gaze and emotion data when customers enter a specific area and look at specific products. The server analyzes this data and displays gaze and emotion maps showing which products customers are interested in and which areas they feel stressed or excited in. Based on the analysis results, the system provides specific advice for improving the next promotion or product placement.

[1439] Prompt Sentence Examples

[1440] "Detect customer gaze points and emotional states from input images and generate gaze maps and emotion maps in real time. The gaze map should identify the products and areas customers are focusing on, and the emotion map should display their emotional states, such as smile, surprise, or excitement. Furthermore, provide specific advice to improve the customer experience based on the data."

[1441] As described above, the present invention aims to optimize store operations and improve customer experience by analyzing customer behavior and emotions in detail in brick-and-mortar stores, enabling quick responses on-site and strategic decisions based on data.

[1442] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1443] Step 1:

[1444] Data Collection Phase

[1445] The terminal (visual data storage device) records the customer's gaze data and emotional data (facial expressions, voice tone, heart rate, etc.) in real time. The input is real-time visual and physiological data, and the output is collected gaze data and emotional data. This data is periodically transmitted to the server via wireless communication. In other words, the terminal collects data and prepares it to be transmitted to the server.

[1446] Step 2:

[1447] Data Receiving Phase

[1448] The server receives and stores the gaze data and emotion data sent from the device. The input is the gaze data and emotion data sent from the device, and the output is the data stored on the server. Specifically, the server records the data sent in a database and prepares for the next analysis phase.

[1449] Step 3:

[1450] Data Preprocessing Phase

[1451] The server preprocesses the received gaze and emotion data. The input is the stored gaze and emotion data, and the output is cleaned and denoised data. This preprocessing includes data cleaning and denoising, such as imputing missing values ​​and correcting anomalous data, and converting the data into an analyzable format.

[1452] Step 4:

[1453] Data analysis phase

[1454] The server analyzes the preprocessed gaze data and emotion data using data analysis means. The input is the preprocessed gaze data and emotion data, and the output is the analysis results visualized as a gaze map, emotion map, or heat map. Specifically, the AI ​​identifies gaze focus points and emotional states and processes them into a format that can be displayed visually.

[1455] Step 5:

[1456] Advice Generation Phase

[1457] The server generates specific advice using the advice generation means based on the analysis results obtained by the data analysis means. The input is the visualized analysis results, and the output is personalized advice. Specifically, it generates improvement suggestions for store layout and promotions based on which areas and products customers are interested in and in which situations they are emotionally moved.

[1458] Step 6:

[1459] Advice Providing Phase

[1460] The terminal receives the advice sent from the server and displays it to the user. The input is the generated advice, and the output is the advice presented to the user. Specifically, the terminal interactively displays the advice content and provides an interface that allows the user to check detailed information.

[1461] Step 7:

[1462] Feedback gathering phase

[1463] Based on the provided advice, the user makes changes to the store layout or improves customer service methods, and evaluates the effectiveness. The input is the specific actions taken based on the advice, and the output is feedback data on the effectiveness of the implemented improvements. Specifically, the visual data storage device is used again to collect gaze data and emotion data, and the areas for improvement are confirmed and evaluated based on the new data.

[1464] Through the above processing steps, it is possible to perform detailed analysis of customer behavior and emotional states in physical stores, and based on that, it is possible to effectively manage stores and improve customer experience.

[1465] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1466] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1467] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1468] [Fourth embodiment]

[1469] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1470] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1471] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1472] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1473] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1474] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1475] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1476] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1477] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1478] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1479] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1480] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1481] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1482] The present invention is embodied as a system including a visual data storage device worn by top athletes and coaches, data analysis means, and advice generation means. This allows users to learn the perspectives and awareness of top athletes and use this knowledge to improve their skills. Specific details of the embodiments of the present invention are described below.

[1483] Key elements of the system

[1484] 1. Visual Data Storage Device

[1485] The visual data storage device is a device worn by top athletes and coaches that records eye movements and focus points in real time. The device is equipped with a camera and sensors, and the acquired gaze data is sent to a server via wireless communication.

[1486] 2. Server

[1487] The server receives, stores, and manages the gaze data sent from the visual data storage device. The server has a database and has the function of individually storing data for each user. Furthermore, the server provides the computing resources to execute the analysis means and the advice generation means.

[1488] 3. Data Analysis Methods

[1489] The data analysis tool is software and algorithms that run on a server. It analyzes gaze data to identify focus points, eye movement patterns, etc. This data is visualized as gaze maps and heat maps and is used to generate advice later.

[1490] 4. Advice Generation Methods

[1491] The advice generator is software that provides personalized advice to users based on the analysis results. It suggests specific areas for improvement and practice methods based on the analyzed gaze data.

[1492] Program processing flow

[1493] 1. Data Collection Phase

[1494] server:

[1495] The server works with the visual data storage device to prepare for data collection. Specifically, the server registers the device ID and user information of the visual data storage device in a database and sets up data reception.

[1496] Device:

[1497] Visual data storage devices are worn by top athletes and coaches to record their eye movements and focus points in real time. The wearer's gaze data is stored on the device and periodically sent to a server.

[1498] User:

[1499] Before practice or a game, the visual data storage device is attached and checked for proper fit. After confirming that the device is attached correctly, data collection begins.

[1500] 2. Data analysis phase

[1501] server:

[1502] The server stores the received gaze data and performs pre-processing, which includes cleaning and formatting the data to convert it into a format that can be analyzed by the data analysis tool.

[1503] Data analysis methods:

[1504] Gaze data is analyzed to identify focus points and eye movement patterns, and the analysis results are visualized as gaze maps and heat maps.

[1505] User:

[1506] Check the analysis results to understand where top athletes and coaches focus their attention. Visually check specific eye movements as gaze maps and heat maps.

[1507] 3. Advice Phase

[1508] server:

[1509] Based on the results of the data analysis, the AI ​​generates personalized advice, which is then sent from the server to the device.

[1510] Device:

[1511] The received advice is displayed to the user, who can then interactively review the advice within the application and receive further information.

[1512] User:

[1513] The player then practices or plays a match based on the advice provided. The visual data storage device is then worn again to collect gaze data and check the progress of the player's improvement. This creates a feedback loop, enabling gradual improvement of skills.

[1514] Specific examples

[1515] For example, if you were to wear a visual data storage device to collect gaze data during a soccer game:

[1516] Data Collection Phase

[1517] Server: Registers the device ID of the visual data storage device and configures data reception.

[1518] Device: The glasses record eye movements in real time during the game and periodically send the data to a server.

[1519] User: Put on the glasses before the match, check the fit, and then record gaze data during the match.

[1520] Data analysis phase

[1521] Server: Cleans the received gaze data and passes it to the data analysis means.

[1522] Data analysis method: AI analyzes gaze data and displays it as a gaze map or heat map.

[1523] Users: Check the analytics results to understand what they focused on during the match.

[1524] Advice Providing Phase

[1525] Server: The AI ​​generates advice based on the analysis results and sends it to the device.

[1526] Device: Display advice within your app so users can find out more information.

[1527] User: Follow the advice and then perform your next practice or match, collecting gaze data again to see any improvements.

[1528] In this way, the present invention provides an effective system for users to study the gaze data of top athletes and coaches and improve their skills.

[1529] The processing flow will be explained below.

[1530] Step 1:

[1531] Server: The server starts the system that links with the visual data storage device and prepares for data transmission. The server registers the device ID and user information of the visual data storage device in the database and sets up data reception.

[1532] Step 2:

[1533] Device: Power on the visual data storage device, check the network connection, pair it with the corresponding mobile app, and verify that the device is working properly.

[1534] Step 3:

[1535] User: Wears the visual data storage device before training or a match. Checks the fit and secures it in place. Presses the Start Session button to begin data collection.

[1536] Step 4:

[1537] Device: The glasses record eye movements in real time and capture gaze data, which is temporarily stored in a buffer on the device and sent to the server at set intervals.

[1538] Step 5:

[1539] Server: Receives gaze data sent from the device periodically and accumulates it in a database. The data is saved with a timestamp for later analysis.

[1540] Step 6:

[1541] Server: Preprocesses the received gaze data, cleaning and denoising the data, and optionally formatting the data into an analyzable format.

[1542] Step 7:

[1543] Data analysis means (server): Analyzes gaze data using an AI model. Specifically, it analyzes gaze movement patterns, detection of attention points, reaction times, etc. The analysis results are visualized as gaze maps and heat maps.

[1544] Step 8:

[1545] Server: Analyzed data is stored on the server and prepared for transmission to the device as needed.

[1546] Step 9:

[1547] Device: Once the analysis results are sent from the server, they are presented to the user. They are displayed as gaze maps or heat maps within the application, allowing the user to visually confirm the results.

[1548] Step 10:

[1549] User: Check the analysis results through the application, specifically understanding where top coaches and athletes are looking and when they move their eyes.

[1550] Step 11:

[1551] Advice generator (server): The AI ​​model generates personalized advice based on the analysis results. This advice includes specific areas for improvement and practice.

[1552] Step 12:

[1553] Server: Sends generated advice to the device, formatted in a user-friendly format.

[1554] Step 13:

[1555] Terminal: Display the received advice to the user, interactively within the application, allowing the user to view more information if needed.

[1556] Step 14:

[1557] User: Based on the advice provided, the user will train and compete, and in the process will use the visual data storage device again to collect data for further improvement.

[1558] Step 15:

[1559] Server: Re-analyzes the data collected and evaluates the user's progress. Always provides feedback based on the latest data.

[1560] Through this series of processes, users can learn in detail about the gaze and awareness of top athletes and coaches, and use this knowledge to improve their own skills.

[1561] Example 1

[1562] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1563] In conventional sports instruction and training, there is a lack of means to collect gaze data from top athletes and coaches in real time and provide specific feedback based on that data. This makes it difficult for individual athletes to easily obtain specific advice on how to improve their skills. In addition, there is a lack of methods to visually represent and present the analysis results in an easy-to-understand manner.

[1564] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1565] In this invention, the server includes a visual data storage device worn by top athletes and coaches, an information processing device that receives and stores gaze data recorded by the visual data storage device, a preprocessing device that cleans the gaze data and passes it to data analysis means, a data analysis means that analyzes the gaze data and identifies focus points and eye movement patterns, a means for visualizing the analysis results as a gaze map or heat map, an advice generation means that provides advice to users based on the analysis results, and a display means that displays the advice on the user's terminal. This allows users to learn the perspectives and awareness of top athletes and use it to improve their own skills.

[1566] A "visual data storage device" is a device worn by top athletes and coaches to record eye movements and focus points in real time.

[1567] The "information processing device" is a device that receives, stores, and manages the gaze data transmitted from the visual data storage device.

[1568] The "pre-processing device" is a device that cleans the received gaze data, removes noise, complements missing data, and passes the data to the data analysis means.

[1569] "Data analysis means" refers to software and algorithms for analyzing gaze data and identifying focus points and eye movement patterns.

[1570] A "gaze map" is a map that visually represents gaze movements based on gaze data.

[1571] A "heat map" is a visual representation that shows gaze points using shades of color.

[1572] An "advice generator" is software and algorithms for providing personalized advice to a user based on the analysis results.

[1573] The "display means" is a device or function for displaying the generated advice and analysis results on the user's terminal.

[1574] A "generative AI model" is an artificial intelligence model that analyzes gaze data and generates gaze maps or heat maps.

[1575] The present invention is embodied as a system including a visual data storage device worn by top athletes and coaches, data analysis means, and advice generation means. This allows users to learn the perspectives and awareness of top athletes and use this knowledge to improve their skills. Specific details of the embodiments of the present invention are described below.

[1576] Key elements of the system

[1577] 1. Visual Data Storage Device

[1578] The visual data storage device is a device worn by top athletes and coaches. It is equipped with cameras and sensors that record eye movements and focus points in real time. The collected gaze data is transmitted to a server via wireless communication.

[1579] 2. Server

[1580] The server receives, stores, and manages gaze data transmitted from the visual data storage device. In particular, the server incorporates a pre-processing unit that cleans the gaze data and passes it to the data analysis unit. The server also provides the computing resources to run the data analysis unit and the advice generation unit.

[1581] 3. Data Analysis Methods

[1582] The data analysis tool is software and algorithms that run on a server. It analyzes gaze data to identify focus points and eye movement patterns. This data is visualized as gaze maps and heat maps and used to generate advice later.

[1583] 4. Advice Generation Methods

[1584] The advice generator is software that provides personalized advice to users based on the analysis results of the data analysis means. It uses a generative AI model to suggest specific areas for improvement and practice methods.

[1585] 5. Display means

[1586] The display means is a device or function for displaying the generated advice and analysis results on the user's terminal. It has an interactive display function, allowing the user to check detailed information.

[1587] Example of operation flow

[1588] For example, consider the case where a visual data storage device is worn by a player during a soccer match to collect gaze data. In this case, the server registers the device ID of the visual data storage device and configures data reception. During the match, the device's camera records the player's eye movements in real time and periodically sends this to the server. The server cleans the received gaze data and passes it to the data analysis means. This means analyzes the gaze data using a generative AI model and generates a gaze map or heat map. The server sends advice generated by the AI ​​to the device, which displays the advice within the application. The player then performs the next practice or match based on the advice provided.

[1589] Prompt Sentence Examples

[1590] "Please explain the algorithm that analyzes focus points and eye movement patterns using gaze data from athletes wearing visual data storage devices."

[1591] With this structure, the system of the present invention allows users to learn the gaze data of top athletes and coaches and use it to improve their own skills.

[1592] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1593] Step 1: Prepare for data collection

[1594] Server Action:

[1595] The server registers the device ID and user information of the visual data storage device in a database. The input is the device ID (e.g., ID12345) of the visual data storage device and the user information, and outputs a database entry for linking and managing the device ID and user information. The server then performs operations to prepare for communication with the visual data storage device.

[1596] Step 2: Start collecting data

[1597] Terminal handling:

[1598] The device is worn by top athletes and coaches. The device's built-in camera and sensors record eye movements and focus points in real time. The input is the athlete's eye movements, and the output is the recorded gaze data. The device operates to collect gaze data in real time.

[1599] Step 3: Send data

[1600] Terminal handling:

[1601] The terminal periodically collects and transmits the gaze data to the server via wireless communication. The input is the gaze data recorded on the terminal, and the output is the gaze data transmitted to the server. The terminal controls the timing of data transmission and performs operations to maintain data consistency.

[1602] Step 4: Save Data

[1603] Server Action:

[1604] The server stores the received gaze data in a database. The input is the gaze data received from the device, and the output is the gaze data stored in the database. It performs operations to verify the accuracy of the data (for example, checking data reception and verifying it before saving).

[1605] Step 5: Data Cleaning

[1606] Server Action:

[1607] The server cleans the stored gaze data, removes noise, and fills in missing data. The input is the stored gaze data, and the output is the cleaned gaze data. It applies noise filtering and data filling algorithms.

[1608] Step 6: Data analysis

[1609] Processing data analysis methods:

[1610] The data analysis means analyzes the cleaned gaze data and identifies focus points and eye movement patterns. The input is the cleaned gaze data and the output is the analysis results. The data analysis means performs operations to execute a focus point detection algorithm and a movement pattern analysis algorithm.

[1611] Step 7: Generate gaze maps and heat maps

[1612] Processing data analysis methods:

[1613] The data analysis means visualizes the analysis results as gaze maps and heat maps. The input is the analysis results, and the output is gaze maps and heat maps. A visualization algorithm is used to graphically display the gaze data.

[1614] Step 8: Advice Generation

[1615] Advice generator processing:

[1616] The advice generation means generates personalized advice for the user based on the analysis results of the gaze map and heat map. The input is the analysis results of the gaze map and heat map, and the output is the generated advice. The system operates to generate personalized feedback using a generative AI model.

[1617] Step 9: Submitting Advice

[1618] Server Action:

[1619] The server sends the generated advice to the terminal. The input is the generated advice and the output is the advice sent to the terminal. It performs actions to ensure the accuracy of the data transmission (e.g., checking the data package and confirming transmission).

[1620] Step 10: Viewing Advice

[1621] Terminal handling:

[1622] The terminal displays the advice received within the application to the user. The input is the advice received from the server, and the output is the advice visually displayed to the user. It provides an interactive display function and performs actions that allow the user to check detailed information.

[1623] Step 11: Act on the advice

[1624] User Action:

[1625] The user then practices or plays a game based on the advice provided. The input is the advice displayed on the device, and the output is the action the user performs. The user then wears the visual data storage device again, collects gaze data during the next game or practice, and performs actions to check for improvement.

[1626] (Application example 1)

[1627] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1628] Currently, there is no real-time advice system to improve the driving skills of autonomous vehicles, so there is a need for autonomous driving systems to provide appropriate feedback on the driving situation by utilizing gaze data from professional drivers. Conventional systems do not sufficiently link the collection and analysis of driving data, making it difficult to improve technology and safety in real time.

[1629] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1630] In this invention, the server includes a means for receiving and storing gaze data, a means for analyzing the gaze data to identify focus points and eye movement patterns, and a means for providing feedback to the autonomous driving AI system based on the analysis results. This makes it possible to improve driving skills in real time using gaze data from professional drivers and provide advice according to driving conditions.

[1631] A "visual data storage device" is a device worn by the pilot to record eye movements and gaze direction.

[1632] A "server" is a central device for receiving, storing, and processing gaze data transmitted from a visual data repository.

[1633] The "data analysis means" is software or algorithm that runs on the server and analyzes the received gaze data to identify focus points and gaze movement patterns.

[1634] The "advice generation means" is software or an algorithm that provides the user with advice for improving their skills based on the analysis results and the gaze data of top pilots.

[1635] "Means for improving autonomous driving technology" refers to a means for supplying analyzed gaze data to the driving AI system of an autonomous vehicle in order to improve driving technology.

[1636] The "real-time advice providing means" is a means for providing appropriate advice for the driving situation in real time.

[1637] A "gaze map" is a map that visually displays gaze movement patterns based on gaze data from professional drivers.

[1638] A "heat map" is a map that visually displays points of focus and the degree of gaze concentration.

[1639] A "driving AI system" is an artificial intelligence system that controls the driving of autonomous vehicles and optimizes driving operations based on external data.

[1640] This invention is implemented as a system that provides a visual data storage device worn by a professional driver, a real-time data analysis means, and appropriate driving advice to improve automated driving technology. The system includes the following main elements:

[1641] Key elements of the system

[1642] 1. Visual Data Storage Device

[1643] These smart glasses are worn by professional drivers and record eye movements and gaze direction in real time, wirelessly transmitting the data to a server.

[1644] 2. Server

[1645] The server receives and stores the gaze data sent from the visual data storage device. It analyzes the data and provides the analysis results to the driving AI system. As a specific example, an Amazon Web Services (AWS) EC2 instance is used.

[1646] 3. Data Analysis Methods

[1647] This is software and algorithms that run on a server to analyze gaze data, identify focus points and gaze movement patterns, and visualize them as gaze maps and heat maps using Python and OpenCV.

[1648] 4. Advice Generation Methods

[1649] It is software or algorithm that provides feedback to autonomous driving AI systems based on analysis results, generating personalized advice using TensorFlow.

[1650] 5. Real-time advice delivery methods

[1651] It is a means of providing appropriate advice in real time on driving situations during automated driving, which will improve automated driving technology and increase driving safety.

[1652] Specific processing explanation of the system

[1653] The server receives gaze data from the visual data storage device and stores it in a database. This data first undergoes a data cleaning process and is formatted for analysis. The gaze data is then analyzed using Python and OpenCV and visualized as gaze maps and heat maps. The analysis results are input into an AI model using TensorFlow to generate personalized advice for the driving AI system.

[1654] Specific examples of processing

[1655] Once the server receives the visual data, it performs data cleaning to remove noise. It is then analyzed using OpenCV to generate a gaze map. For example, it can identify the gaze points of a professional driver when approaching an intersection and provide this to the driving AI system. Based on this data, the generative AI model can provide appropriate driving advice in real time.

[1656] Examples of prompt statements

[1657] Here are some example prompts to input to a generative AI model:

[1658] Based on the gaze data analysis results, identify the points that professional drivers focus on while driving and provide optimal advice and suggestions for improvement for safe driving.

[1659] By using this prompt, the AI ​​model can generate specific advice for improving driving skills based on the analysis results.

[1660] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1661] Step 1:

[1662] Visual data collection

[1663] The user has a professional driver wear smart glasses, which collect real-time gaze data while driving and transmit it to a server via Wi-Fi.

[1664] Input: Professional driver's gaze data

[1665] Output: Gaze data sent to the server

[1666] Step 2:

[1667] Receiving and storing data

[1668] The server receives the gaze data sent from the smart glasses and stores it securely in a database.

[1669] Input: Gaze data sent from smart glasses

[1670] Output: Gaze data stored in a database

[1671] Step 3:

[1672] Data Preprocessing

[1673] The server performs a cleaning process on the received gaze data, which involves removing noise and reformatting the data format.

[1674] Input: Gaze data stored in a database

[1675] Output: Cleaned gaze data

[1676] Step 4:

[1677] Data analysis

[1678] The server analyzes the cleaned gaze data using data analysis tools. It uses OpenCV to generate gaze maps and heat maps. This analysis identifies focus points and gaze movement patterns while driving.

[1679] Input: Cleaned gaze data

[1680] Output: gaze maps and heat maps

[1681] Step 5:

[1682] Advice Generation

[1683] Based on the analysis results, the server uses TensorFlow to generate AI model advice for the driving AI system. It uses prompt sentences to suggest appropriate driving advice and improvements.

[1684] Input: gaze maps and heat maps

[1685] Output: Advice for driving AI systems

[1686] Step 6:

[1687] Real-time advice provided

[1688] The server sends the generated advice to the driving AI system in real time, which then optimizes driving operations based on the advice received, ensuring safe driving.

[1689] Input: Advice for driving AI systems

[1690] Output: Optimized driving operation by the driving AI system

[1691] Through this series of processes, the gaze data of professional drivers is used to improve autonomous driving technology and provide appropriate advice according to the driving situation.

[1692] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1693] The present invention is embodied as a system that combines a visual data storage device worn by top athletes and coaches, data analysis means, and advice generation means with an emotion engine that recognizes the user's emotions. This allows users to learn not only the perspectives and awareness of top athletes, but also their own emotional state, which can be used to improve their skills. Specific details of the embodiments of the present invention are described below.

[1694] Key elements of the system

[1695] 1. Visual Data Storage Device

[1696] The visual data storage device is a device worn by top athletes and coaches that records eye movements and focus points in real time. The device is equipped with a camera and sensors, and the acquired gaze data is sent to a server via wireless communication.

[1697] 2. Emotion Engine

[1698] The emotion engine is a combination of software and hardware that analyzes data such as the user's facial expressions, voice tone, and heart rate to recognize emotions in real time. It is built into the visual data storage device and transmits emotion data along with gaze data to the server.

[1699] 3. Server

[1700] The server receives, stores, and manages the gaze data and emotion data sent from the visual data storage device and emotion engine. The server has a database and has the function of individually storing data for each user. Furthermore, the server provides the computational resources to execute the analysis means and advice generation means.

[1701] 4. Data Analysis Methods

[1702] The data analysis tool is software and algorithms running on a server. It analyzes gaze data and emotion data to identify focus points, eye movement patterns, and emotional states. This data is visualized as gaze maps and heat maps and used to generate advice.

[1703] 5. Advice Generation Methods

[1704] The advice generator is software that provides personalized advice to users based on the analysis results. It suggests specific areas for improvement and practice methods based on the analyzed gaze data and emotion data.

[1705] Program processing flow

[1706] 1. Data Collection Phase

[1707] server:

[1708] The server works with the visual data storage device and emotion engine to prepare for data collection. Specifically, the server registers the device IDs and user information of the visual data storage device and emotion engine in a database and sets up data reception.

[1709] Device:

[1710] Visual data storage devices and emotion engines are worn by top athletes and coaches to record their eye movements, focus points, and emotional states in real time. The wearer's gaze data and emotion data are stored on the device and periodically transmitted to a server.

[1711] User:

[1712] Before practice or a game, the visual data storage device is attached and checked for proper fit. After confirming that the device is attached correctly, data collection begins.

[1713] 2. Data analysis phase

[1714] server:

[1715] The server stores and pre-processes the received gaze and emotion data, including cleaning and denoising the data, and converts it into a format that can be analyzed by the data analysis means.

[1716] Data analysis methods:

[1717] Gaze data and emotional data are analyzed to identify focus points, eye movement patterns, and emotional states. The analysis results are visualized as gaze maps and heat maps.

[1718] User:

[1719] By reviewing the analysis results, you can understand where top athletes and coaches focused and what their emotional state was. Visually check specific eye movements and emotional states as gaze maps and heat maps.

[1720] 3. Advice Phase

[1721] server:

[1722] Based on the results of the data analysis, the AI ​​generates personalized advice, which is then sent from the server to the device.

[1723] Device:

[1724] The received advice is displayed to the user, who can then interactively review the advice within the application and receive further information.

[1725] User:

[1726] The player then practices and plays matches based on the advice provided. The visual data storage device is then worn again to collect gaze and emotional data, and the player's progress is monitored. This creates a feedback loop, enabling gradual improvement in technique and emotional control.

[1727] Specific examples

[1728] For example, if you wear a visual data storage device and an emotion engine to collect gaze data and emotion data during a soccer game:

[1729] Data Collection Phase

[1730] Server: Registers the device IDs of the visual data storage device and emotion engine, and configures data reception.

[1731] Device: The glasses record eye movements and emotional state in real time during the match and periodically transmit the data to a server.

[1732] User: The user puts on the glasses before the match, checks their fit, and then records their gaze and emotion data during the match.

[1733] Data analysis phase

[1734] Server: Cleans the received gaze data and emotion data and passes them to the data analysis means.

[1735] Data analysis method: AI analyzes gaze data and emotion data and displays them as gaze maps and heat maps.

[1736] Users: Review analytics to understand what they focused on and what their emotional state was during a match.

[1737] Advice Providing Phase

[1738] Server: The AI ​​generates advice based on the analysis results and sends it to the device.

[1739] Device: Display advice within your app so users can find out more information.

[1740] User: Based on the advice, conduct the next practice or match, and collect gaze and emotion data again to confirm improvement.

[1741] In this way, the present invention allows users to learn the gaze data and emotion data of top athletes and coaches and use it to improve their own skills and control their emotions.

[1742] The processing flow will be explained below.

[1743] Step 1:

[1744] Server: Registers the device IDs and user information of the visual data storage device and emotion engine in the database, and sets up data reception. Installs the server software on the server and prepares it for operation.

[1745] Step 2:

[1746] Device: Power on the visual data storage device and emotion engine, check the network connection, pair with the corresponding mobile app, and verify that the device is working properly. Calibrate the emotion engine so that it can properly capture data such as the user's facial expressions, tone of voice, and heart rate.

[1747] Step 3:

[1748] User: Before training or a match, wear the visual data storage device and emotion engine and check the fit. After making sure the device is fixed in the correct position, press the start button to begin data collection.

[1749] Step 4:

[1750] Device: The glasses record eye movements and the emotion engine records emotion data in real time, capturing them as gaze data and emotion data. The captured data is temporarily stored in a buffer on the device and sent to the server at set intervals.

[1751] Step 5:

[1752] Server: Receives gaze data and emotion data periodically sent from the device and stores them in a database. The data is saved with a timestamp for later analysis.

[1753] Step 6:

[1754] Server: Preprocesses the received gaze and emotion data, cleaning and denoising the data, and optionally formatting the data to make it more analyzable.

[1755] Step 7:

[1756] Data analysis means (server): Analyzes gaze data and emotional data using an AI model. Specifically, it analyzes gaze movement patterns, detection of attention points, reaction times, and emotional states. The analysis results are visualized as gaze maps and heat maps.

[1757] Step 8:

[1758] Server: The analyzed data is stored on the server and prepared for transmission to the device as needed. The analysis results are formatted in a user-friendly format.

[1759] Step 9:

[1760] Terminal: When the analysis results are sent from the server, they are presented to the user. They are displayed as gaze maps or heat maps, allowing the user to visually confirm the results.

[1761] Step 10:

[1762] Users: View analytics results through the application, specifically to understand where elite athletes and coaches focus their efforts and their emotional state.

[1763] Step 11:

[1764] Advice generation means (server): The AI ​​model generates personalized advice based on the analysis results. This advice includes specific improvements and practice methods related to both gaze data and emotion data.

[1765] Step 12:

[1766] Server: Sends generated advice to the device, formatted in a user-friendly format.

[1767] Step 13:

[1768] Terminal: Display the received advice to the user, interactively within the application, allowing the user to view more information if needed.

[1769] Step 14:

[1770] User: Based on the advice provided, the user trains and competes, again using the visual data storage device and emotion engine in the process to collect data for further improvement.

[1771] Step 15:

[1772] Server: Re-analyzes the newly collected data and evaluates the user's progress. Always provides feedback based on the latest data, and checks the effectiveness of the user's technical improvement and emotional control.

[1773] Through this series of processes, users can learn in detail about the gaze and emotional states of top athletes and coaches, and use this knowledge to improve their own skills and manage their emotions.

[1774] Example 2

[1775] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1776] While conventional sports training systems focus on analyzing the user's visual information and focus points, they lack the technology to provide comprehensive training advice that takes the user's emotional state into account. As a result, feedback on the user's technical improvement is limited, making it difficult to adequately support emotional control and mental growth. Furthermore, real-time data collection and analysis are insufficient, making it difficult for users to receive immediate advice.

[1777] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1778] In this invention, the server includes a visual data storage device worn by top athletes and coaches, an emotion engine that recognizes the user's emotions in real time, means for receiving and saving data recorded by the visual data storage device and emotion engine, means for analyzing the gaze data and emotion data received from the server and identifying focus points, eye movement patterns, and emotional states, and means for providing personalized advice to the user based on the results of the analysis. This enables the user to receive comprehensive advice that takes into account not only visual information but also emotional states, thereby enhancing both technical improvement and emotional control.

[1779] A "visual data storage device" is a device worn by top athletes and coaches that records eye movements and focus points in real time.

[1780] An "emotion engine" is a combination of software and hardware that analyzes data such as a user's facial expressions, voice tone, and heart rate to recognize emotions in real time.

[1781] "Server" refers to a system that receives, stores, and manages data sent from the visual data storage device and emotion engine, and provides the computational resources necessary for analyzing the data and generating advice.

[1782] "Data analysis means" refers to a means for analyzing gaze data and emotion data by software and algorithms executed on a server to identify focus points, eye movement patterns, and emotional states.

[1783] The "advice generation means" is software for providing personalized advice to the user based on the analysis results.

[1784] "Gaze data" refers to data relating to eye movements and focus points recorded by a visual data storage device.

[1785] "Emotion data" refers to data relating to the user's emotional state, such as facial expressions, tone of voice, and heart rate, recognized by the emotion engine.

[1786] A "gaze map" is a visual display of analyzed gaze data, showing focus points and eye movement patterns.

[1787] A "heat map" is a visual representation of the distribution and concentration of analyzed data, showing the density and frequency of data within a specific range.

[1788] "Personalized advice" refers to suggestions for technical instruction and practice methods that are individually generated based on each user's gaze data and emotional data.

[1789] This invention combines a visual data storage device used by top athletes and coaches with an emotion engine that recognizes users' emotions in real time, a server, data analysis means, and advice generation means. This allows users to improve their skills as well as recognize and improve their own emotional state.

[1790] To implement the present invention, the following elements are included:

[1791] 1. Visual Data Storage Device

[1792] The visual data storage device is a device worn by top athletes and coaches that records eye movements and focus points in real time. The device is equipped with a camera and sensors, and the acquired gaze data is transmitted to a server via wireless communication.

[1793] 2. Emotion Engine

[1794] The emotion engine is a combination of software and hardware that analyzes data such as the user's facial expression, voice tone, and heart rate to recognize emotions in real time. This engine works in conjunction with a visual data storage device and transmits emotion data along with gaze data to a server.

[1795] 3. Server

[1796] The server receives, stores, and manages the gaze data and emotion data sent from the visual data storage device and emotion engine. The server has a database and has the function of individually storing data for each user. Furthermore, the server provides computing resources for executing the data analysis means and advice generation means.

[1797] 4. Data Analysis Methods

[1798] The data analysis tool is software and algorithms running on a server that analyzes gaze data and emotion data to identify focus points, eye movement patterns, and emotional states. This data is visualized as gaze maps and heat maps, which are then used to generate advice.

[1799] 5. Advice Generation Methods

[1800] The advice generator is software that provides personalized advice to users based on the analysis results. It suggests specific areas for improvement and practice methods based on the analyzed gaze data and emotion data.

[1801] Specific hardware and software used in the present invention include the following:

[1802] Visual data storage devices (e.g., cameras, sensors)

[1803] Emotion engine (e.g., facial expression analysis software, heart rate sensor)

[1804] Server (database, analysis algorithms, computational resources)

[1805] Data analysis methods (analysis algorithms using AI technology)

[1806] Advice generation means (personalized advice software)

[1807] Specific examples

[1808] For example, when gaze data and emotion data are collected by wearing a visual data storage device and an emotion engine during a soccer match, the system operates as follows.

[1809] Data Collection Phase

[1810] Server: Registers the device IDs of the visual data storage device and emotion engine, and configures data reception.

[1811] Terminal: The visual data storage device and emotion engine record eye movements and emotional states during the match in real time and periodically transmit the data to the server.

[1812] User: The user wears the device before the match, checks the fit, and then records gaze data and emotion data during the match.

[1813] Data analysis phase

[1814] Server: Cleans the received gaze data and emotion data and passes them to the data analysis means.

[1815] Data analysis method: AI analyzes gaze data and emotion data and displays them as gaze maps and heat maps.

[1816] Users: Review analytics to understand what they focused on and what their emotional state was during a match.

[1817] Advice Providing Phase

[1818] Server: The AI ​​generates advice based on the analysis results and sends it to the device.

[1819] Device: Display advice within your app so users can find out more information.

[1820] User: Based on the advice, conduct the next practice or match, and collect gaze and emotion data again to confirm improvement.

[1821] Prompt Sentence Examples

[1822] "Please collect the focus points and emotional state of top soccer players during a match and visualize them as gaze maps and heat maps. Also, please provide specific advice for improving skills based on that data."

[1823] In the above-described manner, the present invention enables a user to learn the gaze data and emotion data of top athletes and coaches, and to use this data to improve their own skills and control their emotions.

[1824] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1825] Step 1:

[1826] Initial Device Setup

[1827] Device:

[1828] The visual data storage device and emotion engine are powered on and initial calibration is performed, which involves adjusting the camera position and checking the accuracy of the sensors.

[1829] Input: Initial data from deployed cameras and sensors.

[1830] Output: Adjusted device state data after calibration.

[1831] User:

[1832] The wearer checks that the device is fitted correctly and checks that there are no problems with comfort or operation when wearing it.

[1833] Input: Physical location of the device.

[1834] Output: Confirmation data for successfully attached devices.

[1835] Step 2:

[1836] Data collection

[1837] Device:

[1838] Data collection begins. The visual data storage device records eye movements and focus points in real time, and the emotion engine monitors the user's facial expressions and heart rate, generating gaze data and emotion data.

[1839] Input: Real-time eye movement data and heart rate data.

[1840] Output: Collected gaze and emotion data.

[1841] User:

[1842] Users go about their normal practice or game routines and are aware that the device is collecting data.

[1843] Input: Practice and game action data.

[1844] Output: Data recorded by the Visual Data Storage Device and Emotion Engine.

[1845] Step 3:

[1846] Data Transfer

[1847] Device:

[1848] The collected data is sent to a server via wireless communication at regular intervals. For example, it can be set to send data at every half-time of a match.

[1849] Input: Collected data (gaze data and emotion data).

[1850] Output: The data sent to the server.

[1851] server:

[1852] Prepare to receive data and receive the data sent.

[1853] Input: Submitted gaze and emotion data.

[1854] Output: Stores the received data.

[1855] Step 4:

[1856] Data storage and preprocessing

[1857] server:

[1858] The received gaze and emotion data is stored in a database, after which the data is cleaned and denoised. During this pre-processing phase, the data is converted into an analyzable format.

[1859] Input: Incoming data (gaze data and emotion data).

[1860] Output: Preprocessed data.

[1861] Step 5:

[1862] Data analysis

[1863] server:

[1864] The data analysis means analyzes the preprocessed data, identifying focus points and eye movement patterns from the gaze data and emotional states from the emotion data, and generates gaze maps and heat maps.

[1865] Input: Preprocessed data.

[1866] Output: gaze maps and heat maps.

[1867] Data analysis methods:

[1868] Using AI technology, the data is analysed to identify focus points, eye movement patterns and emotional states.

[1869] Input: Preprocessed data.

[1870] Output: Analysis results (gaze maps, heat maps).

[1871] Step 6:

[1872] Advice Generation

[1873] server:

[1874] Based on the analysis results, the advice generation means generates personalized advice, proposing technical guidance and practice methods suited to each user.

[1875] Input: Analysis results (gaze map, heat map).

[1876] Output: Personalized advice.

[1877] Step 7:

[1878] Providing advice

[1879] server:

[1880] The generated advice is sent to the device.

[1881] Enter: personalized advice.

[1882] Output: Advice sent.

[1883] Device:

[1884] Display the advice to the user, allowing them to view more information through an interactive application.

[1885] Input: Advice sent by the server.

[1886] Output: The displayed advice.

[1887] User:

[1888] Use the advice provided to prepare for the next practice or game, and then wear the device again to prepare for the next data collection phase.

[1889] Input: The advice shown.

[1890] Output: Improvement actions based on the advice.

[1891] Through these steps, the system uses gaze data and emotion data to provide users with appropriate advice, enhancing both technical improvement and emotional control.

[1892] (Application example 2)

[1893] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1894] Conventional methods for analyzing customer behavior in brick-and-mortar stores do not adequately collect and analyze gaze data and emotional data, making it difficult to accurately grasp customers' true interests and emotional states. This makes it difficult to find specific improvement measures for optimizing store layout and maximizing the effectiveness of promotions. Furthermore, insufficient real-time data collection and analysis makes it difficult to respond immediately.

[1895] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1896] In this invention, the server includes means for receiving and storing gaze data and emotion data, means for analyzing the gaze data and emotion data to identify focus points and emotional states, and means for providing advice to users and visualizing customer behavior and emotions, thereby enabling real-time analysis of customer behavior and emotional states and provision of specific advice for effective store management.

[1897] A "visual data storage device" is a device worn by the customer that records the direction of their gaze and movements of their viewpoint in real time.

[1898] "Gaze data" refers to information about the direction of a customer's gaze and the position of the point of gaze recorded by a visual data storage device.

[1899] "Emotional data" refers to information about a customer's emotional state, such as facial expressions, tone of voice, and heart rate, collected by a visual data storage device.

[1900] A "server" is a computer system that receives, stores, and manages gaze data and emotion data.

[1901] "Data analysis means" refers to software and algorithms that run on a server and have the function of analyzing gaze data and emotion data to identify focus points and emotional states.

[1902] The "advice generating means" is software for providing specific advice to the user based on the analysis results obtained by the data analyzing means.

[1903] A "gaze map" is a visual map that shows which parts of a product customers are focusing on based on analyzed gaze data.

[1904] An "emotion map" is a visual map that shows a customer's emotional state based on analyzed emotional data.

[1905] A "heat map" is a map that visually represents gaze data and emotion data, showing gaze time, emotion intensity, etc., using shades of color.

[1906] The present invention relates to a system for analyzing customer behavior and emotions in a physical store. The system includes a visual data storage device worn by a customer, a server that receives and stores the data, a data analysis means that analyzes the data, and an advice generation means that provides advice to a user.

[1907] Key elements of the system

[1908] 1. Visual Data Storage Device

[1909] The visual data storage device is a device worn by the customer that records the direction of gaze and movement of the viewpoint in real time. This device has a built-in camera and sensors and collects gaze data and emotional data. Emotional data can be obtained from facial expressions, voice tone, heart rate, etc.

[1910] 2. Server

[1911] The server is a computer system that receives, stores, and manages gaze data and emotion data. The server uses multiple databases to store individual customer data and provides computing resources to run the analysis means and advice generation means.

[1912] 3. Data Analysis Methods

[1913] The data analysis means is software and algorithms running on a server that analyzes gaze data and emotion data to identify focus points and emotional states. The analysis results are visualized as gaze maps, emotion maps, or heat maps.

[1914] 4. Advice Generation Methods

[1915] The advice generator is software that provides specific advice to store staff and marketing teams based on the analyzed gaze data and emotion data, enabling them to improve store layout and optimize promotions.

[1916] Explanation of program processing

[1917] The gaze data and emotion data collected by the visual data storage device are transmitted to a server via wireless communication. The server stores the received data and performs preprocessing, which includes data cleaning and noise removal. The preprocessed data is then analyzed by a data analysis means to identify focus points and emotional states. The analysis results, visualized as gaze maps and emotion maps, are passed to an advice generation means.

[1918] Based on the analysis results, the advice generator identifies which areas and products customers are most interested in, and in which situations their emotions are stirred, and suggests specific areas for improvement. This advice can be viewed in real time by store staff and marketing teams, enabling prompt action.

[1919] Specific examples

[1920] For example, in a department store, staff wearing smart glasses observe customer behavior. They collect gaze and emotion data when customers enter a specific area and look at specific products. The server analyzes this data and displays gaze and emotion maps showing which products customers are interested in and which areas they feel stressed or excited in. Based on the analysis results, the system provides specific advice for improving the next promotion or product placement.

[1921] Prompt Sentence Examples

[1922] "Detect customer gaze points and emotional states from input images and generate gaze maps and emotion maps in real time. The gaze map should identify the products and areas customers are focusing on, and the emotion map should display their emotional states, such as smile, surprise, or excitement. Furthermore, provide specific advice to improve the customer experience based on the data."

[1923] As described above, the present invention aims to optimize store operations and improve customer experience by analyzing customer behavior and emotions in detail in brick-and-mortar stores, enabling quick responses on-site and strategic decisions based on data.

[1924] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1925] Step 1:

[1926] Data Collection Phase

[1927] The terminal (visual data storage device) records the customer's gaze data and emotional data (facial expressions, voice tone, heart rate, etc.) in real time. The input is real-time visual and physiological data, and the output is collected gaze data and emotional data. This data is periodically transmitted to the server via wireless communication. In other words, the terminal collects data and prepares it to be transmitted to the server.

[1928] Step 2:

[1929] Data Receiving Phase

[1930] The server receives and stores the gaze data and emotion data sent from the device. The input is the gaze data and emotion data sent from the device, and the output is the data stored on the server. Specifically, the server records the data sent in a database and prepares for the next analysis phase.

[1931] Step 3:

[1932] Data Preprocessing Phase

[1933] The server preprocesses the received gaze and emotion data. The input is the stored gaze and emotion data, and the output is cleaned and denoised data. This preprocessing includes data cleaning and denoising, such as imputing missing values ​​and correcting anomalous data, and converting the data into an analyzable format.

[1934] Step 4:

[1935] Data analysis phase

[1936] The server analyzes the preprocessed gaze data and emotion data using data analysis means. The input is the preprocessed gaze data and emotion data, and the output is the analysis results visualized as a gaze map, emotion map, or heat map. Specifically, the AI ​​identifies gaze focus points and emotional states and processes them into a format that can be displayed visually.

[1937] Step 5:

[1938] Advice Generation Phase

[1939] The server generates specific advice using the advice generation means based on the analysis results obtained by the data analysis means. The input is the visualized analysis results, and the output is personalized advice. Specifically, it generates improvement suggestions for store layout and promotions based on which areas and products customers are interested in and in which situations they are emotionally moved.

[1940] Step 6:

[1941] Advice Providing Phase

[1942] The terminal receives the advice sent from the server and displays it to the user. The input is the generated advice, and the output is the advice presented to the user. Specifically, the terminal interactively displays the advice content and provides an interface that allows the user to check detailed information.

[1943] Step 7:

[1944] Feedback gathering phase

[1945] Based on the provided advice, the user makes changes to the store layout or improves customer service methods, and evaluates the effectiveness. The input is the specific actions taken based on the advice, and the output is feedback data on the effectiveness of the implemented improvements. Specifically, the visual data storage device is used again to collect gaze data and emotion data, and the areas for improvement are confirmed and evaluated based on the new data.

[1946] Through the above processing steps, it is possible to perform detailed analysis of customer behavior and emotional states in physical stores, and based on that, it is possible to effectively manage stores and improve customer experience.

[1947] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1948] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1949] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1950] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1951] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1952] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1953] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1954] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1955] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1956] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1957] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1958] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1959] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1960] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1961] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1962] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1963] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1964] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1965] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1966] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1967] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1968] The following is further disclosed regarding the above embodiment.

[1969] (Claim 1)

[1970] A visual data storage device worn by top athletes and coaches,

[1971] a server that receives and stores the gaze data recorded by the visual data storage device;

[1972] a data analysis means for analyzing the gaze data and identifying focus points and gaze movement patterns;

[1973] an advice generating means for providing advice to a user based on the analysis result;

[1974] A system including:

[1975] (Claim 2)

[1976] 10. The system of claim 1, wherein the visual data storage device records and transmits gaze data to the server in real time.

[1977] (Claim 3)

[1978] The system according to claim 1, wherein the data analysis means uses AI technology to analyze gaze data and visualize it as a gaze map or heat map.

[1979] "Example 1"

[1980] (Claim 1)

[1981] A visual data storage device worn by top athletes and coaches,

[1982] an information processing device that receives and stores the gaze data recorded by the visual data storage device;

[1983] a pre-processing device for cleaning the gaze data and passing the same to a data analysis means;

[1984] a data analysis means for analyzing the gaze data and identifying focus points and gaze movement patterns;

[1985] A means for visualizing the analysis results as a gaze map or a heat map;

[1986] an advice generating means for providing advice to a user based on the analysis result;

[1987] a display means for displaying the advice on a user's terminal;

[1988] A system including:

[1989] (Claim 2)

[1990] 2. The system of claim 1, wherein the visual data storage device records gaze data in real time and transmits it to the information processing device.

[1991] (Claim 3)

[1992] The system of claim 1, wherein the data analysis means analyzes gaze data using a generative AI model and visualizes it as a gaze map or heat map.

[1993] "Application Example 1"

[1994] (Claim 1)

[1995] A visual data storage device worn by top pilots,

[1996] a server that receives and stores the gaze data recorded by the visual data storage device;

[1997] a data analysis means for analyzing the gaze data and identifying focus points and gaze movement patterns;

[1998] an advice generating means for providing advice to a user based on the analysis result;

[1999] A means for supplying the gaze data analyzed by the data analysis means to a vehicle driving AI system in order to improve autonomous driving technology;

[2000] A means of providing appropriate advice in real time on driving situations during autonomous driving;

[2001] A visualization method that displays gaze maps and heat maps based on gaze data collected by professional drivers.

[2002] ...

[2003] A system including:

[2004] (Claim 2)

[2005] 10. The system of claim 1, wherein the visual data storage device records and transmits gaze data to the server in real time.

[2006] (Claim 3)

[2007] The system according to claim 1, wherein the data analysis means uses AI technology to analyze gaze data and visualize it as a gaze map or heat map.

[2008] "Example 2: Combining Emotion Engines"

[2009] (Claim 1)

[2010] A visual data storage device worn by top athletes and coaches,

[2011] An emotion engine that recognizes the user's emotions in real time;

[2012] a server that receives and stores data recorded by the visual data storage device and emotion engine;

[2013] a data analysis means for analyzing the gaze data and emotion data received from the server and identifying a focus point, a gaze movement pattern, and an emotion state;

[2014] an advice generating means for providing personalized advice to a user based on the analysis results;

[2015] ...

[2016] A system including:

[2017] (Claim 2)

[2018] 10. The system of claim 1, wherein the visual data storage device and emotion engine record gaze data and emotion data in real time and transmit them to the server.

[2019] (Claim 3)

[2020] The system according to claim 1, wherein the data analysis means uses AI technology to analyze the gaze data and emotion data and visualize them as a gaze map or a heat map.

[2021] "Application example 2 when combining emotion engines"

[2022] (Claim 1)

[2023] a visual data storage device worn by the customer;

[2024] a server that receives and stores the gaze data and emotion data recorded by the visual data storage device;

[2025] a data analysis means for analyzing the gaze data and emotion data and identifying a focus point and an emotion state;

[2026] an advice generating means for providing advice to a user based on the analysis results and visualizing customer behavior and emotions;

[2027] ...

[2028] A system including:

[2029] (Claim 2)

[2030] 2. The system of claim 1, wherein the visual data storage device records gaze data and emotion data in real time and transmits the data to the server.

[2031] (Claim 3)

[2032] The system according to claim 1, wherein the data analysis means uses AI technology to analyze the gaze data and emotion data and visualize them as a gaze map, emotion map, or heat map. [Explanation of symbols]

[2033] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. A visual data storage device worn by top athletes and coaches, a server that receives and stores the gaze data recorded by the visual data storage device; a data analysis means for analyzing the gaze data and identifying focus points and gaze movement patterns; an advice generating means for providing advice to a user based on the analysis result; A system including:

2. 2. The system of claim 1, wherein the visual data storage device records and transmits gaze data to the server in real time.

3. The system according to claim 1, wherein the data analysis means analyzes the gaze data using AI technology and visualizes it as a gaze map or a heat map.

Images