System

Abstract

A system is provided.SOLUTION: The system according to claim 1, further comprising: means for generating a proposal for improvement of shop operation from a result of the analysis; and means for notifying a shop manager of the proposal for improvement of shop operation, the proposal for improvement of shop operation being generated based on a result of the analysis of the behavior of the customer and the behavior of the AI clerk.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In brick-and-mortar stores, it is difficult to quantitatively evaluate the effectiveness of product displays and customer interactions, and improvements to sales and customer experience tend to be subjective. Furthermore, understanding the situation at multiple stores and proposing improvements takes time and money. The present invention aims to solve these problems and improve store operations efficiently and effectively. [Means for solving the problem]

[0005] The present invention is a system that includes: means for acquiring data in real time from multiple cameras and sensors installed inside and outside a store; means for storing the data acquired from the cameras and sensors in local storage and filtering out noise and incomplete data; means for uploading the filtered data to a server; means for inputting the uploaded data into a generative AI model and analyzing customer behavior and store staff behavior; and means for generating improvement proposals for store operations from the analysis results and notifying the store manager.

[0006] The system also includes a means for converting data acquired from the cameras and sensors into a compressed format and transmitting it to a server via a secure protocol, and a means for the generative AI model to analyze the customer's length of stay and the timing of when a store clerk will speak to them, and propose optimal product placement and timing for speaking to them.

[0007] A "camera" is a device that captures images, converts them into digital data, and stores or transmits them.

[0008] A "sensor" is a device that detects physical phenomena (position, movement, temperature, etc.) and converts them into digital data such as electrical signals.

[0009] "Data acquisition" is the process of collecting information from devices such as cameras and sensors.

[0010] "Local storage" refers to a data storage device inside a device (e.g., a hard disk or SSD).

[0011] "Noise filtering" is a process that removes unnecessary information and false positives from data.

[0012] A "server" is a computer system that provides data and services over a network.

[0013] A "generative AI model" is an algorithm that uses machine learning technology to generate information from data and perform inference and decision-making.

[0014] "Customer behavior" refers to the series of movements and actions that customers take in a store (e.g., length of stay, product selection, etc.).

[0015] "Store clerk actions" refers to business actions that store clerks take toward customers (e.g., greeting customers, providing guidance, etc.).

[0016] "Improvement proposals" are specific advice or instructions for improving business operations or management methods based on the results of data analysis.

[0017] "Notification" is the means or act of transmitting information to the target person (e.g., store manager).

[0018] A "compressed format" is a data format that uses compression technology to reduce the size of data.

[0019] A "protocol" is a set of common rules and procedures required for data communication between different systems.

[0020] "Real-time" refers to data acquisition and processing occurring almost simultaneously. [Brief explanation of the drawings]

[0021] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5]FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0022] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0023] First, the terms used in the following description will be explained.

[0024] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0025] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0026] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0027] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0028] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0029] [First embodiment]

[0030] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0031] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0032] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0033] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0034] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0035] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0036] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0037] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0038] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0039] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0040] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0041] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0042] In order to implement the present invention, a system can be constructed and operated in the following manner.

[0043] System Overview

[0044] This system collects data from cameras and sensors installed inside and outside the store, and uses a generative AI model to analyze customer behavior and store staff activities, thereby proposing improvements to store operations.The main components are as follows:

[0045] 1. Cameras and sensors: These are placed inside and outside the store to capture the movements of customers and store staff.

[0046] 2. Local terminal: collects data from cameras and sensors and performs primary processing and data cleaning.

[0047] 3. Server: Receives data uploaded from local devices and analyzes it using generative AI models.

[0048] 4. User (store manager): Receives improvement suggestions from the server and implements specific actions.

[0049] Explanation of program processing

[0050] Data collection

[0051] 1. The device collects real-time data from cameras and sensors installed inside and outside the store. The cameras capture images, and the sensors detect people's location and movement.

[0052] Example: A camera captures a customer pausing in front of a shelf, and a sensor records the duration of their visit.

[0053] Data storage and cleaning

[0054] 2. The data collected by the device is stored in local storage, and noise and incomplete data are filtered out during storage.

[0055] Example: Removing noise from video data and filling in missing parts of sensor data.

[0056] Uploading data

[0057] 3. The device compresses the cleaned data and uploads it to the server using the specified secure protocol.

[0058] Example: Compress data in gzip format and send it to the server using HTTPS.

[0059] Data analysis

[0060] 4. The server decompresses and derealizes the received data, performs preprocessing, and then inputs it into the generative AI model, which analyzes customer behavior and store clerk behavior.

[0061] Example: Extracting frames from video data, converting sensor data into time series data, and inputting it into a generative AI model.

[0062] 5. The server obtains the analysis results from the generative AI model and generates improvement proposals based on them.

[0063] Example: Identify shelves where customers spend a lot of time but products are not selling, and suggest changing the placement of those products.

[0064] Notification of improvement proposals

[0065] 6. The device notifies the store manager of the improvement suggestions received from the server, for example, via a mobile device or dashboard UI.

[0066] Example: A dashboard might suggest, "Move product A from shelf X to shelf Y."

[0067] Implementing improvement proposals and monitoring their effectiveness

[0068] 7. The user implements the improvement suggestions received and collects the results again, allowing the effectiveness of the improvement suggestions to be monitored.

[0069] Example: After changing product placement, new data is collected again through cameras and sensors to analyze changes in sales.

[0070] Specific examples

[0071] 1. The device collects behavioral data of 500 customers every day, filters out noise, and stores it in local storage.

[0072] 2. The device uploads the compressed data to the server overnight.

[0073] 3. The server decompresses and preprocesses the data and runs it through a generative AI model, detecting, for example, that a particular product is not selling well but many customers still stop in front of it.

[0074] 4. The server generates and notifies the user of a suggestion to change the placement of the products. The user then moves the products based on the suggestion.

[0075] 5. The server monitors subsequent sales data to assess the effectiveness of the changes.

[0076] In this way, by implementing the present invention, sales and customer experience in store operations can be effectively improved.

[0077] The processing flow will be explained below.

[0078] Step 1:

[0079] The device collects data in real time from multiple cameras and sensors installed inside and outside the store.

[0080] How it works: Cameras capture footage of customers, and sensors collect data like their location and movements.

[0081] Step 2:

[0082] The data acquired by the device is temporarily stored in local storage.

[0083] Specific operation: Stores captured video frames and sensor data in a database or file system.

[0084] Step 3:

[0085] The device performs noise filtering on the stored data to remove unnecessary data.

[0086] Specific operation: Automatically eliminates low-resolution footage and falsely detected sensor data, leaving only the necessary data.

[0087] Step 4:

[0088] The terminal periodically compresses the filtered data and uploads it to the server.

[0089] What it does: It converts the data into a compressed format such as gzip and transfers it securely to the server over HTTPS.

[0090] Step 5:

[0091] The server de-realizes the uploaded data and restores it to its original form.

[0092] Specific operation: The received compressed file is decompressed and the original video data and sensor data are restored.

[0093] Step 6:

[0094] The server preprocesses the restored data and converts it into a format that can be input into the generative AI model.

[0095] Specific operation: Video data is divided into frames and sensor data is converted into time series data.

[0096] Step 7:

[0097] The server inputs the preprocessed data into a generative AI model to analyze customer behavior and store clerk behavior.

[0098] Specific operation: The AI ​​model analyzes the customer's length of stay, their movement in front of the product, and the timing of the store clerk's call.

[0099] Step 8:

[0100] The server generates suggestions for improving store operations based on the analysis results from the generated AI model.

[0101] Specific actions: Make specific suggestions such as changing product placement and training content for store staff.

[0102] Step 9:

[0103] The terminal notifies the store manager (user) of the improvement proposal received from the server.

[0104] Specific actions: Display suggestions on mobile devices and dashboards, clearly indicating the actions to be taken.

[0105] Step 10:

[0106] Implement the improvement suggestions received by the user and record the results.

[0107] Specific actions: Move products and adjust the actions of store staff, and enter the results into the system.

[0108] Step 11:

[0109] The server collects data again and monitors the effectiveness of the improvement suggestions.

[0110] Specific Actions: Analyze new customer behavior and sales data to evaluate performance after improvements.

[0111] Example 1

[0112] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0113] In traditional store operations, there is a lack of means to properly analyze customer behavior and employee activity and propose operational improvement measures in real time. This makes it difficult to optimize sales and improve customer experience. In addition, there are poor means to monitor the effectiveness of improvement proposals, making it difficult to quantitatively grasp the actual effects of improvements.

[0114] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0115] In this invention, the server includes means for acquiring data in real time from monitoring devices and detection devices, means for storing the data acquired from the devices in a local storage device and filtering out noise and incomplete data, means for compressing the filtered data and transmitting it to the server via a secure protocol, means for the server to decompress and de-realize the data received, pre-process it, and input it into a generative AI model to analyze customer behavior and employee behavior, and means for generating store operation improvement proposals from the analysis results and notifying the store manager. This makes it possible to dramatically improve the efficiency of store operation and enable real-time monitoring of the effects of improvement proposals and rapid feedback.

[0116] A "monitoring device" is a device that is installed in a target area and acquires video data in real time.

[0117] A "detection device" is a device that is installed in a target area and detects the location and movement of people and objects in real time.

[0118] A "local storage device" is a storage device for temporarily storing collected data.

[0119] "Noise" refers to unnecessary or incorrect information that gets mixed in during the data collection process.

[0120] "Incomplete data" is data that contains missing or incorrect information during the collection process.

[0121] "Filtering" is the process of removing noise and imperfections from collected data.

[0122] "Compression" is a process performed to reduce the volume of data.

[0123] A "secure protocol" is a communication method that ensures security when sending and receiving data.

[0124] The "server" is a central device that receives collected data and processes and analyzes it.

[0125] "Derealization" is the process of returning received data to its original format.

[0126] "Preprocessing" is the process of converting data into an appropriate format before feeding it into a generative AI model.

[0127] A "generative AI model" is an artificial intelligence model that analyzes data and recognizes customer and employee behavior.

[0128] "Analysis" is the process of extracting information for a specific purpose from collected data.

[0129] "Improvement proposals" are specific proposals based on the analysis results to improve the efficiency of store operations and sales.

[0130] "Store Manager" means the person responsible for managing the store's operations and implementing improvement suggestions.

[0131] "Notification" refers to the act of informing the store manager of the analysis results and improvement suggestions.

[0132] "Monitoring" is the process of continuously observing the effectiveness of implemented improvement proposals.

[0133] The present invention is a system that collects data from monitoring devices and detection devices installed inside and outside a store, analyzes the data, and proposes improvements to store operations. The following hardware and software are used to implement the present invention.

[0134] Hardware

[0135] 1. Surveillance devices (cameras): Install multiple cameras inside and outside the store to capture the movements of customers and employees in real time.

[0136] 2. Detection device (sensor): Detects people's movements and staying time and provides location data.

[0137] 3. Local storage device (local terminal): A device for temporarily storing data obtained from cameras and sensors.

[0138] 4. Server: A central processing unit that receives, decompresses, preprocesses, and analyzes data.

[0139] 5. Notification device: A mobile device or dashboard UI for notifying store managers of improvement suggestions.

[0140] software

[0141] 1. Image processing algorithm: Processes the image data collected from the camera and removes noise.

[0142] 2. Data filtering algorithms: Fill in and refine imperfections in sensor data.

[0143] 3. Compression software: Compress the data in gzip format.

[0144] 4. Secure Communication Protocol (HTTPS): A protocol for securely sending and receiving data to and from a server.

[0145] 5. Generative AI model: The artificial intelligence model used for analysis.

[0146] Specific flow of data

[0147] The terminal acquires data from monitoring and detection devices, denoises and filters it, and stores it in local storage. The data is then compressed and sent to the server via a secure protocol. The server decompresses and derealizes the received data, preprocesses it, and inputs it into a generative AI model. This model analyzes customer and employee behavior and generates improvement suggestions. Finally, the server sends these improvement suggestions to a notification terminal, which notifies the store manager. The user implements the improvement suggestions and monitors their effectiveness by collecting new data and reanalyzing it.

[0148] Specific examples

[0149] For example, specific improvement suggestions are made, such as identifying shelves with long customer dwell times but poor sales, and suggesting changes to their layout. Sample prompts are used in conjunction with these suggestions to analyze the generative AI model. Examples of prompts are shown below.

[0150] Example prompt sentence:

[0151] "Analyze specific shelf purchasing behavior based on current dwell time data."

[0152] In this way, the present invention realizes efficient store management and increased sales, and brings about significant improvements over conventional methods.

[0153] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0154] Step 1:

[0155] The terminal receives real-time data from the monitoring and detection devices. The inputs include video data from the monitoring devices and location data from the detection devices, which are required for subsequent processing.

[0156] How it works: Cameras continuously capture customer movements, and sensors record each customer's location and movements.

[0157] Step 2:

[0158] The data acquired by the device is stored in local storage. The input is the raw data acquired in step 1. The output is the stored data. During this process, noise removal and data filtering are performed.

[0159] Specific operation: Image processing algorithms are used to remove noise from video data and fill in missing data.

[0160] Step 3:

[0161] The device compresses the filtered data and sends it to the server via a secure protocol (e.g., HTTPS). The input is the clean data stored in step 2. The output is the compressed data sent to the server.

[0162] Specific operation: Data is compressed in gzip format and uploaded to the server periodically every night.

[0163] Step 4:

[0164] The server decompresses, derealizes, and preprocesses the received data. The input is the compressed data sent in step 3. The output is the preprocessed data.

[0165] Specific operation: The server decompresses the gzip file, divides the video data into frames, and converts the sensor data into time series data.

[0166] Step 5:

[0167] The server inputs the preprocessed data into the generative AI model to analyze customer and employee behavior. The input is the data preprocessed in step 4. The output is the analysis results.

[0168] Specific operation: The frame data and the shaped sensor data are input into an AI model, which then analyzes customer stay times and employee movements.

[0169] Step 6:

[0170] The server generates improvement proposals based on the analysis results. The input is the analysis results obtained in step 5. The output is improvement proposals.

[0171] Specific operation: Identify shelf locations with poor sales based on customer behavior data and suggest product relocation.

[0172] Step 7:

[0173] The server sends the generated improvement proposal to the terminal, which then notifies the user. The input is the improvement proposal generated in step 6. The output is the notified improvement proposal.

[0174] Specific action: The suggestions are displayed on your smartphone or dashboard, and you are notified of the necessary actions.

[0175] Step 8:

[0176] The user executes the improvement proposal received, which generates new data. The input is the improvement proposal notified in step 7. The output is the execution result.

[0177] Specific operation: The user changes, for example, product placement based on the suggestions.

[0178] Step 9:

[0179] The terminal collects new data after implementing the improvement proposal and sends it back to the server to monitor the effectiveness of the improvement proposal. The input is the new data generated in step 8. The output is the monitored effectiveness data.

[0180] Specific actions: Collect new customer behavior data, send it back to the server, and analyze it to evaluate the effectiveness of improvements.

[0181] (Application example 1)

[0182] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0183] Conventional store management systems have difficulty understanding the behavior of customers and store employees in real time and automatically generating and notifying effective improvement proposals. They also lack the means to quickly evaluate the effectiveness of proposals after their implementation and to make continuous improvements. As a result, store managers are unable to quickly and efficiently optimize store operations, and are often left behind in improving sales and customer satisfaction. This problem needs to be solved.

[0184] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0185] In this invention, the server includes means for acquiring data in real time from multiple cameras and sensors installed inside and outside the store, means for storing the data acquired from the cameras and sensors in local storage and filtering out noise and incomplete data, means for uploading the filtered data to the server, means for inputting the uploaded data into a generative AI model and analyzing customer behavior and store employee behavior, means for generating store operation improvement proposals from the analysis results and notifying the store manager, means for monitoring the operation data after the improvement proposals are implemented and evaluating the effectiveness of the improvement proposals, means for viewing the camera feeds in real time using a smartphone and a head-mounted display and checking the analysis results and improvement proposals, and means for receiving notifications of the improvement proposals via the smartphone and the head-mounted display. This makes it possible to analyze customer and store employee behavior in real time, quickly provide store operation improvement proposals, and continuously evaluate their effectiveness.

[0186] A "camera" is a device installed inside or outside a store that captures the movements of customers and store staff as video.

[0187] A "sensor" is a device installed inside and outside a store that detects people's location and movement.

[0188] "Local storage" is a storage device for temporarily storing data acquired from cameras and sensors.

[0189] "Noise filtering" is the process of removing inaccuracies and unnecessary information from data.

[0190] A "server" is a computer system that receives, analyzes, and stores data over a network.

[0191] A "generative AI model" is an artificial intelligence system that analyzes customer behavior and store clerk behavior and automatically generates improvement suggestions.

[0192] A "smartphone" is a small personal digital assistant (PDA) that can view camera feeds and receive notifications.

[0193] A "head-mounted display" is a device worn on the head that displays images in real time.

[0194] A "store manager" is a person who is responsible for running a store and takes action based on improvement proposals.

[0195] "Operational data" refers to data that indicates the operational status of a store, and includes sales information, customer behavior, and the like.

[0196] "Camera feed" is video data sent from a camera in real time.

[0197] "Improvement proposals" are specific proposals for improving store operations based on data analyzed by the generative AI model.

[0198] "Effectiveness evaluation" is the process of analyzing the results after implementing improvement proposals and determining their effectiveness.

[0199] As a specific embodiment for implementing this invention, a system is constructed that utilizes cameras and sensors installed inside and outside the store, a local terminal, a server, a smartphone, and a head-mounted display.

[0200] Hardware and software used

[0201] Camera: A device installed inside and outside a store that captures the movements of customers and store staff as video.

[0202] Sensor: A device installed inside and outside a store that detects people's location and movement.

[0203] Local device: A computer that temporarily stores and preprocesses data acquired from cameras and sensors.

[0204] Server: A computer system that receives, analyzes, and stores data.

[0205] Smartphone: A small personal digital assistant that allows you to view camera feeds and receive notifications.

[0206] Head-mounted display: A device worn on the head that displays images in real time.

[0207] Data processing and calculation

[0208] 1. Data Collection:

[0209] The device collects real-time data from cameras and sensors installed inside and outside the store. The cameras capture images, and the sensors detect people's location and movement.

[0210] Example: A camera captures a customer pausing in front of a shelf, and a sensor records the duration of their visit.

[0211] 2. Data Storage and Filtering:

[0212] The data collected by the device is stored in local storage and noise and incomplete data are filtered out.

[0213] Example: Removing noise from video data and filling in missing parts of sensor data.

[0214] 3. Data upload:

[0215] The device compresses the cleaned data and uploads it to the server using a secure protocol (e.g. HTTPS).

[0216] Example: Compress data in gzip format and send it to a server overnight.

[0217] 4. Data Analysis:

[0218] The server decompresses and derealizes the received data, performs preprocessing, and then inputs it into a generative AI model, which analyzes customer behavior and store clerk behavior.

[0219] Example: Extracting frames from video data, converting sensor data into time series data, and inputting it into a generative AI model.

[0220] 5. Generation and notification of improvement suggestions:

[0221] The server obtains the analysis results using the generative AI model, generates improvement suggestions based on them, and notifies the store manager.

[0222] Example: Identify shelves where customers spend a lot of time but products are not selling, and suggest changing the placement of those products.

[0223] 6. Effect monitoring:

[0224] The user implements the improvement suggestions received and collects the results again. The server then monitors the sales data and evaluates the effectiveness of the changes.

[0225] Example: After changing product placement, new data is collected again through cameras and sensors to analyze changes in sales.

[0226] 7. Smartphone and head-mounted display usage:

[0227] Users can use their smartphones and head-mounted displays to view real-time camera feeds and see analysis results and improvement suggestions.

[0228] Notification of improvement suggestions will be sent via smartphone and head-mounted display.

[0229] Example prompt: "Analyze customer dwell time and generate shelf relocation suggestions for underperforming products."

[0230] The above configuration makes it possible to improve the efficiency of store operations, and is expected to result in increased customer satisfaction and sales.

[0231] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0232] Step 1:

[0233] The terminal acquires data in real time from cameras and sensors installed inside and outside the store. Specifically, the cameras capture images, and the sensors detect the positions and movements of customers and store staff. The input is the camera images and sensor detection data, and the output is a combination of these data. For example, the camera captures a customer stopping in front of a shelf, and the sensor records the duration of their stay.

[0234] Step 2:

[0235] The data collected by the device is stored in local storage and filtered for noise and incomplete data. Specifically, it removes noise from the video data and complements missing sensor data. The input is camera video and sensor detection data, and the output is clean, filtered data. For example, it improves image quality when the video is unclear and interpolates continuous sensor operation data.

[0236] Step 3:

[0237] The terminal compresses the cleaned data and uploads it to a server using a secure protocol (e.g. HTTPS). The input is the filtered data and the output is the compressed data and confirmation of its destination. For example, the data may be compressed in gzip format and sent to the server overnight.

[0238] Step 4:

[0239] The server decompresses and deserializes the data it receives. Specifically, it converts compressed data back to its original format so that it can be analyzed. The input is compressed data, and the output is decompressed data. For example, it decompresses video data and sensor data compressed with Gzip.

[0240] Step 5:

[0241] The server preprocesses the decompressed and deserialized data and inputs it into the generative AI model. Specifically, it extracts the video data frame by frame and converts the sensor data into time series data. The input is the decompressed data, and the output is the input data for the generative AI model. For example, the video data can be converted into an analyzable format frame by frame, and sensor information converted into time series data can be created.

[0242] Step 6:

[0243] The server uses a generative AI model to analyze customer and store clerk behavior and generates suggestions for improving store operations based on the results. The input is the data input to the generative AI model, and the output is the analysis results and suggestions for improvement. For example, it identifies shelves where customers spend a lot of time but products are not selling, and suggests changing the placement of those products.

[0244] Step 7:

[0245] The server notifies the store manager of the improvement suggestion via the smartphone and head-mounted display. The input is the generated improvement suggestion, and the output is a confirmation that the notification has been sent. For example, the smartphone and head-mounted display display a suggestion such as "Please move product A from shelf X to shelf Y."

[0246] Step 8:

[0247] The user implements the improvement suggestions they receive, and the results are collected again using cameras and sensors. The input is the implementation results of the improvement suggestions, and the output is the operational data after the changes. For example, after changing product placement, new data is collected again using cameras and sensors to analyze changes in sales.

[0248] Step 9:

[0249] Based on the data recollected by the server, the effectiveness of the proposed improvements is evaluated and further proposals are made as necessary. The input is the operational data after the changes, and the output is an evaluation of the effectiveness and new proposals for improvements. For example, the system can confirm the increase in sales after changing product placement and further propose optimization of customer flow.

[0250] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0251] In order to implement the present invention, a system can be constructed and operated in the following manner.

[0252] System Overview

[0253] This system collects data from cameras and sensors installed inside and outside the store, and uses a generative AI model and emotion engine to analyze customer behavior, store clerk activities, and even user emotions, to propose improvements to store operations.The main components are as follows:

[0254] 1. Cameras and sensors: Placed inside and outside the store, they capture the movements of customers and store staff, as well as user emotions.

[0255] 2. Local terminal: collects data from cameras and sensors and performs primary processing and data cleaning.

[0256] 3. Server: Receives data uploaded from local devices and analyzes it using generative AI models and emotion engines.

[0257] 4. User (store manager): Receives improvement suggestions from the server and implements specific actions.

[0258] Explanation of program processing

[0259] Data collection

[0260] 1. The device collects data in real time from cameras and sensors installed inside and outside the store. The cameras capture images, and the sensors collect data such as people's location and movements. In addition, an emotion engine analyzes the user's emotions from their facial expressions and voice.

[0261] Example: A camera captures a customer's movements and facial expressions as they pause in front of a shelf, and sensors record their dwell time and emotions.

[0262] Data storage and cleaning

[0263] 2. The device temporarily stores the collected data in local storage, filtering out noise and incomplete data when storing.

[0264] Example: Removing noise from video data and filling in missing parts of sensor data.

[0265] Uploading data

[0266] 3. The device compresses the filtered data and uploads it to the server using the specified secure protocol.

[0267] Example: Compress data in gzip format and send it to the server using HTTPS.

[0268] Data analysis

[0269] 4. The server de-realizes the uploaded data and restores it to its original form.

[0270] Specific operation: The received compressed file is decompressed and the original video data and sensor data are restored.

[0271] 5. The server preprocesses the recovered data and converts it into a format that can be input to the generative AI model and emotion engine.

[0272] Examples: Splitting video data into frames, converting sensor data into time series data, and formatting facial expression data for analysis.

[0273] 6. The server inputs the preprocessed data into the generative AI model and emotion engine to analyze customer behavior, store clerk behavior, and user emotions.

[0274] Example: The AI ​​model analyzes the customer's length of stay, their movement in front of the product, the timing of the store clerk's call, changes in facial expression, etc.

[0275] 7. The server generates suggestions for improving store operations based on the analysis results from the generated AI model and emotion engine. The suggestions also take into account the user's emotional state.

[0276] Example: An emotion engine infers customer satisfaction from their facial expressions, identifies products and placements that cause dissatisfaction, and suggests areas for improvement.

[0277] Notification of improvement proposals

[0278] 8. The device notifies the store manager of the improvement suggestions received from the server, for example, via a mobile device or dashboard UI.

[0279] Example: The dashboard might say, "Move product A from shelf X to shelf Y and have the sales associate provide specific advice to the customer."

[0280] Implementing improvement proposals and monitoring their effectiveness

[0281] 9. Implement the improvement suggestions received from the user and record the results.

[0282] Example: Adjusting merchandise movements and store staff actions and inputting the results into the system.

[0283] 10. The server collects data again and monitors the effectiveness of the proposed improvements.

[0284] Example: Analyzing new customer behavior data, sales data, and customer facial expression data to evaluate performance and customer satisfaction after improvements.

[0285] Specific examples

[0286] 1. The device collects behavioral and emotional data of 500 customers every day, filters out noise, and stores it in local storage.

[0287] 2. The device uploads the compressed data to the server overnight.

[0288] 3. The server decompresses and preprocesses the data, then analyzes it with a generative AI model and emotion engine. For example, it might detect that a particular product is not selling well, but many customers stop in front of it with unhappy expressions.

[0289] 4. The server generates improvement proposals, including suggestions for changing product placement and training for store staff, and notifies them.

[0290] 5. The user implements the suggestions and collects the performance results again.

[0291] 6. The server analyzes the new data, monitors the effectiveness of the suggested improvements, and makes further recommendations.

[0292] In this way, by implementing the present invention, it is possible to effectively improve sales and customer experience in store operations, as well as increase customer satisfaction.

[0293] The processing flow will be explained below.

[0294] Step 1:

[0295] The device collects data in real time from multiple cameras and sensors installed inside and outside the store.

[0296] How it works: Cameras capture footage of customers, and sensors collect their location, movement, and user audio data.

[0297] Step 2:

[0298] The data acquired by the device is temporarily stored in local storage.

[0299] Specific behavior: Stores captured video frames, sensor data, and audio data in a database or file system.

[0300] Step 3:

[0301] The device performs noise filtering on the stored data to remove unnecessary data.

[0302] Specific operation: Filters out low-resolution video, misdetected sensor data, and audio data containing noise, leaving only the necessary data.

[0303] Step 4:

[0304] The device compresses the filtered data and uploads it to the server.

[0305] Specific operation: Compress the data into gzip format and send it securely to the server via HTTPS.

[0306] Step 5:

[0307] The server unpacks the uploaded data and restores it to its original format.

[0308] Specific operation: The received compressed file is decompressed and restored to the original video data, sensor data, and audio data.

[0309] Step 6:

[0310] The server preprocesses the recovered data and converts it into a format that can be input to the generative AI model and emotion engine.

[0311] Specific operations: Video data is divided into frames, sensor data is organized into time series data, and audio data is converted into text.

[0312] Step 7:

[0313] The server inputs the preprocessed data into a generative AI model and emotion engine to analyze customer behavior, store clerk behavior, and user emotions.

[0314] Specific operation: The generative AI model analyzes the customer's dwell time and movement in front of the product, and the emotion engine analyzes the customer's emotions from facial expressions and voice data.

[0315] Step 8:

[0316] The server generates improvement proposals for store operations based on the analysis results.

[0317] Specific operation: The results of the generative AI model and the emotion engine are combined to compile suggestions such as changing product placement, improving the way store staff serve customers, and providing services based on customer emotions.

[0318] Step 9:

[0319] The terminal notifies the store manager (user) of the improvement proposal received from the server.

[0320] Specific actions: Display suggestions on mobile devices and dashboard UI, and clearly indicate specific actions.

[0321] Step 10:

[0322] Implement the improvement suggestions received by the user and record the results.

[0323] Specific actions: Change product placement, give instructions to store staff, implement measures to improve customer service, and feed the results back into the system.

[0324] Step 11:

[0325] The server collects data again and monitors the effectiveness of the improvement suggestions.

[0326] Specific actions: Analyze new customer behavior data, sales data, and customer facial expression data to evaluate the effectiveness of implemented improvement measures.

[0327] Example 2

[0328] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0329] Conventional customer behavior analysis systems for store operations are limited to providing feedback on customer behavior data and store clerk responses, and are unable to accurately grasp the user's emotional state and reflect it in improvement proposals. Furthermore, the analysis targets noisy or incomplete data, resulting in reduced proposal accuracy. The present invention aims to solve these problems and provide more accurate store operation improvement proposals.

[0330] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for uploading the filtered data to the server, a means for derealizing the uploaded data, preprocessing it, and converting it into an analyzable format, and a means for inputting the preprocessed data into a generative AI model and an emotion analysis engine, analyzing customer behavior and clerk behavior, and estimating user emotions. This enables highly accurate data analysis and improvement suggestions that include the user's emotional state.

[0331] A "camera" is a device for capturing video data.

[0332] A "sensor" is a device that detects physical changes and outputs them as data, specifically one that detects a person's position or movement.

[0333] "Local storage" is an internal storage device for temporarily storing data.

[0334] "Filtering" refers to the process of removing noise and imperfections from data.

[0335] "Upload" refers to the process of sending data from a local terminal to a server.

[0336] "Deserialization" is the process of returning serialized data to its original form.

[0337] "Preprocessing" refers to the initial processing of data to prepare it in an analyzable format.

[0338] A "generative AI model" is an algorithm that uses machine learning to analyze data and generate a specific result.

[0339] An "emotion analysis engine" is an algorithm or system for inferring emotions from a user's facial expressions and voice.

[0340] "Customer behavior" refers to data on customer behavior such as location, movement patterns, and length of stay within the store.

[0341] "Employee behavior" refers to data on employee behavior such as their location within the store, movement patterns, and how they interact with customers.

[0342] "Analysis" refers to the act of processing data to extract useful information and patterns.

[0343] "Improvement proposals" refer to proposals based on analysis results aimed at optimizing store operations and customer service.

[0344] "Notification" refers to the process of communicating information to store managers, specifically via mobile devices or dashboards.

[0345] The following describes in detail an embodiment of the present invention: The present invention is a system that utilizes data from cameras and sensors installed inside and outside a store, and analyzes customer behavior, store clerk behavior, and user emotions using a generative AI model and an emotion analysis engine.

[0346] System Overview

[0347] The system mainly includes the following hardware and software:

[0348] Cameras: Multiple cameras are installed inside and outside the store to capture video data.

[0349] Sensors: There are infrared sensors and various motion sensors that collect location information of customers and store staff.

[0350] Device: Stores collected data in local storage and filters out noise and incomplete data.

[0351] Server: De-realizes the filtered data, pre-processes it, and converts it into an analyzable format. It then analyzes it using a generative AI model and sentiment analysis engine to generate improvement suggestions.

[0352] Data collection and cleaning

[0353] The device collects data in real time from cameras and sensors. For example, the camera captures video at 30 frames per second, and sensors acquire location data of customers and store staff. This data is stored in local storage, where noise and incomplete data are filtered out. For example, ghost noise in the video data is removed and incomplete sensor data is complemented.

[0354] Data upload and de-realization

[0355] The device compresses the filtered data in gzip format and uploads it to the server via secure communication using the HTTPS protocol. The server then decompresses and derealizes the received data. For example, it splits the decompressed video data into frames and stores the sensor data in a database.

[0356] Preprocessing and analysis

[0357] The server preprocesses the derealized data and prepares it in a format that can be analyzed by the generative AI model and emotion analysis engine. This includes resizing the video data, extracting frames, and normalizing the sensor data. For example, the server resizes the video data to 256x256 pixels, which is easier for the AI ​​model to handle, and fills in missing parts of the sensor data to prepare it as time-series data.

[0358] The server inputs the preprocessed data into a generative AI model and an emotion analysis engine to analyze customer behavior, store clerk behavior, and user emotions. The AI ​​model analyzes the customer's length of stay, their movement, and the timing of the clerk's interactions, while the emotion analysis engine estimates the customer's emotional state based on facial expressions and voice data. For example, the generative AI model analyzes the customer's length of stay, and the emotion analysis engine analyzes the customer's dissatisfied facial expression.

[0359] Improvement suggestions and notifications

[0360] The server generates suggestions for improving store operations based on the analysis results and the customer's emotional state. For example, suggestions may be generated that include changing product placement or specific guidelines for store staff behavior. These suggestions are notified to store managers via their devices. Notification methods include a dashboard UI or a mobile device. As a specific example, the dashboard may display a message such as, "Move product A from shelf X to shelf Y, and have store staff provide specific advice to customers."

[0361] Implementing improvement proposals and monitoring their effectiveness

[0362] The user implements the improvement proposal and records the results. For example, changing the product layout and adjusting the behavior of the store clerk. The data after the improvement is collected again and entered into the system, allowing the server to analyze it again, monitor the effectiveness of the proposal, and make further improvement proposals. For example, new customer behavior data, sales data, and customer satisfaction data are evaluated, and feedback is provided for further improvements.

[0363] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0364] Step 1:

[0365] The device acquires data in real time from the camera and sensors. The input data is video data from the camera and position and movement data from the sensors. Specifically, the camera captures video at 30 frames per second, and the sensors acquire position information of customers and store staff. The output is raw video data and sensor data.

[0366] Step 2:

[0367] The data collected by the device is temporarily stored in local storage and noise and incomplete data are filtered. The input data are the raw video data and sensor data acquired in the previous step. Specifically, the process removes ghost noise contained in the video data and complements incomplete sensor data. The output is filtered video data and sensor data.

[0368] Step 3:

[0369] The terminal compresses the filtered data in gzip format and uploads it to the server using the HTTPS protocol. The input data is the filtered video data and sensor data. Specifically, the data is compressed in gzip format and transmitted securely using HTTPS. The output is the compressed data uploaded to the server.

[0370] Step 4:

[0371] The server de-realizes the uploaded data and restores it to its original format. The input data is compressed video data and sensor data. Specifically, it decompresses the received gzip format file and restores it to video data and sensor data. The output is the decompressed and restored video data and sensor data.

[0372] Step 5:

[0373] The server preprocesses the restored data and converts it into a format that can be analyzed by the generative AI model and emotion analysis engine. The input data is the derealized video data and sensor data. Specifically, the server splits the video data into frames and resizes it to 256x256 pixels. It also converts the sensor data into time-series data and extracts facial expression and movement features. The output is the preprocessed data.

[0374] Step 6:

[0375] The server inputs the preprocessed data into a generative AI model and emotion analysis engine, which analyzes customer behavior, store clerk behavior, and user emotions. The input data is preprocessed video data and sensor data. Specifically, the generative AI model analyzes customer movements and length of stay, and the emotion analysis engine infers the user's emotions from facial expression data. The output is the analysis results.

[0376] Step 7:

[0377] The server generates improvement proposals for store operations based on the analysis results. The input data are the analysis results. Specific operations include identifying products and placements that cause high customer dissatisfaction from the analysis results and proposing improvements accordingly. The output is specific improvement proposals.

[0378] Step 8:

[0379] The device notifies the user of the improvement proposal received from the server. The input data is the generated improvement proposal. As a specific operation, the proposal content is displayed through the dashboard UI or mobile device. The output is the improvement proposal notified to the user.

[0380] Step 9:

[0381] The user implements the improvement proposals received and records the results. The input data is the notified improvement proposal. Specific actions include moving products and adjusting the actions of store clerks, and inputting the resulting new behavioral and sales data into the system. The output is the implemented improvement proposals and new data.

[0382] Step 10:

[0383] The server re-analyzes the newly collected data and monitors the effectiveness of the proposed improvements. The input data is the new data after execution. Specific operations include evaluating the analysis results and determining how effective the proposals were. Based on this, further improvement proposals are made. The output is an evaluation of the effectiveness of the proposed improvements and new proposals.

[0384] (Application example 2)

[0385] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0386] In recent years, improving store operation efficiency and customer satisfaction have become important issues in the retail industry, but conventional methods have difficulty in conducting detailed analysis of customer behavior and emotions, resulting in a lack of concrete improvement proposals based on this analysis.In addition, while employee activities need to be monitored and supervised efficiently, current systems are unable to adequately achieve this.

[0387] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for acquiring information in real time from observation devices and sensing devices, means for saving the acquired information in a local storage device and filtering out useless or incomplete data, means for transferring the filtered information to a computer, means for inputting the transferred information into a generative AI model and a sentiment analysis engine to analyze customer behavior, employee activities, and user emotions, and means for generating store operation improvement proposals from the analysis results and notifying the manager. This makes it possible to perform detailed analysis of customer behavior, employee activities, and user emotions and provide specific and effective store operation improvement proposals.

[0388] "Observation devices" are devices installed to monitor and record the behavior and situations of customers and employees.

[0389] A "sensing device" is a device that collects information about the location and movement of customers and employees, as well as environmental information.

[0390] "Information" refers to data acquired in real time from observation and sensing devices, and includes video, audio, movement patterns, emotional states, and the like.

[0391] A "local storage device" is a device for temporarily storing information obtained from observation and sensing devices.

[0392] "Filtering" is the process of removing useless or incomplete data from stored information.

[0393] "Computer" refers to a server or cloud computing system for analyzing information transferred from local storage.

[0394] A "generative AI model" is a machine learning model trained to analyze customer and employee behavior and emotions.

[0395] An "emotion analysis engine" is software that analyzes emotions from a user's facial expressions and voice.

[0396] "Analysis results" refers to information about customer behavior, employee activities, and emotional states obtained by the generative AI model and sentiment analysis engine.

[0397] "Improvement proposals" are specific action plans for optimizing store operations and improving customer satisfaction based on the analysis results.

[0398] A "manager" is a person who is responsible for overseeing store operations and implementing improvement suggestions.

[0399] Summary of the Invention

[0400] This invention is a system that uses a generative AI model and an emotion analysis engine to analyze customer behavior, employee activity, and user emotions based on data collected from observation and sensing devices in a physical store, and provides suggestions for improving store operations. This system can improve customer satisfaction and streamline store operations.

[0401] Detailed system description

[0402] Data collection

[0403] The system collects information in real time from observation devices (cameras) and sensing devices (sensors) installed inside and outside the store. The observation devices record the behavior of customers and employees, while the sensing devices collect information on their location, behavior, and environment.

[0404] Storing and Filtering Data

[0405] The collected information is stored on a local storage device (e.g., a device such as a Raspberry Pi). When stored, useless or incomplete data is filtered out, leaving only the necessary information.

[0406] Data Transfer

[0407] The filtered information is converted into a compressed format (eg, gzip format) and transferred to a computer (a server or a cloud computing system) via a secure communication means (eg, HTTPS protocol).

[0408] Data analysis

[0409] The server receives the transferred information and inputs it into a generative AI model and sentiment analysis engine, which analyzes customer behavior, employee activity, and the user's emotional state. The generative AI model is trained using machine learning frameworks such as TensorFlow and PyTorch, and the sentiment analysis engine uses Microsoft Azure Emotion API and Amazon Rekognition.

[0410] Generate and notify improvement suggestions

[0411] Based on the analysis results, the server generates specific suggestions for improving store operations. These suggestions are notified to the manager in real time, who then acts accordingly. Notifications are sent via a smartphone app (e.g., an app developed based on Flutter or React Native).

[0412] Examples of specific examples and prompts

[0413] For example, if many customers stop in front of a certain product but do not end up purchasing it, the generative AI model will determine that there is a problem with the product's placement or explanation, and will suggest moving the product or improving the explanation.In addition, if the emotion analysis engine detects low satisfaction from customers' facial expressions, it will generate suggestions to encourage employees to take a more proactive approach.

[0414] An example prompt is:

[0415] "Customers who pause in front of product A spend a lot of time there, but the percentage of them actually purchasing is low. The sentiment analysis engine indicates that customer satisfaction is low. Therefore, move product A from shelf X to shelf Y, and have an associate provide specific advice to the customer."

[0416] This allows for detailed analysis of customer behavior, employee activities, and user sentiment, making it possible to provide specific and effective suggestions for improving store operations.

[0417] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0418] Step 1:

[0419] The terminal acquires information in real time from observation devices (cameras) and detection devices (sensors) installed inside and outside the store. The input is data from each observation device and detection device, which is received by the terminal. Specifically, the camera records customer behavior, and the sensor collects movement and location information.

[0420] Step 2:

[0421] The device stores the acquired information in a local storage device. Here, it receives the acquired raw data as input and filters out noise and incomplete data. Specifically, it removes noise from the video data and fills in missing parts of the sensor data. The output is clean, filtered data.

[0422] Step 3:

[0423] The terminal converts the filtered data into a compressed format and transfers it to the server. It receives clean data as input and compresses it, for example, using gzip format. It then sends it to the server using a secure communication method (for example, HTTPS protocol). The specific operations include compressing the data, establishing a secure network connection, and transferring the data. The compressed data arrives at the server as output.

[0424] Step 4:

[0425] The server receives the transferred data and decompresses it. It receives compressed data as input and restores it to its original information format. Specifically, it decompresses the received gzip file and returns the video data and sensor data to their original state.

[0426] Step 5:

[0427] The server preprocesses the restored data into a format that can be input to the generative AI model and emotion analysis engine. It receives the decompressed data as input and converts it into a format for analysis. Specifically, it splits the video data into frames, converts the sensor data into time-series data, and formats the facial expression data for analysis. The preprocessed data is obtained as output.

[0428] Step 6:

[0429] The server inputs the preprocessed data into the generative AI model and emotion analysis engine, which analyzes customer behavior, employee activities, and user emotions. The preprocessed data is received as input and analyzed. Specifically, the generative AI model analyzes the customer's length of stay, the timing of employee calls, changes in facial expressions, etc. The analysis results are obtained as output.

[0430] Step 7:

[0431] The server generates improvement proposals for store operations based on the analysis results. It receives the analysis results as input and creates improvement proposals based on them. Specifically, it proposes optimal product placement and employee action plans based on the results of the generative AI model and sentiment analysis engine. The generated improvement proposals are obtained as output.

[0432] Step 8:

[0433] The device notifies the administrator of the improvement proposal received from the server. The device receives the generated improvement proposal as input and communicates it to the administrator. Specifically, the device displays and notifies the administrator of the improvement proposal through a smartphone app (for example, one using Flutter or React Native). The output is that notification to the administrator has been completed.

[0434] Step 9:

[0435] The user implements the notified improvement suggestions and re-enters the results into the device. The results of the suggestions are received as input and fed back to the system. Specific operations include moving products or changing employee behavior in accordance with the presented improvement suggestions, and recording the results in the app. Feedback data is obtained as output.

[0436] Step 10:

[0437] The server collects data again and monitors the effectiveness of the proposed improvements. It receives feedback data as input and evaluates its effectiveness. Specifically, it analyzes the new data after the improvements and evaluates customer behavior, sales data, and customer satisfaction. The output is performance data after the improvements.

[0438] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0439] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0440] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0441] [Second embodiment]

[0442] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0443] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0444] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0445] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0446] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0447] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0448] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0449] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0450] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0451] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0452] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0453] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0454] In order to implement the present invention, a system can be constructed and operated in the following manner.

[0455] System Overview

[0456] This system collects data from cameras and sensors installed inside and outside the store, and uses a generative AI model to analyze customer behavior and store staff activities, thereby proposing improvements to store operations.The main components are as follows:

[0457] 1. Cameras and sensors: These are placed inside and outside the store to capture the movements of customers and store staff.

[0458] 2. Local terminal: collects data from cameras and sensors and performs primary processing and data cleaning.

[0459] 3. Server: Receives data uploaded from local devices and analyzes it using generative AI models.

[0460] 4. User (store manager): Receives improvement suggestions from the server and implements specific actions.

[0461] Explanation of program processing

[0462] Data collection

[0463] 1. The device collects real-time data from cameras and sensors installed inside and outside the store. The cameras capture images, and the sensors detect people's location and movement.

[0464] Example: A camera captures a customer pausing in front of a shelf, and a sensor records the duration of their visit.

[0465] Data storage and cleaning

[0466] 2. The data collected by the device is stored in local storage, and noise and incomplete data are filtered out during storage.

[0467] Example: Removing noise from video data and filling in missing parts of sensor data.

[0468] Uploading data

[0469] 3. The device compresses the cleaned data and uploads it to the server using the specified secure protocol.

[0470] Example: Compress data in gzip format and send it to the server using HTTPS.

[0471] Data analysis

[0472] 4. The server decompresses and derealizes the received data, performs preprocessing, and then inputs it into the generative AI model, which analyzes customer behavior and store clerk behavior.

[0473] Example: Extracting frames from video data, converting sensor data into time series data, and inputting it into a generative AI model.

[0474] 5. The server obtains the analysis results from the generative AI model and generates improvement proposals based on them.

[0475] Example: Identify shelves where customers spend a lot of time but products are not selling, and suggest changing the placement of those products.

[0476] Notification of improvement proposals

[0477] 6. The device notifies the store manager of the improvement suggestions received from the server, for example, via a mobile device or dashboard UI.

[0478] Example: A dashboard might suggest, "Move product A from shelf X to shelf Y."

[0479] Implementing improvement proposals and monitoring their effectiveness

[0480] 7. The user implements the improvement suggestions received and collects the results again, allowing the effectiveness of the improvement suggestions to be monitored.

[0481] Example: After changing product placement, new data is collected again through cameras and sensors to analyze changes in sales.

[0482] Specific examples

[0483] 1. The device collects behavioral data of 500 customers every day, filters out noise, and stores it in local storage.

[0484] 2. The device uploads the compressed data to the server overnight.

[0485] 3. The server decompresses and preprocesses the data and runs it through a generative AI model, detecting, for example, that a particular product is not selling well but many customers still stop in front of it.

[0486] 4. The server generates and notifies the user of a suggestion to change the placement of the products. The user then moves the products based on the suggestion.

[0487] 5. The server monitors subsequent sales data to assess the effectiveness of the changes.

[0488] In this way, by implementing the present invention, sales and customer experience in store operations can be effectively improved.

[0489] The processing flow will be explained below.

[0490] Step 1:

[0491] The device collects data in real time from multiple cameras and sensors installed inside and outside the store.

[0492] How it works: Cameras capture footage of customers, and sensors collect data like their location and movements.

[0493] Step 2:

[0494] The data acquired by the device is temporarily stored in local storage.

[0495] Specific operation: Stores captured video frames and sensor data in a database or file system.

[0496] Step 3:

[0497] The device performs noise filtering on the stored data to remove unnecessary data.

[0498] Specific operation: Automatically eliminates low-resolution footage and falsely detected sensor data, leaving only the necessary data.

[0499] Step 4:

[0500] The terminal periodically compresses the filtered data and uploads it to the server.

[0501] What it does: It converts the data into a compressed format such as gzip and transfers it securely to the server over HTTPS.

[0502] Step 5:

[0503] The server de-realizes the uploaded data and restores it to its original form.

[0504] Specific operation: The received compressed file is decompressed and the original video data and sensor data are restored.

[0505] Step 6:

[0506] The server preprocesses the restored data and converts it into a format that can be input into the generative AI model.

[0507] Specific operation: Video data is divided into frames and sensor data is converted into time series data.

[0508] Step 7:

[0509] The server inputs the preprocessed data into a generative AI model to analyze customer behavior and store clerk behavior.

[0510] Specific operation: The AI ​​model analyzes the customer's length of stay, their movement in front of the product, and the timing of the store clerk's call.

[0511] Step 8:

[0512] The server generates suggestions for improving store operations based on the analysis results from the generated AI model.

[0513] Specific actions: Make specific suggestions such as changing product placement and training content for store staff.

[0514] Step 9:

[0515] The terminal notifies the store manager (user) of the improvement proposal received from the server.

[0516] Specific actions: Display suggestions on mobile devices and dashboards, clearly indicating the actions to be taken.

[0517] Step 10:

[0518] Implement the improvement suggestions received by the user and record the results.

[0519] Specific actions: Move products and adjust the actions of store staff, and enter the results into the system.

[0520] Step 11:

[0521] The server collects data again and monitors the effectiveness of the improvement suggestions.

[0522] Specific Actions: Analyze new customer behavior and sales data to evaluate performance after improvements.

[0523] Example 1

[0524] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0525] In traditional store operations, there is a lack of means to properly analyze customer behavior and employee activity and propose operational improvement measures in real time. This makes it difficult to optimize sales and improve customer experience. In addition, there are poor means to monitor the effectiveness of improvement proposals, making it difficult to quantitatively grasp the actual effects of improvements.

[0526] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0527] In this invention, the server includes means for acquiring data in real time from monitoring devices and detection devices, means for storing the data acquired from the devices in a local storage device and filtering out noise and incomplete data, means for compressing the filtered data and transmitting it to the server via a secure protocol, means for the server to decompress and de-realize the data received, pre-process it, and input it into a generative AI model to analyze customer behavior and employee behavior, and means for generating store operation improvement proposals from the analysis results and notifying the store manager. This makes it possible to dramatically improve the efficiency of store operation and enable real-time monitoring of the effects of improvement proposals and rapid feedback.

[0528] A "monitoring device" is a device that is installed in a target area and acquires video data in real time.

[0529] A "detection device" is a device that is installed in a target area and detects the location and movement of people and objects in real time.

[0530] A "local storage device" is a storage device for temporarily storing collected data.

[0531] "Noise" refers to unnecessary or incorrect information that gets mixed in during the data collection process.

[0532] "Incomplete data" is data that contains missing or incorrect information during the collection process.

[0533] "Filtering" is the process of removing noise and imperfections from collected data.

[0534] "Compression" is a process performed to reduce the volume of data.

[0535] A "secure protocol" is a communication method that ensures security when sending and receiving data.

[0536] The "server" is a central device that receives collected data and processes and analyzes it.

[0537] "Derealization" is the process of returning received data to its original format.

[0538] "Preprocessing" is the process of converting data into an appropriate format before feeding it into a generative AI model.

[0539] A "generative AI model" is an artificial intelligence model that analyzes data and recognizes customer and employee behavior.

[0540] "Analysis" is the process of extracting information for a specific purpose from collected data.

[0541] "Improvement proposals" are specific proposals based on the analysis results to improve the efficiency of store operations and sales.

[0542] "Store Manager" means the person responsible for managing the store's operations and implementing improvement suggestions.

[0543] "Notification" refers to the act of informing the store manager of the analysis results and improvement suggestions.

[0544] "Monitoring" is the process of continuously observing the effectiveness of implemented improvement proposals.

[0545] The present invention is a system that collects data from monitoring devices and detection devices installed inside and outside a store, analyzes the data, and proposes improvements to store operations. The following hardware and software are used to implement the present invention.

[0546] Hardware

[0547] 1. Surveillance devices (cameras): Install multiple cameras inside and outside the store to capture the movements of customers and employees in real time.

[0548] 2. Detection device (sensor): Detects people's movements and staying time and provides location data.

[0549] 3. Local storage device (local terminal): A device for temporarily storing data obtained from cameras and sensors.

[0550] 4. Server: A central processing unit that receives, decompresses, preprocesses, and analyzes data.

[0551] 5. Notification device: A mobile device or dashboard UI for notifying store managers of improvement suggestions.

[0552] software

[0553] 1. Image processing algorithm: Processes the image data collected from the camera and removes noise.

[0554] 2. Data filtering algorithms: Fill in and refine imperfections in sensor data.

[0555] 3. Compression software: Compress the data in gzip format.

[0556] 4. Secure Communication Protocol (HTTPS): A protocol for securely sending and receiving data to and from a server.

[0557] 5. Generative AI model: The artificial intelligence model used for analysis.

[0558] Specific flow of data

[0559] The terminal acquires data from monitoring and detection devices, denoises and filters it, and stores it in local storage. The data is then compressed and sent to the server via a secure protocol. The server decompresses and derealizes the received data, preprocesses it, and inputs it into a generative AI model. This model analyzes customer and employee behavior and generates improvement suggestions. Finally, the server sends these improvement suggestions to a notification terminal, which notifies the store manager. The user implements the improvement suggestions and monitors their effectiveness by collecting new data and reanalyzing it.

[0560] Specific examples

[0561] For example, specific improvement suggestions are made, such as identifying shelves with long customer dwell times but poor sales, and suggesting changes to their layout. Sample prompts are used in conjunction with these suggestions to analyze the generative AI model. Examples of prompts are shown below.

[0562] Example prompt sentence:

[0563] "Analyze specific shelf purchasing behavior based on current dwell time data."

[0564] In this way, the present invention realizes efficient store management and increased sales, and brings about significant improvements over conventional methods.

[0565] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0566] Step 1:

[0567] The terminal receives real-time data from the monitoring and detection devices. The inputs include video data from the monitoring devices and location data from the detection devices, which are required for subsequent processing.

[0568] How it works: Cameras continuously capture customer movements, and sensors record each customer's location and movements.

[0569] Step 2:

[0570] The data acquired by the device is stored in local storage. The input is the raw data acquired in step 1. The output is the stored data. During this process, noise removal and data filtering are performed.

[0571] Specific operation: Image processing algorithms are used to remove noise from video data and fill in missing data.

[0572] Step 3:

[0573] The device compresses the filtered data and sends it to the server via a secure protocol (e.g., HTTPS). The input is the clean data stored in step 2. The output is the compressed data sent to the server.

[0574] Specific operation: Data is compressed in gzip format and uploaded to the server periodically every night.

[0575] Step 4:

[0576] The server decompresses, derealizes, and preprocesses the received data. The input is the compressed data sent in step 3. The output is the preprocessed data.

[0577] Specific operation: The server decompresses the gzip file, divides the video data into frames, and converts the sensor data into time series data.

[0578] Step 5:

[0579] The server inputs the preprocessed data into the generative AI model to analyze customer and employee behavior. The input is the data preprocessed in step 4. The output is the analysis results.

[0580] Specific operation: The frame data and the shaped sensor data are input into an AI model, which then analyzes customer stay times and employee movements.

[0581] Step 6:

[0582] The server generates improvement proposals based on the analysis results. The input is the analysis results obtained in step 5. The output is improvement proposals.

[0583] Specific operation: Identify shelf locations with poor sales based on customer behavior data and suggest product relocation.

[0584] Step 7:

[0585] The server sends the generated improvement proposal to the terminal, which then notifies the user. The input is the improvement proposal generated in step 6. The output is the notified improvement proposal.

[0586] Specific action: The suggestions are displayed on your smartphone or dashboard, and you are notified of the necessary actions.

[0587] Step 8:

[0588] The user executes the improvement proposal received, which generates new data. The input is the improvement proposal notified in step 7. The output is the execution result.

[0589] Specific operation: The user changes, for example, product placement based on the suggestions.

[0590] Step 9:

[0591] The terminal collects new data after implementing the improvement proposal and sends it back to the server to monitor the effectiveness of the improvement proposal. The input is the new data generated in step 8. The output is the monitored effectiveness data.

[0592] Specific actions: Collect new customer behavior data, send it back to the server, and analyze it to evaluate the effectiveness of improvements.

[0593] (Application example 1)

[0594] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0595] Conventional store management systems have difficulty understanding the behavior of customers and store employees in real time and automatically generating and notifying effective improvement proposals. They also lack the means to quickly evaluate the effectiveness of proposals after their implementation and to make continuous improvements. As a result, store managers are unable to quickly and efficiently optimize store operations, and are often left behind in improving sales and customer satisfaction. This problem needs to be solved.

[0596] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0597] In this invention, the server includes means for acquiring data in real time from multiple cameras and sensors installed inside and outside the store, means for storing the data acquired from the cameras and sensors in local storage and filtering out noise and incomplete data, means for uploading the filtered data to the server, means for inputting the uploaded data into a generative AI model and analyzing customer behavior and store employee behavior, means for generating store operation improvement proposals from the analysis results and notifying the store manager, means for monitoring the operation data after the improvement proposals are implemented and evaluating the effectiveness of the improvement proposals, means for viewing the camera feeds in real time using a smartphone and a head-mounted display and checking the analysis results and improvement proposals, and means for receiving notifications of the improvement proposals via the smartphone and the head-mounted display. This makes it possible to analyze customer and store employee behavior in real time, quickly provide store operation improvement proposals, and continuously evaluate their effectiveness.

[0598] A "camera" is a device installed inside or outside a store that captures the movements of customers and store staff as video.

[0599] A "sensor" is a device installed inside and outside a store that detects people's location and movement.

[0600] "Local storage" is a storage device for temporarily storing data acquired from cameras and sensors.

[0601] "Noise filtering" is the process of removing inaccuracies and unnecessary information from data.

[0602] A "server" is a computer system that receives, analyzes, and stores data over a network.

[0603] A "generative AI model" is an artificial intelligence system that analyzes customer behavior and store clerk behavior and automatically generates improvement suggestions.

[0604] A "smartphone" is a small personal digital assistant (PDA) that can view camera feeds and receive notifications.

[0605] A "head-mounted display" is a device worn on the head that displays images in real time.

[0606] A "store manager" is a person who is responsible for running a store and takes action based on improvement proposals.

[0607] "Operational data" refers to data that indicates the operational status of a store, and includes sales information, customer behavior, and the like.

[0608] "Camera feed" is video data sent from a camera in real time.

[0609] "Improvement proposals" are specific proposals for improving store operations based on data analyzed by the generative AI model.

[0610] "Effectiveness evaluation" is the process of analyzing the results after implementing improvement proposals and determining their effectiveness.

[0611] As a specific embodiment for implementing this invention, a system is constructed that utilizes cameras and sensors installed inside and outside the store, a local terminal, a server, a smartphone, and a head-mounted display.

[0612] Hardware and software used

[0613] Camera: A device installed inside and outside a store that captures the movements of customers and store staff as video.

[0614] Sensor: A device installed inside and outside a store that detects people's location and movement.

[0615] Local device: A computer that temporarily stores and preprocesses data acquired from cameras and sensors.

[0616] Server: A computer system that receives, analyzes, and stores data.

[0617] Smartphone: A small personal digital assistant that allows you to view camera feeds and receive notifications.

[0618] Head-mounted display: A device worn on the head that displays images in real time.

[0619] Data processing and calculation

[0620] 1. Data Collection:

[0621] The device collects real-time data from cameras and sensors installed inside and outside the store. The cameras capture images, and the sensors detect people's location and movement.

[0622] Example: A camera captures a customer pausing in front of a shelf, and a sensor records the duration of their visit.

[0623] 2. Data Storage and Filtering:

[0624] The data collected by the device is stored in local storage and noise and incomplete data are filtered out.

[0625] Example: Removing noise from video data and filling in missing parts of sensor data.

[0626] 3. Data upload:

[0627] The device compresses the cleaned data and uploads it to the server using a secure protocol (e.g. HTTPS).

[0628] Example: Compress data in gzip format and send it to a server overnight.

[0629] 4. Data Analysis:

[0630] The server decompresses and derealizes the received data, performs preprocessing, and then inputs it into a generative AI model, which analyzes customer behavior and store clerk behavior.

[0631] Example: Extracting frames from video data, converting sensor data into time series data, and inputting it into a generative AI model.

[0632] 5. Generation and notification of improvement suggestions:

[0633] The server obtains the analysis results using the generative AI model, generates improvement suggestions based on them, and notifies the store manager.

[0634] Example: Identify shelves where customers spend a lot of time but products are not selling, and suggest changing the placement of those products.

[0635] 6. Effect monitoring:

[0636] The user implements the improvement suggestions received and collects the results again. The server then monitors the sales data and evaluates the effectiveness of the changes.

[0637] Example: After changing product placement, new data is collected again through cameras and sensors to analyze changes in sales.

[0638] 7. Smartphone and head-mounted display usage:

[0639] Users can use their smartphones and head-mounted displays to view real-time camera feeds and see analysis results and improvement suggestions.

[0640] Notification of improvement suggestions will be sent via smartphone and head-mounted display.

[0641] Example prompt: "Analyze customer dwell time and generate shelf relocation suggestions for underperforming products."

[0642] The above configuration makes it possible to improve the efficiency of store operations, and is expected to result in increased customer satisfaction and sales.

[0643] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0644] Step 1:

[0645] The terminal acquires data in real time from cameras and sensors installed inside and outside the store. Specifically, the cameras capture images, and the sensors detect the positions and movements of customers and store staff. The input is the camera images and sensor detection data, and the output is a combination of these data. For example, the camera captures a customer stopping in front of a shelf, and the sensor records the duration of their stay.

[0646] Step 2:

[0647] The data collected by the device is stored in local storage and filtered for noise and incomplete data. Specifically, it removes noise from the video data and complements missing sensor data. The input is camera video and sensor detection data, and the output is clean, filtered data. For example, it improves image quality when the video is unclear and interpolates continuous sensor operation data.

[0648] Step 3:

[0649] The terminal compresses the cleaned data and uploads it to a server using a secure protocol (e.g. HTTPS). The input is the filtered data and the output is the compressed data and confirmation of its destination. For example, the data may be compressed in gzip format and sent to the server overnight.

[0650] Step 4:

[0651] The server decompresses and deserializes the data it receives. Specifically, it converts compressed data back to its original format so that it can be analyzed. The input is compressed data, and the output is decompressed data. For example, it decompresses video data and sensor data compressed with Gzip.

[0652] Step 5:

[0653] The server preprocesses the decompressed and deserialized data and inputs it into the generative AI model. Specifically, it extracts the video data frame by frame and converts the sensor data into time series data. The input is the decompressed data, and the output is the input data for the generative AI model. For example, the video data can be converted into an analyzable format frame by frame, and sensor information converted into time series data can be created.

[0654] Step 6:

[0655] The server uses a generative AI model to analyze customer and store clerk behavior and generates suggestions for improving store operations based on the results. The input is the data input to the generative AI model, and the output is the analysis results and suggestions for improvement. For example, it identifies shelves where customers spend a lot of time but products are not selling, and suggests changing the placement of those products.

[0656] Step 7:

[0657] The server notifies the store manager of the improvement suggestion via the smartphone and head-mounted display. The input is the generated improvement suggestion, and the output is a confirmation that the notification has been sent. For example, the smartphone and head-mounted display display a suggestion such as "Please move product A from shelf X to shelf Y."

[0658] Step 8:

[0659] The user implements the improvement suggestions they receive, and the results are collected again using cameras and sensors. The input is the implementation results of the improvement suggestions, and the output is the operational data after the changes. For example, after changing product placement, new data is collected again using cameras and sensors to analyze changes in sales.

[0660] Step 9:

[0661] Based on the data recollected by the server, the effectiveness of the proposed improvements is evaluated and further proposals are made as necessary. The input is the operational data after the changes, and the output is an evaluation of the effectiveness and new proposals for improvements. For example, the system can confirm the increase in sales after changing product placement and further propose optimization of customer flow.

[0662] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0663] In order to implement the present invention, a system can be constructed and operated in the following manner.

[0664] System Overview

[0665] This system collects data from cameras and sensors installed inside and outside the store, and uses a generative AI model and emotion engine to analyze customer behavior, store clerk activities, and even user emotions, to propose improvements to store operations.The main components are as follows:

[0666] 1. Cameras and sensors: Placed inside and outside the store, they capture the movements of customers and store staff, as well as user emotions.

[0667] 2. Local terminal: collects data from cameras and sensors and performs primary processing and data cleaning.

[0668] 3. Server: Receives data uploaded from local devices and analyzes it using generative AI models and emotion engines.

[0669] 4. User (store manager): Receives improvement suggestions from the server and implements specific actions.

[0670] Explanation of program processing

[0671] Data collection

[0672] 1. The device collects data in real time from cameras and sensors installed inside and outside the store. The cameras capture images, and the sensors collect data such as people's location and movements. In addition, an emotion engine analyzes the user's emotions from their facial expressions and voice.

[0673] Example: A camera captures a customer's movements and facial expressions as they pause in front of a shelf, and sensors record their dwell time and emotions.

[0674] Data storage and cleaning

[0675] 2. The device temporarily stores the collected data in local storage, filtering out noise and incomplete data when storing.

[0676] Example: Removing noise from video data and filling in missing parts of sensor data.

[0677] Uploading data

[0678] 3. The device compresses the filtered data and uploads it to the server using the specified secure protocol.

[0679] Example: Compress data in gzip format and send it to the server using HTTPS.

[0680] Data analysis

[0681] 4. The server de-realizes the uploaded data and restores it to its original form.

[0682] Specific operation: The received compressed file is decompressed and the original video data and sensor data are restored.

[0683] 5. The server preprocesses the recovered data and converts it into a format that can be input to the generative AI model and emotion engine.

[0684] Examples: Splitting video data into frames, converting sensor data into time series data, and formatting facial expression data for analysis.

[0685] 6. The server inputs the preprocessed data into the generative AI model and emotion engine to analyze customer behavior, store clerk behavior, and user emotions.

[0686] Example: The AI ​​model analyzes the customer's length of stay, their movement in front of the product, the timing of the store clerk's call, changes in facial expression, etc.

[0687] 7. The server generates suggestions for improving store operations based on the analysis results from the generated AI model and emotion engine. The suggestions also take into account the user's emotional state.

[0688] Example: An emotion engine infers customer satisfaction from their facial expressions, identifies products and placements that cause dissatisfaction, and suggests areas for improvement.

[0689] Notification of improvement proposals

[0690] 8. The device notifies the store manager of the improvement suggestions received from the server, for example, via a mobile device or dashboard UI.

[0691] Example: The dashboard might say, "Move product A from shelf X to shelf Y and have the sales associate provide specific advice to the customer."

[0692] Implementing improvement proposals and monitoring their effectiveness

[0693] 9. Implement the improvement suggestions received from the user and record the results.

[0694] Example: Adjusting merchandise movements and store staff actions and inputting the results into the system.

[0695] 10. The server collects data again and monitors the effectiveness of the proposed improvements.

[0696] Example: Analyzing new customer behavior data, sales data, and customer facial expression data to evaluate performance and customer satisfaction after improvements.

[0697] Specific examples

[0698] 1. The device collects behavioral and emotional data of 500 customers every day, filters out noise, and stores it in local storage.

[0699] 2. The device uploads the compressed data to the server overnight.

[0700] 3. The server decompresses and preprocesses the data, then analyzes it with a generative AI model and emotion engine. For example, it might detect that a particular product is not selling well, but many customers stop in front of it with unhappy expressions.

[0701] 4. The server generates improvement proposals, including suggestions for changing product placement and training for store staff, and notifies them.

[0702] 5. The user implements the suggestions and collects the performance results again.

[0703] 6. The server analyzes the new data, monitors the effectiveness of the suggested improvements, and makes further recommendations.

[0704] In this way, by implementing the present invention, it is possible to effectively improve sales and customer experience in store operations, as well as increase customer satisfaction.

[0705] The processing flow will be explained below.

[0706] Step 1:

[0707] The device collects data in real time from multiple cameras and sensors installed inside and outside the store.

[0708] How it works: Cameras capture footage of customers, and sensors collect their location, movement, and user audio data.

[0709] Step 2:

[0710] The data acquired by the device is temporarily stored in local storage.

[0711] Specific behavior: Stores captured video frames, sensor data, and audio data in a database or file system.

[0712] Step 3:

[0713] The device performs noise filtering on the stored data to remove unnecessary data.

[0714] Specific operation: Filters out low-resolution video, misdetected sensor data, and audio data containing noise, leaving only the necessary data.

[0715] Step 4:

[0716] The device compresses the filtered data and uploads it to the server.

[0717] Specific operation: Compress the data into gzip format and send it securely to the server via HTTPS.

[0718] Step 5:

[0719] The server unpacks the uploaded data and restores it to its original format.

[0720] Specific operation: The received compressed file is decompressed and restored to the original video data, sensor data, and audio data.

[0721] Step 6:

[0722] The server preprocesses the recovered data and converts it into a format that can be input to the generative AI model and emotion engine.

[0723] Specific operations: Video data is divided into frames, sensor data is organized into time series data, and audio data is converted into text.

[0724] Step 7:

[0725] The server inputs the preprocessed data into a generative AI model and emotion engine to analyze customer behavior, store clerk behavior, and user emotions.

[0726] Specific operation: The generative AI model analyzes the customer's dwell time and movement in front of the product, and the emotion engine analyzes the customer's emotions from facial expressions and voice data.

[0727] Step 8:

[0728] The server generates improvement proposals for store operations based on the analysis results.

[0729] Specific operation: The results of the generative AI model and the emotion engine are combined to compile suggestions such as changing product placement, improving the way store staff serve customers, and providing services based on customer emotions.

[0730] Step 9:

[0731] The terminal notifies the store manager (user) of the improvement proposal received from the server.

[0732] Specific actions: Display suggestions on mobile devices and dashboard UI, and clearly indicate specific actions.

[0733] Step 10:

[0734] Implement the improvement suggestions received by the user and record the results.

[0735] Specific actions: Change product placement, give instructions to store staff, implement measures to improve customer service, and feed the results back into the system.

[0736] Step 11:

[0737] The server collects data again and monitors the effectiveness of the improvement suggestions.

[0738] Specific actions: Analyze new customer behavior data, sales data, and customer facial expression data to evaluate the effectiveness of implemented improvement measures.

[0739] Example 2

[0740] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0741] Conventional customer behavior analysis systems for store operations are limited to providing feedback on customer behavior data and store clerk responses, and are unable to accurately grasp the user's emotional state and reflect it in improvement proposals. Furthermore, the analysis targets noisy or incomplete data, resulting in reduced proposal accuracy. The present invention aims to solve these problems and provide more accurate store operation improvement proposals.

[0742] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for uploading the filtered data to the server, a means for derealizing the uploaded data, preprocessing it, and converting it into an analyzable format, and a means for inputting the preprocessed data into a generative AI model and an emotion analysis engine, analyzing customer behavior and clerk behavior, and estimating user emotions. This enables highly accurate data analysis and improvement suggestions that include the user's emotional state.

[0743] A "camera" is a device for capturing video data.

[0744] A "sensor" is a device that detects physical changes and outputs them as data, specifically one that detects a person's position or movement.

[0745] "Local storage" is an internal storage device for temporarily storing data.

[0746] "Filtering" refers to the process of removing noise and imperfections from data.

[0747] "Upload" refers to the process of sending data from a local terminal to a server.

[0748] "Deserialization" is the process of returning serialized data to its original form.

[0749] "Preprocessing" refers to the initial processing of data to prepare it in an analyzable format.

[0750] A "generative AI model" is an algorithm that uses machine learning to analyze data and generate a specific result.

[0751] An "emotion analysis engine" is an algorithm or system for inferring emotions from a user's facial expressions and voice.

[0752] "Customer behavior" refers to data on customer behavior such as location, movement patterns, and length of stay within the store.

[0753] "Employee behavior" refers to data on employee behavior such as their location within the store, movement patterns, and how they interact with customers.

[0754] "Analysis" refers to the act of processing data to extract useful information and patterns.

[0755] "Improvement proposals" refer to proposals based on analysis results aimed at optimizing store operations and customer service.

[0756] "Notification" refers to the process of communicating information to store managers, specifically via mobile devices or dashboards.

[0757] The following describes in detail an embodiment of the present invention: The present invention is a system that utilizes data from cameras and sensors installed inside and outside a store, and analyzes customer behavior, store clerk behavior, and user emotions using a generative AI model and an emotion analysis engine.

[0758] System Overview

[0759] The system mainly includes the following hardware and software:

[0760] Cameras: Multiple cameras are installed inside and outside the store to capture video data.

[0761] Sensors: There are infrared sensors and various motion sensors that collect location information of customers and store staff.

[0762] Device: Stores collected data in local storage and filters out noise and incomplete data.

[0763] Server: De-realizes the filtered data, pre-processes it, and converts it into an analyzable format. It then analyzes it using a generative AI model and sentiment analysis engine to generate improvement suggestions.

[0764] Data collection and cleaning

[0765] The device collects data in real time from cameras and sensors. For example, the camera captures video at 30 frames per second, and sensors acquire location data of customers and store staff. This data is stored in local storage, where noise and incomplete data are filtered out. For example, ghost noise in the video data is removed and incomplete sensor data is complemented.

[0766] Data upload and de-realization

[0767] The device compresses the filtered data in gzip format and uploads it to the server via secure communication using the HTTPS protocol. The server then decompresses and derealizes the received data. For example, it splits the decompressed video data into frames and stores the sensor data in a database.

[0768] Preprocessing and analysis

[0769] The server preprocesses the derealized data and prepares it in a format that can be analyzed by the generative AI model and emotion analysis engine. This includes resizing the video data, extracting frames, and normalizing the sensor data. For example, the server resizes the video data to 256x256 pixels, which is easier for the AI ​​model to handle, and fills in missing parts of the sensor data to prepare it as time-series data.

[0770] The server inputs the preprocessed data into a generative AI model and an emotion analysis engine to analyze customer behavior, store clerk behavior, and user emotions. The AI ​​model analyzes the customer's length of stay, their movement, and the timing of the clerk's interactions, while the emotion analysis engine estimates the customer's emotional state based on facial expressions and voice data. For example, the generative AI model analyzes the customer's length of stay, and the emotion analysis engine analyzes the customer's dissatisfied facial expression.

[0771] Improvement suggestions and notifications

[0772] The server generates suggestions for improving store operations based on the analysis results and the customer's emotional state. For example, suggestions may be generated that include changing product placement or specific guidelines for store staff behavior. These suggestions are notified to store managers via their devices. Notification methods include a dashboard UI or a mobile device. As a specific example, the dashboard may display a message such as, "Move product A from shelf X to shelf Y, and have store staff provide specific advice to customers."

[0773] Implementing improvement proposals and monitoring their effectiveness

[0774] The user implements the improvement proposal and records the results. For example, changing the product layout and adjusting the behavior of the store clerk. The data after the improvement is collected again and entered into the system, allowing the server to analyze it again, monitor the effectiveness of the proposal, and make further improvement proposals. For example, new customer behavior data, sales data, and customer satisfaction data are evaluated, and feedback is provided for further improvements.

[0775] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0776] Step 1:

[0777] The device acquires data in real time from the camera and sensors. The input data is video data from the camera and position and movement data from the sensors. Specifically, the camera captures video at 30 frames per second, and the sensors acquire position information of customers and store staff. The output is raw video data and sensor data.

[0778] Step 2:

[0779] The data collected by the device is temporarily stored in local storage and noise and incomplete data are filtered. The input data are the raw video data and sensor data acquired in the previous step. Specifically, the process removes ghost noise contained in the video data and complements incomplete sensor data. The output is filtered video data and sensor data.

[0780] Step 3:

[0781] The terminal compresses the filtered data in gzip format and uploads it to the server using the HTTPS protocol. The input data is the filtered video data and sensor data. Specifically, the data is compressed in gzip format and transmitted securely using HTTPS. The output is the compressed data uploaded to the server.

[0782] Step 4:

[0783] The server de-realizes the uploaded data and restores it to its original format. The input data is compressed video data and sensor data. Specifically, it decompresses the received gzip format file and restores it to video data and sensor data. The output is the decompressed and restored video data and sensor data.

[0784] Step 5:

[0785] The server preprocesses the restored data and converts it into a format that can be analyzed by the generative AI model and emotion analysis engine. The input data is the derealized video data and sensor data. Specifically, the server splits the video data into frames and resizes it to 256x256 pixels. It also converts the sensor data into time-series data and extracts facial expression and movement features. The output is the preprocessed data.

[0786] Step 6:

[0787] The server inputs the preprocessed data into a generative AI model and emotion analysis engine, which analyzes customer behavior, store clerk behavior, and user emotions. The input data is preprocessed video data and sensor data. Specifically, the generative AI model analyzes customer movements and length of stay, and the emotion analysis engine infers the user's emotions from facial expression data. The output is the analysis results.

[0788] Step 7:

[0789] The server generates improvement proposals for store operations based on the analysis results. The input data are the analysis results. Specific operations include identifying products and placements that cause high customer dissatisfaction from the analysis results and proposing improvements accordingly. The output is specific improvement proposals.

[0790] Step 8:

[0791] The device notifies the user of the improvement proposal received from the server. The input data is the generated improvement proposal. As a specific operation, the proposal content is displayed through the dashboard UI or mobile device. The output is the improvement proposal notified to the user.

[0792] Step 9:

[0793] The user implements the improvement proposals received and records the results. The input data is the notified improvement proposal. Specific actions include moving products and adjusting the actions of store clerks, and inputting the resulting new behavioral and sales data into the system. The output is the implemented improvement proposals and new data.

[0794] Step 10:

[0795] The server re-analyzes the newly collected data and monitors the effectiveness of the proposed improvements. The input data is the new data after execution. Specific operations include evaluating the analysis results and determining how effective the proposals were. Based on this, further improvement proposals are made. The output is an evaluation of the effectiveness of the proposed improvements and new proposals.

[0796] (Application example 2)

[0797] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0798] In recent years, improving store operation efficiency and customer satisfaction have become important issues in the retail industry, but conventional methods have difficulty in conducting detailed analysis of customer behavior and emotions, resulting in a lack of concrete improvement proposals based on this analysis.In addition, while employee activities need to be monitored and supervised efficiently, current systems are unable to adequately achieve this.

[0799] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for acquiring information in real time from observation devices and sensing devices, means for saving the acquired information in a local storage device and filtering out useless or incomplete data, means for transferring the filtered information to a computer, means for inputting the transferred information into a generative AI model and a sentiment analysis engine to analyze customer behavior, employee activities, and user emotions, and means for generating store operation improvement proposals from the analysis results and notifying the manager. This makes it possible to perform detailed analysis of customer behavior, employee activities, and user emotions and provide specific and effective store operation improvement proposals.

[0800] "Observation devices" are devices installed to monitor and record the behavior and situations of customers and employees.

[0801] A "sensing device" is a device that collects information about the location and movement of customers and employees, as well as environmental information.

[0802] "Information" refers to data acquired in real time from observation and sensing devices, and includes video, audio, movement patterns, emotional states, and the like.

[0803] A "local storage device" is a device for temporarily storing information obtained from observation and sensing devices.

[0804] "Filtering" is the process of removing useless or incomplete data from stored information.

[0805] "Computer" refers to a server or cloud computing system for analyzing information transferred from local storage.

[0806] A "generative AI model" is a machine learning model trained to analyze customer and employee behavior and emotions.

[0807] An "emotion analysis engine" is software that analyzes emotions from a user's facial expressions and voice.

[0808] "Analysis results" refers to information about customer behavior, employee activities, and emotional states obtained by the generative AI model and sentiment analysis engine.

[0809] "Improvement proposals" are specific action plans for optimizing store operations and improving customer satisfaction based on the analysis results.

[0810] A "manager" is a person who is responsible for overseeing store operations and implementing improvement suggestions.

[0811] Summary of the Invention

[0812] This invention is a system that uses a generative AI model and an emotion analysis engine to analyze customer behavior, employee activity, and user emotions based on data collected from observation and sensing devices in a physical store, and provides suggestions for improving store operations. This system can improve customer satisfaction and streamline store operations.

[0813] Detailed system description

[0814] Data collection

[0815] The system collects information in real time from observation devices (cameras) and sensing devices (sensors) installed inside and outside the store. The observation devices record the behavior of customers and employees, while the sensing devices collect information on their location, behavior, and environment.

[0816] Storing and Filtering Data

[0817] The collected information is stored on a local storage device (e.g., a device such as a Raspberry Pi). When stored, useless or incomplete data is filtered out, leaving only the necessary information.

[0818] Data Transfer

[0819] The filtered information is converted into a compressed format (eg, gzip format) and transferred to a computer (a server or a cloud computing system) via a secure communication means (eg, HTTPS protocol).

[0820] Data analysis

[0821] The server receives the transferred information and inputs it into a generative AI model and sentiment analysis engine, which analyzes customer behavior, employee activity, and the user's emotional state. The generative AI model is trained using machine learning frameworks such as TensorFlow and PyTorch, and the sentiment analysis engine uses Microsoft Azure Emotion API and Amazon Rekognition.

[0822] Generate and notify improvement suggestions

[0823] Based on the analysis results, the server generates specific suggestions for improving store operations. These suggestions are notified to the manager in real time, who then acts accordingly. Notifications are sent via a smartphone app (e.g., an app developed based on Flutter or React Native).

[0824] Examples of specific examples and prompts

[0825] For example, if many customers stop in front of a certain product but do not end up purchasing it, the generative AI model will determine that there is a problem with the product's placement or explanation, and will suggest moving the product or improving the explanation.In addition, if the emotion analysis engine detects low satisfaction from customers' facial expressions, it will generate suggestions to encourage employees to take a more proactive approach.

[0826] An example prompt is:

[0827] "Customers who pause in front of product A spend a lot of time there, but the percentage of them actually purchasing is low. The sentiment analysis engine indicates that customer satisfaction is low. Therefore, move product A from shelf X to shelf Y, and have an associate provide specific advice to the customer."

[0828] This allows for detailed analysis of customer behavior, employee activities, and user sentiment, making it possible to provide specific and effective suggestions for improving store operations.

[0829] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0830] Step 1:

[0831] The terminal acquires information in real time from observation devices (cameras) and detection devices (sensors) installed inside and outside the store. The input is data from each observation device and detection device, which is received by the terminal. Specifically, the camera records customer behavior, and the sensor collects movement and location information.

[0832] Step 2:

[0833] The device stores the acquired information in a local storage device. Here, it receives the acquired raw data as input and filters out noise and incomplete data. Specifically, it removes noise from the video data and fills in missing parts of the sensor data. The output is clean, filtered data.

[0834] Step 3:

[0835] The terminal converts the filtered data into a compressed format and transfers it to the server. It receives clean data as input and compresses it, for example, using gzip format. It then sends it to the server using a secure communication method (for example, HTTPS protocol). The specific operations include compressing the data, establishing a secure network connection, and transferring the data. The compressed data arrives at the server as output.

[0836] Step 4:

[0837] The server receives the transferred data and decompresses it. It receives compressed data as input and restores it to its original information format. Specifically, it decompresses the received gzip file and returns the video data and sensor data to their original state.

[0838] Step 5:

[0839] The server preprocesses the restored data into a format that can be input to the generative AI model and emotion analysis engine. It receives the decompressed data as input and converts it into a format for analysis. Specifically, it splits the video data into frames, converts the sensor data into time-series data, and formats the facial expression data for analysis. The preprocessed data is obtained as output.

[0840] Step 6:

[0841] The server inputs the preprocessed data into the generative AI model and emotion analysis engine, which analyzes customer behavior, employee activities, and user emotions. The preprocessed data is received as input and analyzed. Specifically, the generative AI model analyzes the customer's length of stay, the timing of employee calls, changes in facial expressions, etc. The analysis results are obtained as output.

[0842] Step 7:

[0843] The server generates improvement proposals for store operations based on the analysis results. It receives the analysis results as input and creates improvement proposals based on them. Specifically, it proposes optimal product placement and employee action plans based on the results of the generative AI model and sentiment analysis engine. The generated improvement proposals are obtained as output.

[0844] Step 8:

[0845] The device notifies the administrator of the improvement proposal received from the server. The device receives the generated improvement proposal as input and communicates it to the administrator. Specifically, the device displays and notifies the administrator of the improvement proposal through a smartphone app (for example, one using Flutter or React Native). The output is that notification to the administrator has been completed.

[0846] Step 9:

[0847] The user implements the notified improvement suggestions and re-enters the results into the device. The results of the suggestions are received as input and fed back to the system. Specific operations include moving products or changing employee behavior in accordance with the presented improvement suggestions, and recording the results in the app. Feedback data is obtained as output.

[0848] Step 10:

[0849] The server collects data again and monitors the effectiveness of the proposed improvements. It receives feedback data as input and evaluates its effectiveness. Specifically, it analyzes the new data after the improvements and evaluates customer behavior, sales data, and customer satisfaction. The output is performance data after the improvements.

[0850] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0851] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0852] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0853] [Third embodiment]

[0854] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0855] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0856] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0857] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0858] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0859] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0860] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0861] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0862] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0863] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0864] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0865] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0866] In order to implement the present invention, a system can be constructed and operated in the following manner.

[0867] System Overview

[0868] This system collects data from cameras and sensors installed inside and outside the store, and uses a generative AI model to analyze customer behavior and store staff activities, thereby proposing improvements to store operations.The main components are as follows:

[0869] 1. Cameras and sensors: These are placed inside and outside the store to capture the movements of customers and store staff.

[0870] 2. Local terminal: collects data from cameras and sensors and performs primary processing and data cleaning.

[0871] 3. Server: Receives data uploaded from local devices and analyzes it using generative AI models.

[0872] 4. User (store manager): Receives improvement suggestions from the server and implements specific actions.

[0873] Explanation of program processing

[0874] Data collection

[0875] 1. The device collects real-time data from cameras and sensors installed inside and outside the store. The cameras capture images, and the sensors detect people's location and movement.

[0876] Example: A camera captures a customer pausing in front of a shelf, and a sensor records the duration of their visit.

[0877] Data storage and cleaning

[0878] 2. The data collected by the device is stored in local storage, and noise and incomplete data are filtered out during storage.

[0879] Example: Removing noise from video data and filling in missing parts of sensor data.

[0880] Uploading data

[0881] 3. The device compresses the cleaned data and uploads it to the server using the specified secure protocol.

[0882] Example: Compress data in gzip format and send it to the server using HTTPS.

[0883] Data analysis

[0884] 4. The server decompresses and derealizes the received data, performs preprocessing, and then inputs it into the generative AI model, which analyzes customer behavior and store clerk behavior.

[0885] Example: Extracting frames from video data, converting sensor data into time series data, and inputting it into a generative AI model.

[0886] 5. The server obtains the analysis results from the generative AI model and generates improvement proposals based on them.

[0887] Example: Identify shelves where customers spend a lot of time but products are not selling, and suggest changing the placement of those products.

[0888] Notification of improvement proposals

[0889] 6. The device notifies the store manager of the improvement suggestions received from the server, for example, via a mobile device or dashboard UI.

[0890] Example: A dashboard might suggest, "Move product A from shelf X to shelf Y."

[0891] Implementing improvement proposals and monitoring their effectiveness

[0892] 7. The user implements the improvement suggestions received and collects the results again, allowing the effectiveness of the improvement suggestions to be monitored.

[0893] Example: After changing product placement, new data is collected again through cameras and sensors to analyze changes in sales.

[0894] Specific examples

[0895] 1. The device collects behavioral data of 500 customers every day, filters out noise, and stores it in local storage.

[0896] 2. The device uploads the compressed data to the server overnight.

[0897] 3. The server decompresses and preprocesses the data and runs it through a generative AI model, detecting, for example, that a particular product is not selling well but many customers still stop in front of it.

[0898] 4. The server generates and notifies the user of a suggestion to change the placement of the products. The user then moves the products based on the suggestion.

[0899] 5. The server monitors subsequent sales data to assess the effectiveness of the changes.

[0900] In this way, by implementing the present invention, sales and customer experience in store operations can be effectively improved.

[0901] The processing flow will be explained below.

[0902] Step 1:

[0903] The device collects data in real time from multiple cameras and sensors installed inside and outside the store.

[0904] How it works: Cameras capture footage of customers, and sensors collect data like their location and movements.

[0905] Step 2:

[0906] The data acquired by the device is temporarily stored in local storage.

[0907] Specific operation: Stores captured video frames and sensor data in a database or file system.

[0908] Step 3:

[0909] The device performs noise filtering on the stored data to remove unnecessary data.

[0910] Specific operation: Automatically eliminates low-resolution footage and falsely detected sensor data, leaving only the necessary data.

[0911] Step 4:

[0912] The terminal periodically compresses the filtered data and uploads it to the server.

[0913] What it does: It converts the data into a compressed format such as gzip and transfers it securely to the server over HTTPS.

[0914] Step 5:

[0915] The server de-realizes the uploaded data and restores it to its original form.

[0916] Specific operation: The received compressed file is decompressed and the original video data and sensor data are restored.

[0917] Step 6:

[0918] The server preprocesses the restored data and converts it into a format that can be input into the generative AI model.

[0919] Specific operation: Video data is divided into frames and sensor data is converted into time series data.

[0920] Step 7:

[0921] The server inputs the preprocessed data into a generative AI model to analyze customer behavior and store clerk behavior.

[0922] Specific operation: The AI ​​model analyzes the customer's length of stay, their movement in front of the product, and the timing of the store clerk's call.

[0923] Step 8:

[0924] The server generates suggestions for improving store operations based on the analysis results from the generated AI model.

[0925] Specific actions: Make specific suggestions such as changing product placement and training content for store staff.

[0926] Step 9:

[0927] The terminal notifies the store manager (user) of the improvement proposal received from the server.

[0928] Specific actions: Display suggestions on mobile devices and dashboards, clearly indicating the actions to be taken.

[0929] Step 10:

[0930] Implement the improvement suggestions received by the user and record the results.

[0931] Specific actions: Move products and adjust the actions of store staff, and enter the results into the system.

[0932] Step 11:

[0933] The server collects data again and monitors the effectiveness of the improvement suggestions.

[0934] Specific Actions: Analyze new customer behavior and sales data to evaluate performance after improvements.

[0935] Example 1

[0936] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0937] In traditional store operations, there is a lack of means to properly analyze customer behavior and employee activity and propose operational improvement measures in real time. This makes it difficult to optimize sales and improve customer experience. In addition, there are poor means to monitor the effectiveness of improvement proposals, making it difficult to quantitatively grasp the actual effects of improvements.

[0938] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0939] In this invention, the server includes means for acquiring data in real time from monitoring devices and detection devices, means for storing the data acquired from the devices in a local storage device and filtering out noise and incomplete data, means for compressing the filtered data and transmitting it to the server via a secure protocol, means for the server to decompress and de-realize the data received, pre-process it, and input it into a generative AI model to analyze customer behavior and employee behavior, and means for generating store operation improvement proposals from the analysis results and notifying the store manager. This makes it possible to dramatically improve the efficiency of store operation and enable real-time monitoring of the effects of improvement proposals and rapid feedback.

[0940] A "monitoring device" is a device that is installed in a target area and acquires video data in real time.

[0941] A "detection device" is a device that is installed in a target area and detects the location and movement of people and objects in real time.

[0942] A "local storage device" is a storage device for temporarily storing collected data.

[0943] "Noise" refers to unnecessary or incorrect information that gets mixed in during the data collection process.

[0944] "Incomplete data" is data that contains missing or incorrect information during the collection process.

[0945] "Filtering" is the process of removing noise and imperfections from collected data.

[0946] "Compression" is a process performed to reduce the volume of data.

[0947] A "secure protocol" is a communication method that ensures security when sending and receiving data.

[0948] The "server" is a central device that receives collected data and processes and analyzes it.

[0949] "Derealization" is the process of returning received data to its original format.

[0950] "Preprocessing" is the process of converting data into an appropriate format before feeding it into a generative AI model.

[0951] A "generative AI model" is an artificial intelligence model that analyzes data and recognizes customer and employee behavior.

[0952] "Analysis" is the process of extracting information for a specific purpose from collected data.

[0953] "Improvement proposals" are specific proposals based on the analysis results to improve the efficiency of store operations and sales.

[0954] "Store Manager" means the person responsible for managing the store's operations and implementing improvement suggestions.

[0955] "Notification" refers to the act of informing the store manager of the analysis results and improvement suggestions.

[0956] "Monitoring" is the process of continuously observing the effectiveness of implemented improvement proposals.

[0957] The present invention is a system that collects data from monitoring devices and detection devices installed inside and outside a store, analyzes the data, and proposes improvements to store operations. The following hardware and software are used to implement the present invention.

[0958] Hardware

[0959] 1. Surveillance devices (cameras): Install multiple cameras inside and outside the store to capture the movements of customers and employees in real time.

[0960] 2. Detection device (sensor): Detects people's movements and staying time and provides location data.

[0961] 3. Local storage device (local terminal): A device for temporarily storing data obtained from cameras and sensors.

[0962] 4. Server: A central processing unit that receives, decompresses, preprocesses, and analyzes data.

[0963] 5. Notification device: A mobile device or dashboard UI for notifying store managers of improvement suggestions.

[0964] software

[0965] 1. Image processing algorithm: Processes the image data collected from the camera and removes noise.

[0966] 2. Data filtering algorithms: Fill in and refine imperfections in sensor data.

[0967] 3. Compression software: Compress the data in gzip format.

[0968] 4. Secure Communication Protocol (HTTPS): A protocol for securely sending and receiving data to and from a server.

[0969] 5. Generative AI model: The artificial intelligence model used for analysis.

[0970] Specific flow of data

[0971] The terminal acquires data from monitoring and detection devices, denoises and filters it, and stores it in local storage. The data is then compressed and sent to the server via a secure protocol. The server decompresses and derealizes the received data, preprocesses it, and inputs it into a generative AI model. This model analyzes customer and employee behavior and generates improvement suggestions. Finally, the server sends these improvement suggestions to a notification terminal, which notifies the store manager. The user implements the improvement suggestions and monitors their effectiveness by collecting new data and reanalyzing it.

[0972] Specific examples

[0973] For example, specific improvement suggestions are made, such as identifying shelves with long customer dwell times but poor sales, and suggesting changes to their layout. Sample prompts are used in conjunction with these suggestions to analyze the generative AI model. Examples of prompts are shown below.

[0974] Example prompt sentence:

[0975] "Analyze specific shelf purchasing behavior based on current dwell time data."

[0976] In this way, the present invention realizes efficient store management and increased sales, and brings about significant improvements over conventional methods.

[0977] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0978] Step 1:

[0979] The terminal receives real-time data from the monitoring and detection devices. The inputs include video data from the monitoring devices and location data from the detection devices, which are required for subsequent processing.

[0980] How it works: Cameras continuously capture customer movements, and sensors record each customer's location and movements.

[0981] Step 2:

[0982] The data acquired by the device is stored in local storage. The input is the raw data acquired in step 1. The output is the stored data. During this process, noise removal and data filtering are performed.

[0983] Specific operation: Image processing algorithms are used to remove noise from video data and fill in missing data.

[0984] Step 3:

[0985] The device compresses the filtered data and sends it to the server via a secure protocol (e.g., HTTPS). The input is the clean data stored in step 2. The output is the compressed data sent to the server.

[0986] Specific operation: Data is compressed in gzip format and uploaded to the server periodically every night.

[0987] Step 4:

[0988] The server decompresses, derealizes, and preprocesses the received data. The input is the compressed data sent in step 3. The output is the preprocessed data.

[0989] Specific operation: The server decompresses the gzip file, divides the video data into frames, and converts the sensor data into time series data.

[0990] Step 5:

[0991] The server inputs the preprocessed data into the generative AI model to analyze customer and employee behavior. The input is the data preprocessed in step 4. The output is the analysis results.

[0992] Specific operation: The frame data and the shaped sensor data are input into an AI model, which then analyzes customer stay times and employee movements.

[0993] Step 6:

[0994] The server generates improvement proposals based on the analysis results. The input is the analysis results obtained in step 5. The output is improvement proposals.

[0995] Specific operation: Identify shelf locations with poor sales based on customer behavior data and suggest product relocation.

[0996] Step 7:

[0997] The server sends the generated improvement proposal to the terminal, which then notifies the user. The input is the improvement proposal generated in step 6. The output is the notified improvement proposal.

[0998] Specific action: The suggestions are displayed on your smartphone or dashboard, and you are notified of the necessary actions.

[0999] Step 8:

[1000] The user executes the improvement proposal received, which generates new data. The input is the improvement proposal notified in step 7. The output is the execution result.

[1001] Specific operation: The user changes, for example, product placement based on the suggestions.

[1002] Step 9:

[1003] The terminal collects new data after implementing the improvement proposal and sends it back to the server to monitor the effectiveness of the improvement proposal. The input is the new data generated in step 8. The output is the monitored effectiveness data.

[1004] Specific actions: Collect new customer behavior data, send it back to the server, and analyze it to evaluate the effectiveness of improvements.

[1005] (Application example 1)

[1006] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1007] Conventional store management systems have difficulty understanding the behavior of customers and store employees in real time and automatically generating and notifying effective improvement proposals. They also lack the means to quickly evaluate the effectiveness of proposals after their implementation and to make continuous improvements. As a result, store managers are unable to quickly and efficiently optimize store operations, and are often left behind in improving sales and customer satisfaction. This problem needs to be solved.

[1008] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1009] In this invention, the server includes means for acquiring data in real time from multiple cameras and sensors installed inside and outside the store, means for storing the data acquired from the cameras and sensors in local storage and filtering out noise and incomplete data, means for uploading the filtered data to the server, means for inputting the uploaded data into a generative AI model and analyzing customer behavior and store employee behavior, means for generating store operation improvement proposals from the analysis results and notifying the store manager, means for monitoring the operation data after the improvement proposals are implemented and evaluating the effectiveness of the improvement proposals, means for viewing the camera feeds in real time using a smartphone and a head-mounted display and checking the analysis results and improvement proposals, and means for receiving notifications of the improvement proposals via the smartphone and the head-mounted display. This makes it possible to analyze customer and store employee behavior in real time, quickly provide store operation improvement proposals, and continuously evaluate their effectiveness.

[1010] A "camera" is a device installed inside or outside a store that captures the movements of customers and store staff as video.

[1011] A "sensor" is a device installed inside and outside a store that detects people's location and movement.

[1012] "Local storage" is a storage device for temporarily storing data acquired from cameras and sensors.

[1013] "Noise filtering" is the process of removing inaccuracies and unnecessary information from data.

[1014] A "server" is a computer system that receives, analyzes, and stores data over a network.

[1015] A "generative AI model" is an artificial intelligence system that analyzes customer behavior and store clerk behavior and automatically generates improvement suggestions.

[1016] A "smartphone" is a small personal digital assistant (PDA) that can view camera feeds and receive notifications.

[1017] A "head-mounted display" is a device worn on the head that displays images in real time.

[1018] A "store manager" is a person who is responsible for running a store and takes action based on improvement proposals.

[1019] "Operational data" refers to data that indicates the operational status of a store, and includes sales information, customer behavior, and the like.

[1020] "Camera feed" is video data sent from a camera in real time.

[1021] "Improvement proposals" are specific proposals for improving store operations based on data analyzed by the generative AI model.

[1022] "Effectiveness evaluation" is the process of analyzing the results after implementing improvement proposals and determining their effectiveness.

[1023] As a specific embodiment for implementing this invention, a system is constructed that utilizes cameras and sensors installed inside and outside the store, a local terminal, a server, a smartphone, and a head-mounted display.

[1024] Hardware and software used

[1025] Camera: A device installed inside and outside a store that captures the movements of customers and store staff as video.

[1026] Sensor: A device installed inside and outside a store that detects people's location and movement.

[1027] Local device: A computer that temporarily stores and preprocesses data acquired from cameras and sensors.

[1028] Server: A computer system that receives, analyzes, and stores data.

[1029] Smartphone: A small personal digital assistant that allows you to view camera feeds and receive notifications.

[1030] Head-mounted display: A device worn on the head that displays images in real time.

[1031] Data processing and calculation

[1032] 1. Data Collection:

[1033] The device collects real-time data from cameras and sensors installed inside and outside the store. The cameras capture images, and the sensors detect people's location and movement.

[1034] Example: A camera captures a customer pausing in front of a shelf, and a sensor records the duration of their visit.

[1035] 2. Data Storage and Filtering:

[1036] The data collected by the device is stored in local storage and noise and incomplete data are filtered out.

[1037] Example: Removing noise from video data and filling in missing parts of sensor data.

[1038] 3. Data upload:

[1039] The device compresses the cleaned data and uploads it to the server using a secure protocol (e.g. HTTPS).

[1040] Example: Compress data in gzip format and send it to a server overnight.

[1041] 4. Data Analysis:

[1042] The server decompresses and derealizes the received data, performs preprocessing, and then inputs it into a generative AI model, which analyzes customer behavior and store clerk behavior.

[1043] Example: Extracting frames from video data, converting sensor data into time series data, and inputting it into a generative AI model.

[1044] 5. Generation and notification of improvement suggestions:

[1045] The server obtains the analysis results using the generative AI model, generates improvement suggestions based on them, and notifies the store manager.

[1046] Example: Identify shelves where customers spend a lot of time but products are not selling, and suggest changing the placement of those products.

[1047] 6. Effect monitoring:

[1048] The user implements the improvement suggestions received and collects the results again. The server then monitors the sales data and evaluates the effectiveness of the changes.

[1049] Example: After changing product placement, new data is collected again through cameras and sensors to analyze changes in sales.

[1050] 7. Smartphone and head-mounted display usage:

[1051] Users can use their smartphones and head-mounted displays to view real-time camera feeds and see analysis results and improvement suggestions.

[1052] Notification of improvement suggestions will be sent via smartphone and head-mounted display.

[1053] Example prompt: "Analyze customer dwell time and generate shelf relocation suggestions for underperforming products."

[1054] The above configuration makes it possible to improve the efficiency of store operations, and is expected to result in increased customer satisfaction and sales.

[1055] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1056] Step 1:

[1057] The terminal acquires data in real time from cameras and sensors installed inside and outside the store. Specifically, the cameras capture images, and the sensors detect the positions and movements of customers and store staff. The input is the camera images and sensor detection data, and the output is a combination of these data. For example, the camera captures a customer stopping in front of a shelf, and the sensor records the duration of their stay.

[1058] Step 2:

[1059] The data collected by the device is stored in local storage and filtered for noise and incomplete data. Specifically, it removes noise from the video data and complements missing sensor data. The input is camera video and sensor detection data, and the output is clean, filtered data. For example, it improves image quality when the video is unclear and interpolates continuous sensor operation data.

[1060] Step 3:

[1061] The terminal compresses the cleaned data and uploads it to a server using a secure protocol (e.g. HTTPS). The input is the filtered data and the output is the compressed data and confirmation of its destination. For example, the data may be compressed in gzip format and sent to the server overnight.

[1062] Step 4:

[1063] The server decompresses and deserializes the data it receives. Specifically, it converts compressed data back to its original format so that it can be analyzed. The input is compressed data, and the output is decompressed data. For example, it decompresses video data and sensor data compressed with Gzip.

[1064] Step 5:

[1065] The server preprocesses the decompressed and deserialized data and inputs it into the generative AI model. Specifically, it extracts the video data frame by frame and converts the sensor data into time series data. The input is the decompressed data, and the output is the input data for the generative AI model. For example, the video data can be converted into an analyzable format frame by frame, and sensor information converted into time series data can be created.

[1066] Step 6:

[1067] The server uses a generative AI model to analyze customer and store clerk behavior and generates suggestions for improving store operations based on the results. The input is the data input to the generative AI model, and the output is the analysis results and suggestions for improvement. For example, it identifies shelves where customers spend a lot of time but products are not selling, and suggests changing the placement of those products.

[1068] Step 7:

[1069] The server notifies the store manager of the improvement suggestion via the smartphone and head-mounted display. The input is the generated improvement suggestion, and the output is a confirmation that the notification has been sent. For example, the smartphone and head-mounted display display a suggestion such as "Please move product A from shelf X to shelf Y."

[1070] Step 8:

[1071] The user implements the improvement suggestions they receive, and the results are collected again using cameras and sensors. The input is the implementation results of the improvement suggestions, and the output is the operational data after the changes. For example, after changing product placement, new data is collected again using cameras and sensors to analyze changes in sales.

[1072] Step 9:

[1073] Based on the data recollected by the server, the effectiveness of the proposed improvements is evaluated and further proposals are made as necessary. The input is the operational data after the changes, and the output is an evaluation of the effectiveness and new proposals for improvements. For example, the system can confirm the increase in sales after changing product placement and further propose optimization of customer flow.

[1074] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1075] In order to implement the present invention, a system can be constructed and operated in the following manner.

[1076] System Overview

[1077] This system collects data from cameras and sensors installed inside and outside the store, and uses a generative AI model and emotion engine to analyze customer behavior, store clerk activities, and even user emotions, to propose improvements to store operations.The main components are as follows:

[1078] 1. Cameras and sensors: Placed inside and outside the store, they capture the movements of customers and store staff, as well as user emotions.

[1079] 2. Local terminal: collects data from cameras and sensors and performs primary processing and data cleaning.

[1080] 3. Server: Receives data uploaded from local devices and analyzes it using generative AI models and emotion engines.

[1081] 4. User (store manager): Receives improvement suggestions from the server and implements specific actions.

[1082] Explanation of program processing

[1083] Data collection

[1084] 1. The device collects data in real time from cameras and sensors installed inside and outside the store. The cameras capture images, and the sensors collect data such as people's location and movements. In addition, an emotion engine analyzes the user's emotions from their facial expressions and voice.

[1085] Example: A camera captures a customer's movements and facial expressions as they pause in front of a shelf, and sensors record their dwell time and emotions.

[1086] Data storage and cleaning

[1087] 2. The device temporarily stores the collected data in local storage, filtering out noise and incomplete data when storing.

[1088] Example: Removing noise from video data and filling in missing parts of sensor data.

[1089] Uploading data

[1090] 3. The device compresses the filtered data and uploads it to the server using the specified secure protocol.

[1091] Example: Compress data in gzip format and send it to the server using HTTPS.

[1092] Data analysis

[1093] 4. The server de-realizes the uploaded data and restores it to its original form.

[1094] Specific operation: The received compressed file is decompressed and the original video data and sensor data are restored.

[1095] 5. The server preprocesses the recovered data and converts it into a format that can be input to the generative AI model and emotion engine.

[1096] Examples: Splitting video data into frames, converting sensor data into time series data, and formatting facial expression data for analysis.

[1097] 6. The server inputs the preprocessed data into the generative AI model and emotion engine to analyze customer behavior, store clerk behavior, and user emotions.

[1098] Example: The AI ​​model analyzes the customer's length of stay, their movement in front of the product, the timing of the store clerk's call, changes in facial expression, etc.

[1099] 7. The server generates suggestions for improving store operations based on the analysis results from the generated AI model and emotion engine. The suggestions also take into account the user's emotional state.

[1100] Example: An emotion engine infers customer satisfaction from their facial expressions, identifies products and placements that cause dissatisfaction, and suggests areas for improvement.

[1101] Notification of improvement proposals

[1102] 8. The device notifies the store manager of the improvement suggestions received from the server, for example, via a mobile device or dashboard UI.

[1103] Example: The dashboard might say, "Move product A from shelf X to shelf Y and have the sales associate provide specific advice to the customer."

[1104] Implementing improvement proposals and monitoring their effectiveness

[1105] 9. Implement the improvement suggestions received from the user and record the results.

[1106] Example: Adjusting merchandise movements and store staff actions and inputting the results into the system.

[1107] 10. The server collects data again and monitors the effectiveness of the proposed improvements.

[1108] Example: Analyzing new customer behavior data, sales data, and customer facial expression data to evaluate performance and customer satisfaction after improvements.

[1109] Specific examples

[1110] 1. The device collects behavioral and emotional data of 500 customers every day, filters out noise, and stores it in local storage.

[1111] 2. The device uploads the compressed data to the server overnight.

[1112] 3. The server decompresses and preprocesses the data, then analyzes it with a generative AI model and emotion engine. For example, it might detect that a particular product is not selling well, but many customers stop in front of it with unhappy expressions.

[1113] 4. The server generates improvement proposals, including suggestions for changing product placement and training for store staff, and notifies them.

[1114] 5. The user implements the suggestions and collects the performance results again.

[1115] 6. The server analyzes the new data, monitors the effectiveness of the suggested improvements, and makes further recommendations.

[1116] In this way, by implementing the present invention, it is possible to effectively improve sales and customer experience in store operations, as well as increase customer satisfaction.

[1117] The processing flow will be explained below.

[1118] Step 1:

[1119] The device collects data in real time from multiple cameras and sensors installed inside and outside the store.

[1120] How it works: Cameras capture footage of customers, and sensors collect their location, movement, and user audio data.

[1121] Step 2:

[1122] The data acquired by the device is temporarily stored in local storage.

[1123] Specific behavior: Stores captured video frames, sensor data, and audio data in a database or file system.

[1124] Step 3:

[1125] The device performs noise filtering on the stored data to remove unnecessary data.

[1126] Specific operation: Filters out low-resolution video, misdetected sensor data, and audio data containing noise, leaving only the necessary data.

[1127] Step 4:

[1128] The device compresses the filtered data and uploads it to the server.

[1129] Specific operation: Compress the data into gzip format and send it securely to the server via HTTPS.

[1130] Step 5:

[1131] The server unpacks the uploaded data and restores it to its original format.

[1132] Specific operation: The received compressed file is decompressed and restored to the original video data, sensor data, and audio data.

[1133] Step 6:

[1134] The server preprocesses the recovered data and converts it into a format that can be input to the generative AI model and emotion engine.

[1135] Specific operations: Video data is divided into frames, sensor data is organized into time series data, and audio data is converted into text.

[1136] Step 7:

[1137] The server inputs the preprocessed data into a generative AI model and emotion engine to analyze customer behavior, store clerk behavior, and user emotions.

[1138] Specific operation: The generative AI model analyzes the customer's dwell time and movement in front of the product, and the emotion engine analyzes the customer's emotions from facial expressions and voice data.

[1139] Step 8:

[1140] The server generates improvement proposals for store operations based on the analysis results.

[1141] Specific operation: The results of the generative AI model and the emotion engine are combined to compile suggestions such as changing product placement, improving the way store staff serve customers, and providing services based on customer emotions.

[1142] Step 9:

[1143] The terminal notifies the store manager (user) of the improvement proposal received from the server.

[1144] Specific actions: Display suggestions on mobile devices and dashboard UI, and clearly indicate specific actions.

[1145] Step 10:

[1146] Implement the improvement suggestions received by the user and record the results.

[1147] Specific actions: Change product placement, give instructions to store staff, implement measures to improve customer service, and feed the results back into the system.

[1148] Step 11:

[1149] The server collects data again and monitors the effectiveness of the improvement suggestions.

[1150] Specific actions: Analyze new customer behavior data, sales data, and customer facial expression data to evaluate the effectiveness of implemented improvement measures.

[1151] Example 2

[1152] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1153] Conventional customer behavior analysis systems for store operations are limited to providing feedback on customer behavior data and store clerk responses, and are unable to accurately grasp the user's emotional state and reflect it in improvement proposals. Furthermore, the analysis targets noisy or incomplete data, resulting in reduced proposal accuracy. The present invention aims to solve these problems and provide more accurate store operation improvement proposals.

[1154] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for uploading the filtered data to the server, a means for derealizing the uploaded data, preprocessing it, and converting it into an analyzable format, and a means for inputting the preprocessed data into a generative AI model and an emotion analysis engine, analyzing customer behavior and clerk behavior, and estimating user emotions. This enables highly accurate data analysis and improvement suggestions that include the user's emotional state.

[1155] A "camera" is a device for capturing video data.

[1156] A "sensor" is a device that detects physical changes and outputs them as data, specifically one that detects a person's position or movement.

[1157] "Local storage" is an internal storage device for temporarily storing data.

[1158] "Filtering" refers to the process of removing noise and imperfections from data.

[1159] "Upload" refers to the process of sending data from a local terminal to a server.

[1160] "Deserialization" is the process of returning serialized data to its original form.

[1161] "Preprocessing" refers to the initial processing of data to prepare it in an analyzable format.

[1162] A "generative AI model" is an algorithm that uses machine learning to analyze data and generate a specific result.

[1163] An "emotion analysis engine" is an algorithm or system for inferring emotions from a user's facial expressions and voice.

[1164] "Customer behavior" refers to data on customer behavior such as location, movement patterns, and length of stay within the store.

[1165] "Employee behavior" refers to data on employee behavior such as their location within the store, movement patterns, and how they interact with customers.

[1166] "Analysis" refers to the act of processing data to extract useful information and patterns.

[1167] "Improvement proposals" refer to proposals based on analysis results aimed at optimizing store operations and customer service.

[1168] "Notification" refers to the process of communicating information to store managers, specifically via mobile devices or dashboards.

[1169] The following describes in detail an embodiment of the present invention: The present invention is a system that utilizes data from cameras and sensors installed inside and outside a store, and analyzes customer behavior, store clerk behavior, and user emotions using a generative AI model and an emotion analysis engine.

[1170] System Overview

[1171] The system mainly includes the following hardware and software:

[1172] Cameras: Multiple cameras are installed inside and outside the store to capture video data.

[1173] Sensors: There are infrared sensors and various motion sensors that collect location information of customers and store staff.

[1174] Device: Stores collected data in local storage and filters out noise and incomplete data.

[1175] Server: De-realizes the filtered data, pre-processes it, and converts it into an analyzable format. It then analyzes it using a generative AI model and sentiment analysis engine to generate improvement suggestions.

[1176] Data collection and cleaning

[1177] The device collects data in real time from cameras and sensors. For example, the camera captures video at 30 frames per second, and sensors acquire location data of customers and store staff. This data is stored in local storage, where noise and incomplete data are filtered out. For example, ghost noise in the video data is removed and incomplete sensor data is complemented.

[1178] Data upload and de-realization

[1179] The device compresses the filtered data in gzip format and uploads it to the server via secure communication using the HTTPS protocol. The server then decompresses and derealizes the received data. For example, it splits the decompressed video data into frames and stores the sensor data in a database.

[1180] Preprocessing and analysis

[1181] The server preprocesses the derealized data and prepares it in a format that can be analyzed by the generative AI model and emotion analysis engine. This includes resizing the video data, extracting frames, and normalizing the sensor data. For example, the server resizes the video data to 256x256 pixels, which is easier for the AI ​​model to handle, and fills in missing parts of the sensor data to prepare it as time-series data.

[1182] The server inputs the preprocessed data into a generative AI model and an emotion analysis engine to analyze customer behavior, store clerk behavior, and user emotions. The AI ​​model analyzes the customer's length of stay, their movement, and the timing of the clerk's interactions, while the emotion analysis engine estimates the customer's emotional state based on facial expressions and voice data. For example, the generative AI model analyzes the customer's length of stay, and the emotion analysis engine analyzes the customer's dissatisfied facial expression.

[1183] Improvement suggestions and notifications

[1184] The server generates suggestions for improving store operations based on the analysis results and the customer's emotional state. For example, suggestions may be generated that include changing product placement or specific guidelines for store staff behavior. These suggestions are notified to store managers via their devices. Notification methods include a dashboard UI or a mobile device. As a specific example, the dashboard may display a message such as, "Move product A from shelf X to shelf Y, and have store staff provide specific advice to customers."

[1185] Implementing improvement proposals and monitoring their effectiveness

[1186] The user implements the improvement proposal and records the results. For example, changing the product layout and adjusting the behavior of the store clerk. The data after the improvement is collected again and entered into the system, allowing the server to analyze it again, monitor the effectiveness of the proposal, and make further improvement proposals. For example, new customer behavior data, sales data, and customer satisfaction data are evaluated, and feedback is provided for further improvements.

[1187] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1188] Step 1:

[1189] The device acquires data in real time from the camera and sensors. The input data is video data from the camera and position and movement data from the sensors. Specifically, the camera captures video at 30 frames per second, and the sensors acquire position information of customers and store staff. The output is raw video data and sensor data.

[1190] Step 2:

[1191] The data collected by the device is temporarily stored in local storage and noise and incomplete data are filtered. The input data are the raw video data and sensor data acquired in the previous step. Specifically, the process removes ghost noise contained in the video data and complements incomplete sensor data. The output is filtered video data and sensor data.

[1192] Step 3:

[1193] The terminal compresses the filtered data in gzip format and uploads it to the server using the HTTPS protocol. The input data is the filtered video data and sensor data. Specifically, the data is compressed in gzip format and transmitted securely using HTTPS. The output is the compressed data uploaded to the server.

[1194] Step 4:

[1195] The server de-realizes the uploaded data and restores it to its original format. The input data is compressed video data and sensor data. Specifically, it decompresses the received gzip format file and restores it to video data and sensor data. The output is the decompressed and restored video data and sensor data.

[1196] Step 5:

[1197] The server preprocesses the restored data and converts it into a format that can be analyzed by the generative AI model and emotion analysis engine. The input data is the derealized video data and sensor data. Specifically, the server splits the video data into frames and resizes it to 256x256 pixels. It also converts the sensor data into time-series data and extracts facial expression and movement features. The output is the preprocessed data.

[1198] Step 6:

[1199] The server inputs the preprocessed data into a generative AI model and emotion analysis engine, which analyzes customer behavior, store clerk behavior, and user emotions. The input data is preprocessed video data and sensor data. Specifically, the generative AI model analyzes customer movements and length of stay, and the emotion analysis engine infers the user's emotions from facial expression data. The output is the analysis results.

[1200] Step 7:

[1201] The server generates improvement proposals for store operations based on the analysis results. The input data are the analysis results. Specific operations include identifying products and placements that cause high customer dissatisfaction from the analysis results and proposing improvements accordingly. The output is specific improvement proposals.

[1202] Step 8:

[1203] The device notifies the user of the improvement proposal received from the server. The input data is the generated improvement proposal. As a specific operation, the proposal content is displayed through the dashboard UI or mobile device. The output is the improvement proposal notified to the user.

[1204] Step 9:

[1205] The user implements the improvement proposals received and records the results. The input data is the notified improvement proposal. Specific actions include moving products and adjusting the actions of store clerks, and inputting the resulting new behavioral and sales data into the system. The output is the implemented improvement proposals and new data.

[1206] Step 10:

[1207] The server re-analyzes the newly collected data and monitors the effectiveness of the proposed improvements. The input data is the new data after execution. Specific operations include evaluating the analysis results and determining how effective the proposals were. Based on this, further improvement proposals are made. The output is an evaluation of the effectiveness of the proposed improvements and new proposals.

[1208] (Application example 2)

[1209] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1210] In recent years, improving store operation efficiency and customer satisfaction have become important issues in the retail industry, but conventional methods have difficulty in conducting detailed analysis of customer behavior and emotions, resulting in a lack of concrete improvement proposals based on this analysis.In addition, while employee activities need to be monitored and supervised efficiently, current systems are unable to adequately achieve this.

[1211] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for acquiring information in real time from observation devices and sensing devices, means for saving the acquired information in a local storage device and filtering out useless or incomplete data, means for transferring the filtered information to a computer, means for inputting the transferred information into a generative AI model and a sentiment analysis engine to analyze customer behavior, employee activities, and user emotions, and means for generating store operation improvement proposals from the analysis results and notifying the manager. This makes it possible to perform detailed analysis of customer behavior, employee activities, and user emotions and provide specific and effective store operation improvement proposals.

[1212] "Observation devices" are devices installed to monitor and record the behavior and situations of customers and employees.

[1213] A "sensing device" is a device that collects information about the location and movement of customers and employees, as well as environmental information.

[1214] "Information" refers to data acquired in real time from observation and sensing devices, and includes video, audio, movement patterns, emotional states, and the like.

[1215] A "local storage device" is a device for temporarily storing information obtained from observation and sensing devices.

[1216] "Filtering" is the process of removing useless or incomplete data from stored information.

[1217] "Computer" refers to a server or cloud computing system for analyzing information transferred from local storage.

[1218] A "generative AI model" is a machine learning model trained to analyze customer and employee behavior and emotions.

[1219] An "emotion analysis engine" is software that analyzes emotions from a user's facial expressions and voice.

[1220] "Analysis results" refers to information about customer behavior, employee activities, and emotional states obtained by the generative AI model and sentiment analysis engine.

[1221] "Improvement proposals" are specific action plans for optimizing store operations and improving customer satisfaction based on the analysis results.

[1222] A "manager" is a person who is responsible for overseeing store operations and implementing improvement suggestions.

[1223] Summary of the Invention

[1224] This invention is a system that uses a generative AI model and an emotion analysis engine to analyze customer behavior, employee activity, and user emotions based on data collected from observation and sensing devices in a physical store, and provides suggestions for improving store operations. This system can improve customer satisfaction and streamline store operations.

[1225] Detailed system description

[1226] Data collection

[1227] The system collects information in real time from observation devices (cameras) and sensing devices (sensors) installed inside and outside the store. The observation devices record the behavior of customers and employees, while the sensing devices collect information on their location, behavior, and environment.

[1228] Storing and Filtering Data

[1229] The collected information is stored on a local storage device (e.g., a device such as a Raspberry Pi). When stored, useless or incomplete data is filtered out, leaving only the necessary information.

[1230] Data Transfer

[1231] The filtered information is converted into a compressed format (eg, gzip format) and transferred to a computer (a server or a cloud computing system) via a secure communication means (eg, HTTPS protocol).

[1232] Data analysis

[1233] The server receives the transferred information and inputs it into a generative AI model and sentiment analysis engine, which analyzes customer behavior, employee activity, and the user's emotional state. The generative AI model is trained using machine learning frameworks such as TensorFlow and PyTorch, and the sentiment analysis engine uses Microsoft Azure Emotion API and Amazon Rekognition.

[1234] Generate and notify improvement suggestions

[1235] Based on the analysis results, the server generates specific suggestions for improving store operations. These suggestions are notified to the manager in real time, who then acts accordingly. Notifications are sent via a smartphone app (e.g., an app developed based on Flutter or React Native).

[1236] Examples of specific examples and prompts

[1237] For example, if many customers stop in front of a certain product but do not end up purchasing it, the generative AI model will determine that there is a problem with the product's placement or explanation, and will suggest moving the product or improving the explanation.In addition, if the emotion analysis engine detects low satisfaction from customers' facial expressions, it will generate suggestions to encourage employees to take a more proactive approach.

[1238] An example prompt is:

[1239] "Customers who pause in front of product A spend a lot of time there, but the percentage of them actually purchasing is low. The sentiment analysis engine indicates that customer satisfaction is low. Therefore, move product A from shelf X to shelf Y, and have an associate provide specific advice to the customer."

[1240] This allows for detailed analysis of customer behavior, employee activities, and user sentiment, making it possible to provide specific and effective suggestions for improving store operations.

[1241] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1242] Step 1:

[1243] The terminal acquires information in real time from observation devices (cameras) and detection devices (sensors) installed inside and outside the store. The input is data from each observation device and detection device, which is received by the terminal. Specifically, the camera records customer behavior, and the sensor collects movement and location information.

[1244] Step 2:

[1245] The device stores the acquired information in a local storage device. Here, it receives the acquired raw data as input and filters out noise and incomplete data. Specifically, it removes noise from the video data and fills in missing parts of the sensor data. The output is clean, filtered data.

[1246] Step 3:

[1247] The terminal converts the filtered data into a compressed format and transfers it to the server. It receives clean data as input and compresses it, for example, using gzip format. It then sends it to the server using a secure communication method (for example, HTTPS protocol). The specific operations include compressing the data, establishing a secure network connection, and transferring the data. The compressed data arrives at the server as output.

[1248] Step 4:

[1249] The server receives the transferred data and decompresses it. It receives compressed data as input and restores it to its original information format. Specifically, it decompresses the received gzip file and returns the video data and sensor data to their original state.

[1250] Step 5:

[1251] The server preprocesses the restored data into a format that can be input to the generative AI model and emotion analysis engine. It receives the decompressed data as input and converts it into a format for analysis. Specifically, it splits the video data into frames, converts the sensor data into time-series data, and formats the facial expression data for analysis. The preprocessed data is obtained as output.

[1252] Step 6:

[1253] The server inputs the preprocessed data into the generative AI model and emotion analysis engine, which analyzes customer behavior, employee activities, and user emotions. The preprocessed data is received as input and analyzed. Specifically, the generative AI model analyzes the customer's length of stay, the timing of employee calls, changes in facial expressions, etc. The analysis results are obtained as output.

[1254] Step 7:

[1255] The server generates improvement proposals for store operations based on the analysis results. It receives the analysis results as input and creates improvement proposals based on them. Specifically, it proposes optimal product placement and employee action plans based on the results of the generative AI model and sentiment analysis engine. The generated improvement proposals are obtained as output.

[1256] Step 8:

[1257] The device notifies the administrator of the improvement proposal received from the server. The device receives the generated improvement proposal as input and communicates it to the administrator. Specifically, the device displays and notifies the administrator of the improvement proposal through a smartphone app (for example, one using Flutter or React Native). The output is that notification to the administrator has been completed.

[1258] Step 9:

[1259] The user implements the notified improvement suggestions and re-enters the results into the device. The results of the suggestions are received as input and fed back to the system. Specific operations include moving products or changing employee behavior in accordance with the presented improvement suggestions, and recording the results in the app. Feedback data is obtained as output.

[1260] Step 10:

[1261] The server collects data again and monitors the effectiveness of the proposed improvements. It receives feedback data as input and evaluates its effectiveness. Specifically, it analyzes the new data after the improvements and evaluates customer behavior, sales data, and customer satisfaction. The output is performance data after the improvements.

[1262] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1263] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1264] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1265] [Fourth embodiment]

[1266] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1267] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1268] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1269] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1270] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1271] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1272] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1273] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1274] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1275] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1276] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1277] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1278] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1279] In order to implement the present invention, a system can be constructed and operated in the following manner.

[1280] System Overview

[1281] This system collects data from cameras and sensors installed inside and outside the store, and uses a generative AI model to analyze customer behavior and store staff activities, thereby proposing improvements to store operations.The main components are as follows:

[1282] 1. Cameras and sensors: These are placed inside and outside the store to capture the movements of customers and store staff.

[1283] 2. Local terminal: collects data from cameras and sensors and performs primary processing and data cleaning.

[1284] 3. Server: Receives data uploaded from local devices and analyzes it using generative AI models.

[1285] 4. User (store manager): Receives improvement suggestions from the server and implements specific actions.

[1286] Explanation of program processing

[1287] Data collection

[1288] 1. The device collects real-time data from cameras and sensors installed inside and outside the store. The cameras capture images, and the sensors detect people's location and movement.

[1289] Example: A camera captures a customer pausing in front of a shelf, and a sensor records the duration of their visit.

[1290] Data storage and cleaning

[1291] 2. The data collected by the device is stored in local storage, and noise and incomplete data are filtered out during storage.

[1292] Example: Removing noise from video data and filling in missing parts of sensor data.

[1293] Uploading data

[1294] 3. The device compresses the cleaned data and uploads it to the server using the specified secure protocol.

[1295] Example: Compress data in gzip format and send it to the server using HTTPS.

[1296] Data analysis

[1297] 4. The server decompresses and derealizes the received data, performs preprocessing, and then inputs it into the generative AI model, which analyzes customer behavior and store clerk behavior.

[1298] Example: Extracting frames from video data, converting sensor data into time series data, and inputting it into a generative AI model.

[1299] 5. The server obtains the analysis results from the generative AI model and generates improvement proposals based on them.

[1300] Example: Identify shelves where customers spend a lot of time but products are not selling, and suggest changing the placement of those products.

[1301] Notification of improvement proposals

[1302] 6. The device notifies the store manager of the improvement suggestions received from the server, for example, via a mobile device or dashboard UI.

[1303] Example: A dashboard might suggest, "Move product A from shelf X to shelf Y."

[1304] Implementing improvement proposals and monitoring their effectiveness

[1305] 7. The user implements the improvement suggestions received and collects the results again, allowing the effectiveness of the improvement suggestions to be monitored.

[1306] Example: After changing product placement, new data is collected again through cameras and sensors to analyze changes in sales.

[1307] Specific examples

[1308] 1. The device collects behavioral data of 500 customers every day, filters out noise, and stores it in local storage.

[1309] 2. The device uploads the compressed data to the server overnight.

[1310] 3. The server decompresses and preprocesses the data and runs it through a generative AI model, detecting, for example, that a particular product is not selling well but many customers still stop in front of it.

[1311] 4. The server generates and notifies the user of a suggestion to change the placement of the products. The user then moves the products based on the suggestion.

[1312] 5. The server monitors subsequent sales data to assess the effectiveness of the changes.

[1313] In this way, by implementing the present invention, sales and customer experience in store operations can be effectively improved.

[1314] The processing flow will be explained below.

[1315] Step 1:

[1316] The device collects data in real time from multiple cameras and sensors installed inside and outside the store.

[1317] How it works: Cameras capture footage of customers, and sensors collect data like their location and movements.

[1318] Step 2:

[1319] The data acquired by the device is temporarily stored in local storage.

[1320] Specific operation: Stores captured video frames and sensor data in a database or file system.

[1321] Step 3:

[1322] The device performs noise filtering on the stored data to remove unnecessary data.

[1323] Specific operation: Automatically eliminates low-resolution footage and falsely detected sensor data, leaving only the necessary data.

[1324] Step 4:

[1325] The terminal periodically compresses the filtered data and uploads it to the server.

[1326] What it does: It converts the data into a compressed format such as gzip and transfers it securely to the server over HTTPS.

[1327] Step 5:

[1328] The server de-realizes the uploaded data and restores it to its original form.

[1329] Specific operation: The received compressed file is decompressed and the original video data and sensor data are restored.

[1330] Step 6:

[1331] The server preprocesses the restored data and converts it into a format that can be input into the generative AI model.

[1332] Specific operation: Video data is divided into frames and sensor data is converted into time series data.

[1333] Step 7:

[1334] The server inputs the preprocessed data into a generative AI model to analyze customer behavior and store clerk behavior.

[1335] Specific operation: The AI ​​model analyzes the customer's length of stay, their movement in front of the product, and the timing of the store clerk's call.

[1336] Step 8:

[1337] The server generates suggestions for improving store operations based on the analysis results from the generated AI model.

[1338] Specific actions: Make specific suggestions such as changing product placement and training content for store staff.

[1339] Step 9:

[1340] The terminal notifies the store manager (user) of the improvement proposal received from the server.

[1341] Specific actions: Display suggestions on mobile devices and dashboards, clearly indicating the actions to be taken.

[1342] Step 10:

[1343] Implement the improvement suggestions received by the user and record the results.

[1344] Specific actions: Move products and adjust the actions of store staff, and enter the results into the system.

[1345] Step 11:

[1346] The server collects data again and monitors the effectiveness of the improvement suggestions.

[1347] Specific Actions: Analyze new customer behavior and sales data to evaluate performance after improvements.

[1348] Example 1

[1349] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1350] In traditional store operations, there is a lack of means to properly analyze customer behavior and employee activity and propose operational improvement measures in real time. This makes it difficult to optimize sales and improve customer experience. In addition, there are poor means to monitor the effectiveness of improvement proposals, making it difficult to quantitatively grasp the actual effects of improvements.

[1351] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1352] In this invention, the server includes means for acquiring data in real time from monitoring devices and detection devices, means for storing the data acquired from the devices in a local storage device and filtering out noise and incomplete data, means for compressing the filtered data and transmitting it to the server via a secure protocol, means for the server to decompress and de-realize the data received, pre-process it, and input it into a generative AI model to analyze customer behavior and employee behavior, and means for generating store operation improvement proposals from the analysis results and notifying the store manager. This makes it possible to dramatically improve the efficiency of store operation and enable real-time monitoring of the effects of improvement proposals and rapid feedback.

[1353] A "monitoring device" is a device that is installed in a target area and acquires video data in real time.

[1354] A "detection device" is a device that is installed in a target area and detects the location and movement of people and objects in real time.

[1355] A "local storage device" is a storage device for temporarily storing collected data.

[1356] "Noise" refers to unnecessary or incorrect information that gets mixed in during the data collection process.

[1357] "Incomplete data" is data that contains missing or incorrect information during the collection process.

[1358] "Filtering" is the process of removing noise and imperfections from collected data.

[1359] "Compression" is a process performed to reduce the volume of data.

[1360] A "secure protocol" is a communication method that ensures security when sending and receiving data.

[1361] The "server" is a central device that receives collected data and processes and analyzes it.

[1362] "Derealization" is the process of returning received data to its original format.

[1363] "Preprocessing" is the process of converting data into an appropriate format before feeding it into a generative AI model.

[1364] A "generative AI model" is an artificial intelligence model that analyzes data and recognizes customer and employee behavior.

[1365] "Analysis" is the process of extracting information for a specific purpose from collected data.

[1366] "Improvement proposals" are specific proposals based on the analysis results to improve the efficiency of store operations and sales.

[1367] "Store Manager" means the person responsible for managing the store's operations and implementing improvement suggestions.

[1368] "Notification" refers to the act of informing the store manager of the analysis results and improvement suggestions.

[1369] "Monitoring" is the process of continuously observing the effectiveness of implemented improvement proposals.

[1370] The present invention is a system that collects data from monitoring devices and detection devices installed inside and outside a store, analyzes the data, and proposes improvements to store operations. The following hardware and software are used to implement the present invention.

[1371] Hardware

[1372] 1. Surveillance devices (cameras): Install multiple cameras inside and outside the store to capture the movements of customers and employees in real time.

[1373] 2. Detection device (sensor): Detects people's movements and staying time and provides location data.

[1374] 3. Local storage device (local terminal): A device for temporarily storing data obtained from cameras and sensors.

[1375] 4. Server: A central processing unit that receives, decompresses, preprocesses, and analyzes data.

[1376] 5. Notification device: A mobile device or dashboard UI for notifying store managers of improvement suggestions.

[1377] software

[1378] 1. Image processing algorithm: Processes the image data collected from the camera and removes noise.

[1379] 2. Data filtering algorithms: Fill in and refine imperfections in sensor data.

[1380] 3. Compression software: Compress the data in gzip format.

[1381] 4. Secure Communication Protocol (HTTPS): A protocol for securely sending and receiving data to and from a server.

[1382] 5. Generative AI model: The artificial intelligence model used for analysis.

[1383] Specific flow of data

[1384] The terminal acquires data from monitoring and detection devices, denoises and filters it, and stores it in local storage. The data is then compressed and sent to the server via a secure protocol. The server decompresses and derealizes the received data, preprocesses it, and inputs it into a generative AI model. This model analyzes customer and employee behavior and generates improvement suggestions. Finally, the server sends these improvement suggestions to a notification terminal, which notifies the store manager. The user implements the improvement suggestions and monitors their effectiveness by collecting new data and reanalyzing it.

[1385] Specific examples

[1386] For example, specific improvement suggestions are made, such as identifying shelves with long customer dwell times but poor sales, and suggesting changes to their layout. Sample prompts are used in conjunction with these suggestions to analyze the generative AI model. Examples of prompts are shown below.

[1387] Example prompt sentence:

[1388] "Analyze specific shelf purchasing behavior based on current dwell time data."

[1389] In this way, the present invention realizes efficient store management and increased sales, and brings about significant improvements over conventional methods.

[1390] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1391] Step 1:

[1392] The terminal receives real-time data from the monitoring and detection devices. The inputs include video data from the monitoring devices and location data from the detection devices, which are required for subsequent processing.

[1393] How it works: Cameras continuously capture customer movements, and sensors record each customer's location and movements.

[1394] Step 2:

[1395] The data acquired by the device is stored in local storage. The input is the raw data acquired in step 1. The output is the stored data. During this process, noise removal and data filtering are performed.

[1396] Specific operation: Image processing algorithms are used to remove noise from video data and fill in missing data.

[1397] Step 3:

[1398] The device compresses the filtered data and sends it to the server via a secure protocol (e.g., HTTPS). The input is the clean data stored in step 2. The output is the compressed data sent to the server.

[1399] Specific operation: Data is compressed in gzip format and uploaded to the server periodically every night.

[1400] Step 4:

[1401] The server decompresses, derealizes, and preprocesses the received data. The input is the compressed data sent in step 3. The output is the preprocessed data.

[1402] Specific operation: The server decompresses the gzip file, divides the video data into frames, and converts the sensor data into time series data.

[1403] Step 5:

[1404] The server inputs the preprocessed data into the generative AI model to analyze customer and employee behavior. The input is the data preprocessed in step 4. The output is the analysis results.

[1405] Specific operation: The frame data and the shaped sensor data are input into an AI model, which then analyzes customer stay times and employee movements.

[1406] Step 6:

[1407] The server generates improvement proposals based on the analysis results. The input is the analysis results obtained in step 5. The output is improvement proposals.

[1408] Specific operation: Identify shelf locations with poor sales based on customer behavior data and suggest product relocation.

[1409] Step 7:

[1410] The server sends the generated improvement proposal to the terminal, which then notifies the user. The input is the improvement proposal generated in step 6. The output is the notified improvement proposal.

[1411] Specific action: The suggestions are displayed on your smartphone or dashboard, and you are notified of the necessary actions.

[1412] Step 8:

[1413] The user executes the improvement proposal received, which generates new data. The input is the improvement proposal notified in step 7. The output is the execution result.

[1414] Specific operation: The user changes, for example, product placement based on the suggestions.

[1415] Step 9:

[1416] The terminal collects new data after implementing the improvement proposal and sends it back to the server to monitor the effectiveness of the improvement proposal. The input is the new data generated in step 8. The output is the monitored effectiveness data.

[1417] Specific actions: Collect new customer behavior data, send it back to the server, and analyze it to evaluate the effectiveness of improvements.

[1418] (Application example 1)

[1419] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1420] Conventional store management systems have difficulty understanding the behavior of customers and store employees in real time and automatically generating and notifying effective improvement proposals. They also lack the means to quickly evaluate the effectiveness of proposals after their implementation and to make continuous improvements. As a result, store managers are unable to quickly and efficiently optimize store operations, and are often left behind in improving sales and customer satisfaction. This problem needs to be solved.

[1421] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1422] In this invention, the server includes means for acquiring data in real time from multiple cameras and sensors installed inside and outside the store, means for storing the data acquired from the cameras and sensors in local storage and filtering out noise and incomplete data, means for uploading the filtered data to the server, means for inputting the uploaded data into a generative AI model and analyzing customer behavior and store employee behavior, means for generating store operation improvement proposals from the analysis results and notifying the store manager, means for monitoring the operation data after the improvement proposals are implemented and evaluating the effectiveness of the improvement proposals, means for viewing the camera feeds in real time using a smartphone and a head-mounted display and checking the analysis results and improvement proposals, and means for receiving notifications of the improvement proposals via the smartphone and the head-mounted display. This makes it possible to analyze customer and store employee behavior in real time, quickly provide store operation improvement proposals, and continuously evaluate their effectiveness.

[1423] A "camera" is a device installed inside or outside a store that captures the movements of customers and store staff as video.

[1424] A "sensor" is a device installed inside and outside a store that detects people's location and movement.

[1425] "Local storage" is a storage device for temporarily storing data acquired from cameras and sensors.

[1426] "Noise filtering" is the process of removing inaccuracies and unnecessary information from data.

[1427] A "server" is a computer system that receives, analyzes, and stores data over a network.

[1428] A "generative AI model" is an artificial intelligence system that analyzes customer behavior and store clerk behavior and automatically generates improvement suggestions.

[1429] A "smartphone" is a small personal digital assistant (PDA) that can view camera feeds and receive notifications.

[1430] A "head-mounted display" is a device worn on the head that displays images in real time.

[1431] A "store manager" is a person who is responsible for running a store and takes action based on improvement proposals.

[1432] "Operational data" refers to data that indicates the operational status of a store, and includes sales information, customer behavior, and the like.

[1433] "Camera feed" is video data sent from a camera in real time.

[1434] "Improvement proposals" are specific proposals for improving store operations based on data analyzed by the generative AI model.

[1435] "Effectiveness evaluation" is the process of analyzing the results after implementing improvement proposals and determining their effectiveness.

[1436] As a specific embodiment for implementing this invention, a system is constructed that utilizes cameras and sensors installed inside and outside the store, a local terminal, a server, a smartphone, and a head-mounted display.

[1437] Hardware and software used

[1438] Camera: A device installed inside and outside a store that captures the movements of customers and store staff as video.

[1439] Sensor: A device installed inside and outside a store that detects people's location and movement.

[1440] Local device: A computer that temporarily stores and preprocesses data acquired from cameras and sensors.

[1441] Server: A computer system that receives, analyzes, and stores data.

[1442] Smartphone: A small personal digital assistant that allows you to view camera feeds and receive notifications.

[1443] Head-mounted display: A device worn on the head that displays images in real time.

[1444] Data processing and calculation

[1445] 1. Data Collection:

[1446] The device collects real-time data from cameras and sensors installed inside and outside the store. The cameras capture images, and the sensors detect people's location and movement.

[1447] Example: A camera captures a customer pausing in front of a shelf, and a sensor records the duration of their visit.

[1448] 2. Data Storage and Filtering:

[1449] The data collected by the device is stored in local storage and noise and incomplete data are filtered out.

[1450] Example: Removing noise from video data and filling in missing parts of sensor data.

[1451] 3. Data upload:

[1452] The device compresses the cleaned data and uploads it to the server using a secure protocol (e.g. HTTPS).

[1453] Example: Compress data in gzip format and send it to a server overnight.

[1454] 4. Data Analysis:

[1455] The server decompresses and derealizes the received data, performs preprocessing, and then inputs it into a generative AI model, which analyzes customer behavior and store clerk behavior.

[1456] Example: Extracting frames from video data, converting sensor data into time series data, and inputting it into a generative AI model.

[1457] 5. Generation and notification of improvement suggestions:

[1458] The server obtains the analysis results using the generative AI model, generates improvement suggestions based on them, and notifies the store manager.

[1459] Example: Identify shelves where customers spend a lot of time but products are not selling, and suggest changing the placement of those products.

[1460] 6. Effect monitoring:

[1461] The user implements the improvement suggestions received and collects the results again. The server then monitors the sales data and evaluates the effectiveness of the changes.

[1462] Example: After changing product placement, new data is collected again through cameras and sensors to analyze changes in sales.

[1463] 7. Smartphone and head-mounted display usage:

[1464] Users can use their smartphones and head-mounted displays to view real-time camera feeds and see analysis results and improvement suggestions.

[1465] Notification of improvement suggestions will be sent via smartphone and head-mounted display.

[1466] Example prompt: "Analyze customer dwell time and generate shelf relocation suggestions for underperforming products."

[1467] The above configuration makes it possible to improve the efficiency of store operations, and is expected to result in increased customer satisfaction and sales.

[1468] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1469] Step 1:

[1470] The terminal acquires data in real time from cameras and sensors installed inside and outside the store. Specifically, the cameras capture images, and the sensors detect the positions and movements of customers and store staff. The input is the camera images and sensor detection data, and the output is a combination of these data. For example, the camera captures a customer stopping in front of a shelf, and the sensor records the duration of their stay.

[1471] Step 2:

[1472] The data collected by the device is stored in local storage and filtered for noise and incomplete data. Specifically, it removes noise from the video data and complements missing sensor data. The input is camera video and sensor detection data, and the output is clean, filtered data. For example, it improves image quality when the video is unclear and interpolates continuous sensor operation data.

[1473] Step 3:

[1474] The terminal compresses the cleaned data and uploads it to a server using a secure protocol (e.g. HTTPS). The input is the filtered data and the output is the compressed data and confirmation of its destination. For example, the data may be compressed in gzip format and sent to the server overnight.

[1475] Step 4:

[1476] The server decompresses and deserializes the data it receives. Specifically, it converts compressed data back to its original format so that it can be analyzed. The input is compressed data, and the output is decompressed data. For example, it decompresses video data and sensor data compressed with Gzip.

[1477] Step 5:

[1478] The server preprocesses the decompressed and deserialized data and inputs it into the generative AI model. Specifically, it extracts the video data frame by frame and converts the sensor data into time series data. The input is the decompressed data, and the output is the input data for the generative AI model. For example, the video data can be converted into an analyzable format frame by frame, and sensor information converted into time series data can be created.

[1479] Step 6:

[1480] The server uses a generative AI model to analyze customer and store clerk behavior and generates suggestions for improving store operations based on the results. The input is the data input to the generative AI model, and the output is the analysis results and suggestions for improvement. For example, it identifies shelves where customers spend a lot of time but products are not selling, and suggests changing the placement of those products.

[1481] Step 7:

[1482] The server notifies the store manager of the improvement suggestion via the smartphone and head-mounted display. The input is the generated improvement suggestion, and the output is a confirmation that the notification has been sent. For example, the smartphone and head-mounted display display a suggestion such as "Please move product A from shelf X to shelf Y."

[1483] Step 8:

[1484] The user implements the improvement suggestions they receive, and the results are collected again using cameras and sensors. The input is the implementation results of the improvement suggestions, and the output is the operational data after the changes. For example, after changing product placement, new data is collected again using cameras and sensors to analyze changes in sales.

[1485] Step 9:

[1486] Based on the data recollected by the server, the effectiveness of the proposed improvements is evaluated and further proposals are made as necessary. The input is the operational data after the changes, and the output is an evaluation of the effectiveness and new proposals for improvements. For example, the system can confirm the increase in sales after changing product placement and further propose optimization of customer flow.

[1487] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1488] In order to implement the present invention, a system can be constructed and operated in the following manner.

[1489] System Overview

[1490] This system collects data from cameras and sensors installed inside and outside the store, and uses a generative AI model and emotion engine to analyze customer behavior, store clerk activities, and even user emotions, to propose improvements to store operations.The main components are as follows:

[1491] 1. Cameras and sensors: Placed inside and outside the store, they capture the movements of customers and store staff, as well as user emotions.

[1492] 2. Local terminal: collects data from cameras and sensors and performs primary processing and data cleaning.

[1493] 3. Server: Receives data uploaded from local devices and analyzes it using generative AI models and emotion engines.

[1494] 4. User (store manager): Receives improvement suggestions from the server and implements specific actions.

[1495] Explanation of program processing

[1496] Data collection

[1497] 1. The device collects data in real time from cameras and sensors installed inside and outside the store. The cameras capture images, and the sensors collect data such as people's location and movements. In addition, an emotion engine analyzes the user's emotions from their facial expressions and voice.

[1498] Example: A camera captures a customer's movements and facial expressions as they pause in front of a shelf, and sensors record their dwell time and emotions.

[1499] Data storage and cleaning

[1500] 2. The device temporarily stores the collected data in local storage, filtering out noise and incomplete data when storing.

[1501] Example: Removing noise from video data and filling in missing parts of sensor data.

[1502] Uploading data

[1503] 3. The device compresses the filtered data and uploads it to the server using the specified secure protocol.

[1504] Example: Compress data in gzip format and send it to the server using HTTPS.

[1505] Data analysis

[1506] 4. The server de-realizes the uploaded data and restores it to its original form.

[1507] Specific operation: The received compressed file is decompressed and the original video data and sensor data are restored.

[1508] 5. The server preprocesses the recovered data and converts it into a format that can be input to the generative AI model and emotion engine.

[1509] Examples: Splitting video data into frames, converting sensor data into time series data, and formatting facial expression data for analysis.

[1510] 6. The server inputs the preprocessed data into the generative AI model and emotion engine to analyze customer behavior, store clerk behavior, and user emotions.

[1511] Example: The AI ​​model analyzes the customer's length of stay, their movement in front of the product, the timing of the store clerk's call, changes in facial expression, etc.

[1512] 7. The server generates suggestions for improving store operations based on the analysis results from the generated AI model and emotion engine. The suggestions also take into account the user's emotional state.

[1513] Example: An emotion engine infers customer satisfaction from their facial expressions, identifies products and placements that cause dissatisfaction, and suggests areas for improvement.

[1514] Notification of improvement proposals

[1515] 8. The device notifies the store manager of the improvement suggestions received from the server, for example, via a mobile device or dashboard UI.

[1516] Example: The dashboard might say, "Move product A from shelf X to shelf Y and have the sales associate provide specific advice to the customer."

[1517] Implementing improvement proposals and monitoring their effectiveness

[1518] 9. Implement the improvement suggestions received from the user and record the results.

[1519] Example: Adjusting merchandise movements and store staff actions and inputting the results into the system.

[1520] 10. The server collects data again and monitors the effectiveness of the proposed improvements.

[1521] Example: Analyzing new customer behavior data, sales data, and customer facial expression data to evaluate performance and customer satisfaction after improvements.

[1522] Specific examples

[1523] 1. The device collects behavioral and emotional data of 500 customers every day, filters out noise, and stores it in local storage.

[1524] 2. The device uploads the compressed data to the server overnight.

[1525] 3. The server decompresses and preprocesses the data, then analyzes it with a generative AI model and emotion engine. For example, it might detect that a particular product is not selling well, but many customers stop in front of it with unhappy expressions.

[1526] 4. The server generates improvement proposals, including suggestions for changing product placement and training for store staff, and notifies them.

[1527] 5. The user implements the suggestions and collects the performance results again.

[1528] 6. The server analyzes the new data, monitors the effectiveness of the suggested improvements, and makes further recommendations.

[1529] In this way, by implementing the present invention, it is possible to effectively improve sales and customer experience in store operations, as well as increase customer satisfaction.

[1530] The processing flow will be explained below.

[1531] Step 1:

[1532] The device collects data in real time from multiple cameras and sensors installed inside and outside the store.

[1533] How it works: Cameras capture footage of customers, and sensors collect their location, movement, and user audio data.

[1534] Step 2:

[1535] The data acquired by the device is temporarily stored in local storage.

[1536] Specific behavior: Stores captured video frames, sensor data, and audio data in a database or file system.

[1537] Step 3:

[1538] The device performs noise filtering on the stored data to remove unnecessary data.

[1539] Specific operation: Filters out low-resolution video, misdetected sensor data, and audio data containing noise, leaving only the necessary data.

[1540] Step 4:

[1541] The device compresses the filtered data and uploads it to the server.

[1542] Specific operation: Compress the data into gzip format and send it securely to the server via HTTPS.

[1543] Step 5:

[1544] The server unpacks the uploaded data and restores it to its original format.

[1545] Specific operation: The received compressed file is decompressed and restored to the original video data, sensor data, and audio data.

[1546] Step 6:

[1547] The server preprocesses the recovered data and converts it into a format that can be input to the generative AI model and emotion engine.

[1548] Specific operations: Video data is divided into frames, sensor data is organized into time series data, and audio data is converted into text.

[1549] Step 7:

[1550] The server inputs the preprocessed data into a generative AI model and emotion engine to analyze customer behavior, store clerk behavior, and user emotions.

[1551] Specific operation: The generative AI model analyzes the customer's dwell time and movement in front of the product, and the emotion engine analyzes the customer's emotions from facial expressions and voice data.

[1552] Step 8:

[1553] The server generates improvement proposals for store operations based on the analysis results.

[1554] Specific operation: The results of the generative AI model and the emotion engine are combined to compile suggestions such as changing product placement, improving the way store staff serve customers, and providing services based on customer emotions.

[1555] Step 9:

[1556] The terminal notifies the store manager (user) of the improvement proposal received from the server.

[1557] Specific actions: Display suggestions on mobile devices and dashboard UI, and clearly indicate specific actions.

[1558] Step 10:

[1559] Implement the improvement suggestions received by the user and record the results.

[1560] Specific actions: Change product placement, give instructions to store staff, implement measures to improve customer service, and feed the results back into the system.

[1561] Step 11:

[1562] The server collects data again and monitors the effectiveness of the improvement suggestions.

[1563] Specific actions: Analyze new customer behavior data, sales data, and customer facial expression data to evaluate the effectiveness of implemented improvement measures.

[1564] Example 2

[1565] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1566] Conventional customer behavior analysis systems for store operations are limited to providing feedback on customer behavior data and store clerk responses, and are unable to accurately grasp the user's emotional state and reflect it in improvement proposals. Furthermore, the analysis targets noisy or incomplete data, resulting in reduced proposal accuracy. The present invention aims to solve these problems and provide more accurate store operation improvement proposals.

[1567] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for uploading the filtered data to the server, a means for derealizing the uploaded data, preprocessing it, and converting it into an analyzable format, and a means for inputting the preprocessed data into a generative AI model and an emotion analysis engine, analyzing customer behavior and clerk behavior, and estimating user emotions. This enables highly accurate data analysis and improvement suggestions that include the user's emotional state.

[1568] A "camera" is a device for capturing video data.

[1569] A "sensor" is a device that detects physical changes and outputs them as data, specifically one that detects a person's position or movement.

[1570] "Local storage" is an internal storage device for temporarily storing data.

[1571] "Filtering" refers to the process of removing noise and imperfections from data.

[1572] "Upload" refers to the process of sending data from a local terminal to a server.

[1573] "Deserialization" is the process of returning serialized data to its original form.

[1574] "Preprocessing" refers to the initial processing of data to prepare it in an analyzable format.

[1575] A "generative AI model" is an algorithm that uses machine learning to analyze data and generate a specific result.

[1576] An "emotion analysis engine" is an algorithm or system for inferring emotions from a user's facial expressions and voice.

[1577] "Customer behavior" refers to data on customer behavior such as location, movement patterns, and length of stay within the store.

[1578] "Employee behavior" refers to data on employee behavior such as their location within the store, movement patterns, and how they interact with customers.

[1579] "Analysis" refers to the act of processing data to extract useful information and patterns.

[1580] "Improvement proposals" refer to proposals based on analysis results aimed at optimizing store operations and customer service.

[1581] "Notification" refers to the process of communicating information to store managers, specifically via mobile devices or dashboards.

[1582] The following describes in detail an embodiment of the present invention: The present invention is a system that utilizes data from cameras and sensors installed inside and outside a store, and analyzes customer behavior, store clerk behavior, and user emotions using a generative AI model and an emotion analysis engine.

[1583] System Overview

[1584] The system mainly includes the following hardware and software:

[1585] Cameras: Multiple cameras are installed inside and outside the store to capture video data.

[1586] Sensors: There are infrared sensors and various motion sensors that collect location information of customers and store staff.

[1587] Device: Stores collected data in local storage and filters out noise and incomplete data.

[1588] Server: De-realizes the filtered data, pre-processes it, and converts it into an analyzable format. It then analyzes it using a generative AI model and sentiment analysis engine to generate improvement suggestions.

[1589] Data collection and cleaning

[1590] The device collects data in real time from cameras and sensors. For example, the camera captures video at 30 frames per second, and sensors acquire location data of customers and store staff. This data is stored in local storage, where noise and incomplete data are filtered out. For example, ghost noise in the video data is removed and incomplete sensor data is complemented.

[1591] Data upload and de-realization

[1592] The device compresses the filtered data in gzip format and uploads it to the server via secure communication using the HTTPS protocol. The server then decompresses and derealizes the received data. For example, it splits the decompressed video data into frames and stores the sensor data in a database.

[1593] Preprocessing and analysis

[1594] The server preprocesses the derealized data and prepares it in a format that can be analyzed by the generative AI model and emotion analysis engine. This includes resizing the video data, extracting frames, and normalizing the sensor data. For example, the server resizes the video data to 256x256 pixels, which is easier for the AI ​​model to handle, and fills in missing parts of the sensor data to prepare it as time-series data.

[1595] The server inputs the preprocessed data into a generative AI model and an emotion analysis engine to analyze customer behavior, store clerk behavior, and user emotions. The AI ​​model analyzes the customer's length of stay, their movement, and the timing of the clerk's interactions, while the emotion analysis engine estimates the customer's emotional state based on facial expressions and voice data. For example, the generative AI model analyzes the customer's length of stay, and the emotion analysis engine analyzes the customer's dissatisfied facial expression.

[1596] Improvement suggestions and notifications

[1597] The server generates suggestions for improving store operations based on the analysis results and the customer's emotional state. For example, suggestions may be generated that include changing product placement or specific guidelines for store staff behavior. These suggestions are notified to store managers via their devices. Notification methods include a dashboard UI or a mobile device. As a specific example, the dashboard may display a message such as, "Move product A from shelf X to shelf Y, and have store staff provide specific advice to customers."

[1598] Implementing improvement proposals and monitoring their effectiveness

[1599] The user implements the improvement proposal and records the results. For example, changing the product layout and adjusting the behavior of the store clerk. The data after the improvement is collected again and entered into the system, allowing the server to analyze it again, monitor the effectiveness of the proposal, and make further improvement proposals. For example, new customer behavior data, sales data, and customer satisfaction data are evaluated, and feedback is provided for further improvements.

[1600] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1601] Step 1:

[1602] The device acquires data in real time from the camera and sensors. The input data is video data from the camera and position and movement data from the sensors. Specifically, the camera captures video at 30 frames per second, and the sensors acquire position information of customers and store staff. The output is raw video data and sensor data.

[1603] Step 2:

[1604] The data collected by the device is temporarily stored in local storage and noise and incomplete data are filtered. The input data are the raw video data and sensor data acquired in the previous step. Specifically, the process removes ghost noise contained in the video data and complements incomplete sensor data. The output is filtered video data and sensor data.

[1605] Step 3:

[1606] The terminal compresses the filtered data in gzip format and uploads it to the server using the HTTPS protocol. The input data is the filtered video data and sensor data. Specifically, the data is compressed in gzip format and transmitted securely using HTTPS. The output is the compressed data uploaded to the server.

[1607] Step 4:

[1608] The server de-realizes the uploaded data and restores it to its original format. The input data is compressed video data and sensor data. Specifically, it decompresses the received gzip format file and restores it to video data and sensor data. The output is the decompressed and restored video data and sensor data.

[1609] Step 5:

[1610] The server preprocesses the restored data and converts it into a format that can be analyzed by the generative AI model and emotion analysis engine. The input data is the derealized video data and sensor data. Specifically, the server splits the video data into frames and resizes it to 256x256 pixels. It also converts the sensor data into time-series data and extracts facial expression and movement features. The output is the preprocessed data.

[1611] Step 6:

[1612] The server inputs the preprocessed data into a generative AI model and emotion analysis engine, which analyzes customer behavior, store clerk behavior, and user emotions. The input data is preprocessed video data and sensor data. Specifically, the generative AI model analyzes customer movements and length of stay, and the emotion analysis engine infers the user's emotions from facial expression data. The output is the analysis results.

[1613] Step 7:

[1614] The server generates improvement proposals for store operations based on the analysis results. The input data are the analysis results. Specific operations include identifying products and placements that cause high customer dissatisfaction from the analysis results and proposing improvements accordingly. The output is specific improvement proposals.

[1615] Step 8:

[1616] The device notifies the user of the improvement proposal received from the server. The input data is the generated improvement proposal. As a specific operation, the proposal content is displayed through the dashboard UI or mobile device. The output is the improvement proposal notified to the user.

[1617] Step 9:

[1618] The user implements the improvement proposals received and records the results. The input data is the notified improvement proposal. Specific actions include moving products and adjusting the actions of store clerks, and inputting the resulting new behavioral and sales data into the system. The output is the implemented improvement proposals and new data.

[1619] Step 10:

[1620] The server re-analyzes the newly collected data and monitors the effectiveness of the proposed improvements. The input data is the new data after execution. Specific operations include evaluating the analysis results and determining how effective the proposals were. Based on this, further improvement proposals are made. The output is an evaluation of the effectiveness of the proposed improvements and new proposals.

[1621] (Application example 2)

[1622] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1623] In recent years, improving store operation efficiency and customer satisfaction have become important issues in the retail industry, but conventional methods have difficulty in conducting detailed analysis of customer behavior and emotions, resulting in a lack of concrete improvement proposals based on this analysis.In addition, while employee activities need to be monitored and supervised efficiently, current systems are unable to adequately achieve this.

[1624] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for acquiring information in real time from observation devices and sensing devices, means for saving the acquired information in a local storage device and filtering out useless or incomplete data, means for transferring the filtered information to a computer, means for inputting the transferred information into a generative AI model and a sentiment analysis engine to analyze customer behavior, employee activities, and user emotions, and means for generating store operation improvement proposals from the analysis results and notifying the manager. This makes it possible to perform detailed analysis of customer behavior, employee activities, and user emotions and provide specific and effective store operation improvement proposals.

[1625] "Observation devices" are devices installed to monitor and record the behavior and situations of customers and employees.

[1626] A "sensing device" is a device that collects information about the location and movement of customers and employees, as well as environmental information.

[1627] "Information" refers to data acquired in real time from observation and sensing devices, and includes video, audio, movement patterns, emotional states, and the like.

[1628] A "local storage device" is a device for temporarily storing information obtained from observation and sensing devices.

[1629] "Filtering" is the process of removing useless or incomplete data from stored information.

[1630] "Computer" refers to a server or cloud computing system for analyzing information transferred from local storage.

[1631] A "generative AI model" is a machine learning model trained to analyze customer and employee behavior and emotions.

[1632] An "emotion analysis engine" is software that analyzes emotions from a user's facial expressions and voice.

[1633] "Analysis results" refers to information about customer behavior, employee activities, and emotional states obtained by the generative AI model and sentiment analysis engine.

[1634] "Improvement proposals" are specific action plans for optimizing store operations and improving customer satisfaction based on the analysis results.

[1635] A "manager" is a person who is responsible for overseeing store operations and implementing improvement suggestions.

[1636] Summary of the Invention

[1637] This invention is a system that uses a generative AI model and an emotion analysis engine to analyze customer behavior, employee activity, and user emotions based on data collected from observation and sensing devices in a physical store, and provides suggestions for improving store operations. This system can improve customer satisfaction and streamline store operations.

[1638] Detailed system description

[1639] Data collection

[1640] The system collects information in real time from observation devices (cameras) and sensing devices (sensors) installed inside and outside the store. The observation devices record the behavior of customers and employees, while the sensing devices collect information on their location, behavior, and environment.

[1641] Storing and Filtering Data

[1642] The collected information is stored on a local storage device (e.g., a device such as a Raspberry Pi). When stored, useless or incomplete data is filtered out, leaving only the necessary information.

[1643] Data Transfer

[1644] The filtered information is converted into a compressed format (eg, gzip format) and transferred to a computer (a server or a cloud computing system) via a secure communication means (eg, HTTPS protocol).

[1645] Data analysis

[1646] The server receives the transferred information and inputs it into a generative AI model and sentiment analysis engine, which analyzes customer behavior, employee activity, and the user's emotional state. The generative AI model is trained using machine learning frameworks such as TensorFlow and PyTorch, and the sentiment analysis engine uses Microsoft Azure Emotion API and Amazon Rekognition.

[1647] Generate and notify improvement suggestions

[1648] Based on the analysis results, the server generates specific suggestions for improving store operations. These suggestions are notified to the manager in real time, who then acts accordingly. Notifications are sent via a smartphone app (e.g., an app developed based on Flutter or React Native).

[1649] Examples of specific examples and prompts

[1650] For example, if many customers stop in front of a certain product but do not end up purchasing it, the generative AI model will determine that there is a problem with the product's placement or explanation, and will suggest moving the product or improving the explanation.In addition, if the emotion analysis engine detects low satisfaction from customers' facial expressions, it will generate suggestions to encourage employees to take a more proactive approach.

[1651] An example prompt is:

[1652] "Customers who pause in front of product A spend a lot of time there, but the percentage of them actually purchasing is low. The sentiment analysis engine indicates that customer satisfaction is low. Therefore, move product A from shelf X to shelf Y, and have an associate provide specific advice to the customer."

[1653] This allows for detailed analysis of customer behavior, employee activities, and user sentiment, making it possible to provide specific and effective suggestions for improving store operations.

[1654] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1655] Step 1:

[1656] The terminal acquires information in real time from observation devices (cameras) and detection devices (sensors) installed inside and outside the store. The input is data from each observation device and detection device, which is received by the terminal. Specifically, the camera records customer behavior, and the sensor collects movement and location information.

[1657] Step 2:

[1658] The device stores the acquired information in a local storage device. Here, it receives the acquired raw data as input and filters out noise and incomplete data. Specifically, it removes noise from the video data and fills in missing parts of the sensor data. The output is clean, filtered data.

[1659] Step 3:

[1660] The terminal converts the filtered data into a compressed format and transfers it to the server. It receives clean data as input and compresses it, for example, using gzip format. It then sends it to the server using a secure communication method (for example, HTTPS protocol). The specific operations include compressing the data, establishing a secure network connection, and transferring the data. The compressed data arrives at the server as output.

[1661] Step 4:

[1662] The server receives the transferred data and decompresses it. It receives compressed data as input and restores it to its original information format. Specifically, it decompresses the received gzip file and returns the video data and sensor data to their original state.

[1663] Step 5:

[1664] The server preprocesses the restored data into a format that can be input to the generative AI model and emotion analysis engine. It receives the decompressed data as input and converts it into a format for analysis. Specifically, it splits the video data into frames, converts the sensor data into time-series data, and formats the facial expression data for analysis. The preprocessed data is obtained as output.

[1665] Step 6:

[1666] The server inputs the preprocessed data into the generative AI model and emotion analysis engine, which analyzes customer behavior, employee activities, and user emotions. The preprocessed data is received as input and analyzed. Specifically, the generative AI model analyzes the customer's length of stay, the timing of employee calls, changes in facial expressions, etc. The analysis results are obtained as output.

[1667] Step 7:

[1668] The server generates improvement proposals for store operations based on the analysis results. It receives the analysis results as input and creates improvement proposals based on them. Specifically, it proposes optimal product placement and employee action plans based on the results of the generative AI model and sentiment analysis engine. The generated improvement proposals are obtained as output.

[1669] Step 8:

[1670] The device notifies the administrator of the improvement proposal received from the server. The device receives the generated improvement proposal as input and communicates it to the administrator. Specifically, the device displays and notifies the administrator of the improvement proposal through a smartphone app (for example, one using Flutter or React Native). The output is that notification to the administrator has been completed.

[1671] Step 9:

[1672] The user implements the notified improvement suggestions and re-enters the results into the device. The results of the suggestions are received as input and fed back to the system. Specific operations include moving products or changing employee behavior in accordance with the presented improvement suggestions, and recording the results in the app. Feedback data is obtained as output.

[1673] Step 10:

[1674] The server collects data again and monitors the effectiveness of the proposed improvements. It receives feedback data as input and evaluates its effectiveness. Specifically, it analyzes the new data after the improvements and evaluates customer behavior, sales data, and customer satisfaction. The output is performance data after the improvements.

[1675] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1676] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1677] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1678] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1679] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1680] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1681] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1682] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1683] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1684] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1685] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1686] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1687] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1688] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1689] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1690] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1691] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1692] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1693] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1694] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1695] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1696] The following is further disclosed regarding the above embodiment.

[1697] (Claim 1)

[1698] A means of acquiring data in real time from multiple cameras and sensors installed inside and outside the store;

[1699] means for storing data acquired from said cameras and sensors in a local storage and filtering out noise and incomplete data;

[1700] means for uploading the filtered data to a server;

[1701] A means for inputting the uploaded data into a generative AI model to analyze customer behavior and store clerk behavior;

[1702] The system includes a means for generating improvement proposals for store operations from the analysis results and notifying the store manager.

[1703] (Claim 2)

[1704] 10. The system of claim 1, further comprising means for converting data obtained from the cameras and sensors into a compressed format and transmitting the data to a server via a secure protocol.

[1705] (Claim 3)

[1706] The system according to claim 1, wherein the generative AI model includes means for analyzing the customer's length of stay and the timing of the store clerk's greeting, and proposing optimal product placement and greeting timing.

[1707] "Example 1"

[1708] (Claim 1)

[1709] means for acquiring data in real time from a plurality of monitoring and detecting devices installed inside and outside the store;

[1710] means for storing data acquired from said monitoring and sensing devices in a local storage device and filtering out noise and incomplete data;

[1711] means for compressing the filtered data and transmitting it to a server via a secure protocol;

[1712] A means for decompressing, derealizing, preprocessing, and inputting the data received by the server into a generative AI model to analyze customer behavior and employee behavior;

[1713] The system includes a means for generating improvement proposals for store operations from the analysis results and notifying the store manager.

[1714] (Claim 2)

[1715] The system according to claim 1, further comprising means for notifying the store manager of improvement suggestions, such as changing the placement of specific products, based on the results of the analysis by the generative AI model.

[1716] (Claim 3)

[1717] 2. The system according to claim 1, further comprising means for the store manager to implement the improvement suggestions received, re-collect and analyze the resulting new data, and monitor the effectiveness of the improvements.

[1718] "Application Example 1"

[1719] (Claim 1)

[1720] A means of acquiring data in real time from multiple cameras and sensors installed inside and outside the store;

[1721] means for storing data acquired from said cameras and sensors in a local storage and filtering out noise and incomplete data;

[1722] means for uploading the filtered data to a server;

[1723] A means for inputting the uploaded data into a generative AI model to analyze customer behavior and store clerk behavior;

[1724] a means for generating store operation improvement proposals based on the analysis results and notifying the store manager;

[1725] a means for monitoring operational data after the implementation of the improvement proposal and evaluating the effectiveness of the improvement proposal;

[1726] A way to view the camera feed in real time using a smartphone or head-mounted display, and review analysis results and improvement suggestions.

[1727] A system including the smartphone and a means for receiving notification of the improvement suggestion through a head-mounted display.

[1728] (Claim 2)

[1729] 10. The system of claim 1, further comprising means for converting data obtained from the cameras and sensors into a compressed format and transmitting the data to a server via a secure protocol.

[1730] (Claim 3)

[1731] The system according to claim 1, wherein the generative AI model includes means for analyzing the customer's length of stay and the timing of the store clerk's greeting, and proposing optimal product placement and greeting timing.

[1732] "Example 2: Combining Emotion Engines"

[1733] (Claim 1)

[1734] A means of acquiring data in real time from multiple cameras and sensors installed inside and outside the store;

[1735] means for storing data acquired from said cameras and sensors in a local storage and filtering out noise and incomplete data;

[1736] means for uploading the filtered data to a server;

[1737] means for de-realizing, pre-processing and converting the uploaded data into an analyzable format;

[1738] A means for inputting the preprocessed data into a generative AI model and a sentiment analysis engine, analyzing customer behavior and store clerk behavior, and estimating user sentiment;

[1739] The system includes a means for generating improvement proposals for store operations from the analysis results and emotional state and notifying the store manager.

[1740] (Claim 2)

[1741] 10. The system of claim 1, further comprising means for converting data obtained from the cameras and sensors into a compressed format and transmitting the data to a server via a secure protocol.

[1742] (Claim 3)

[1743] The system of claim 1, wherein the generative AI model includes means for analyzing a customer's length of stay, movement patterns, and timing of an employee's interaction, and for suggesting optimal product placement, methods of interacting with the customer, and actions to improve customer satisfaction.

[1744] "Application example 2 when combining emotion engines"

[1745] (Claim 1)

[1746] A means for acquiring information in real time from a plurality of observation devices and sensing devices installed inside and outside the store;

[1747] means for storing information obtained from said observation and sensing devices in a local storage device and filtering out useless or incomplete data;

[1748] means for transferring the filtered information to a computer;

[1749] A means for inputting the transferred information into a generative AI model and a sentiment analysis engine to analyze customer behavior, employee activity, and user sentiment;

[1750] The system includes a means for generating improvement proposals for store operations from the analysis results and notifying the manager.

[1751] (Claim 2)

[1752] 10. The system of claim 1, further comprising means for converting information obtained from said observation and sensing devices into a compressed format and transmitting the information to a computer via secure communications means.

[1753] (Claim 3)

[1754] The system of claim 1, wherein the generative AI model and emotion analysis engine include means for analyzing the customer's length of stay and the timing of employee calls, as well as the user's emotions, and for proposing optimal product placement and improvement measures. [Explanation of symbols]

[1755] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. A means of acquiring data in real time from multiple cameras and sensors installed inside and outside the store; means for storing data acquired from said cameras and sensors in a local storage and filtering out noise and incomplete data; means for uploading the filtered data to a server; A means for inputting the uploaded data into a generative AI model to analyze customer behavior and store clerk behavior; The system includes a means for generating improvement proposals for store operations from the analysis results and notifying the store manager.

2. 10. The system of claim 1, further comprising means for converting data obtained from the cameras and sensors into a compressed format and transmitting the data to a server via a secure protocol.

3. The system according to claim 1, wherein the generative AI model includes means for analyzing the customer's stay time and the timing of the store clerk's greeting, and proposing optimal product placement and greeting timing.

Images