system

Abstract

We provide the system. [Solution] A means for analyzing video data acquired from a surveillance device to detect individual people, A means for recognizing the face of a detected person and estimating their age and gender, A method for identifying repeat customers by matching them with an existing database based on specific characteristics, A means to aggregate visitor attributes and generate reports based on the collected data, A system that includes means of providing improvement advice to each store.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor, the method including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a character of the chatbot, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] Modern monitoring devices are mainly used for crime prevention and evidence collection after an incident, but their data has not been fully utilized for business purposes. Therefore, it is required to grasp in detail the characteristics and behavior patterns of visitors and utilize them for optimizing store operations and constructing marketing strategies. In particular, the purpose is to improve the efficiency of store operations by analyzing attribute data such as the age, gender, and repeat rate of visitors, as well as the customer flow information in the store and the degree of interest in products that were not purchased.

Means for Solving the Problems

[0005] The present invention provides a system that includes means for analyzing video data acquired from a monitoring device to detect individual people, means for recognizing the faces of detected people and estimating their age and gender, means for identifying repeat customers by comparing them with an existing database based on specific characteristics, means for aggregating visitor attributes based on the collected data and generating reports, and means for providing improvement advice to each store. Furthermore, by providing a system that includes means for analyzing customer movement data within a store to calculate the appropriate placement of products and means for notifying store managers of advice based on the calculated placement, as well as means for collecting data on products that customers showed interest in but did not purchase, and means for evaluating the level of interest and analyzing it as marketing data, the present invention solves the above problems and realizes a detailed understanding of customer behavior and effective utilization of that data.

[0006] A "surveillance device" is a system that includes hardware for acquiring video data, and primarily refers to a camera.

[0007] "Video data" refers to video data acquired by surveillance equipment, which includes visual information.

[0008] "Analysis" refers to the process of extracting specific information from acquired video data and converting it into an understandable format.

[0009] "Person" is a concept that refers to an individual human being detected within video data.

[0010] "Detection" refers to the process of finding a target object or person from video data under specific conditions.

[0011] "Facial recognition" is a technology that identifies a specific person based on facial images extracted from video data.

[0012] "Age" is attribute information that indicates the estimated age of the detected person.

[0013] "Gender" is attribute information indicating the presumed gender of the detected person.

[0014] "Specific feature" is data including unique facial attributes used for person recognition and discrimination.

[0015] "Database" is a storage device that systematically stores specific information and enables easy search and collation.

[0016] "Repeat customer" is a concept referring to a customer who has visited the store in the past and visits again.

[0017] "Attribute" refers to a set of data indicating characteristic information of visitors.

[0018] "Aggregation" refers to the process of organizing and statistically summarizing the collected data.

[0019] "Report" is a document created based on the aggregated data and shows the analysis results.

[0020] "Flow line data" is data including information indicating the movement route of customers in the store.

[0021] "Layout" refers to the location where each element of products and facilities is physically arranged.

[0022] "Degree of interest" is an index for evaluating the degree of interest shown by customers towards specific products or areas.

[0023] "Marketing data" is data utilized for sales promotion and market strategies based on customer behavior analysis.

Brief Explanation of Drawings

[0024] [Figure 1] It is a conceptual diagram showing an example of the configuration of the data processing system according to the first embodiment. [Figure 2]It is a conceptual diagram showing an example of the main functions of a data processing device and a smart device according to the first embodiment. [Figure 3] It is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] It is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] It is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] It is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] It is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] It is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] It shows an emotion map to which multiple emotions are mapped. [Figure 10] It shows an emotion map to which multiple emotions are mapped. [Figure 11] It is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] It is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] It is a sequence diagram showing the processing flow of the data processing system in Example 2 when an emotion engine is combined. [Figure 14] It is a sequence diagram showing the processing flow of the data processing system in Application Example 2 when an emotion engine is combined.

Embodiments for Carrying Out the Invention

[0025] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0026] First, let's explain the terminology used in the following explanation.

[0027] In the following embodiments, the signed processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Furthermore, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include CPU (Central Processing Unit), GPU (Graphics Processing Unit), GPGPU (General-Purpose computing on Graphics Processing Units), and APU (Accelerated Processing Unit).

[0028] In the following embodiments, signed RAM (Random Access Memory) is a memory that temporarily stores information and is used as work memory by the processor.

[0029] In the following embodiments, the signed storage is one or more non-volatile storage devices that store various programs and various parameters. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes.

[0030] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0031] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0032] [First Embodiment]

[0033] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0034] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0035] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0036] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0037] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0038] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0039] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0040] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0041] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0042] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0043] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0044] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0045] This invention relates to a system that analyzes video data acquired from a surveillance device, detects individual people, performs facial recognition and attribute estimation, and analyzes customer behavior based on the results. The program and processing of this system are described in detail below.

[0046] Explain the program's processing in natural language.

[0047] 1. Automatic counting of customer visitors

[0048] Surveillance devices are installed at the store entrance and inside the store to capture video of visitors in real time. The video data is sent to a server, which uses an AI model to detect people in the video. It performs facial recognition on each person and estimates their age and gender. This information is temporarily stored in a database to determine if a customer is a repeat customer. Specifically, the server compares the facial features with an existing database, and if there is a match, the person is registered as a repeat customer. If there is no match, the person is registered as a new visitor in the database. Finally, visitor attribute data is compiled, and daily and monthly reports are generated. These reports are notified to the store operations manager.

[0049] Specific example

[0050] server

[0051] The server receives the video data and uses an AI model to detect people.

[0052] The server performs facial recognition and estimates age and gender. For example, it might save the information in the database as "male in his 30s."

[0053] The server matches the facial features against the database, counts the customer as a "repeat customer," and updates the database.

[0054] 2. Analysis of in-store movement and layout advice

[0055] Multiple monitoring devices within the store capture customer movement data in real time and transmit it to a server. The server analyzes this data to track customer movements. It correlates the movement data with product placement data to identify which products are attracting the most interest. Using an AI model, the server calculates the optimal product placement based on the movement data and notifies store managers with specific advice.

[0056] Specific example

[0057] server

[0058] The server analyzes customer movement patterns and determines that "Customer A has shown interest in product X."

[0059] Based on the customer flow data, the server advises and notifies the user to place product Y near the entrance.

[0060] 3. Analysis of interest in products that were not purchased.

[0061] Data on products that customers picked up but did not purchase is extracted from the video and sent to a server. The server analyzes this data to evaluate the customer's level of interest. Specifically, it correlates the customer's movement with the data on the target products and calculates an interest score. This data is then notified to marketing personnel and used for sales promotion activities.

[0062] Specific example

[0063] server

[0064] The server identifies product B, which the user picked up but did not purchase, and rates its level of interest as "high".

[0065] The server generates a marketing report based on the evaluation data and notifies the client of the marketing measures that should be adopted.

[0066] 4. Observation of customer behavior by time of day and operational advice

[0067] Monitoring devices capture video data of visitors at specific time intervals and transmit it to a server. The server organizes this data by time period and analyzes customer behavior patterns. Based on customer inflow and outflow, product attention levels, and sales data for specific time periods, an AI model calculates areas for improvement in store operations. This advice is communicated to store managers for their implementation.

[0068] Specific example

[0069] server

[0070] The server analyzes customer data by time of day and identifies that "customer traffic increases around 2 PM."

[0071] Based on the analysis results, the server will notify you of the advice to "deploy additional staff by 2 PM."

[0072] The above describes the operation and specific examples of the system in an embodiment for carrying out the present invention. This makes it possible to efficiently analyze customer behavior data and optimize store operations.

[0073] The following describes the processing flow.

[0074] Program processing flow

[0075] Automatic counting of customer visitors

[0076] Step 1:

[0077] The terminal acquires video data from the monitoring device and transmits it to the server in real time.

[0078] Step 2:

[0079] The server analyzes the received video data using an AI model to detect people in the video.

[0080] Step 3:

[0081] The server performs facial recognition on detected individuals and estimates their age and gender.

[0082] Step 4:

[0083] The server temporarily stores the estimated facial data and compares it with an existing database based on specific features.

[0084] Step 5:

[0085] The server identifies repeat customers and new visitors based on the results of matching them with the database, and updates and registers their respective data.

[0086] Step 6:

[0087] The server aggregates visitor attribute data, generates daily and monthly reports, and notifies store operations staff.

[0088] Analysis of in-store movement and layout advice

[0089] Step 1:

[0090] The terminal acquires video data from multiple monitoring devices within the store and transmits it to the server in real time.

[0091] Step 2:

[0092] The server analyzes the received video data using an AI model to track customer movement.

[0093] Step 3:

[0094] The server correlates customer movement data with product placement data to identify which products customers showed interest in.

[0095] Step 4:

[0096] The server calculates the optimal placement of products based on customer movement data.

[0097] Step 5:

[0098] Based on the calculated server placement, the server generates specific advice and notifies the store operations manager.

[0099] Analysis of interest in products that were not purchased.

[0100] Step 1:

[0101] The terminal acquires video data from the monitoring device and transmits it to the server in real time.

[0102] Step 2:

[0103] The server analyzes the video data to identify products that customers picked up but did not purchase.

[0104] Step 3:

[0105] The server tracks customer movement and evaluates their level of interest in products they did not purchase.

[0106] Step 4:

[0107] The server stores interest data in a database and organizes it as marketing data.

[0108] Step 5:

[0109] The server generates marketing reports and notifies marketing personnel.

[0110] Observation of customer behavior by time of day and operational advice

[0111] Step 1:

[0112] The terminal captures video data from the monitoring device, categorized by time period, and transmits it to the server in real time.

[0113] Step 2:

[0114] The server organizes the video data received by time period and compiles data on customer inflow and outflow.

[0115] Step 3:

[0116] The server analyzes data for each time period to identify customer behavior patterns.

[0117] Step 4:

[0118] Based on the data analyzed by the server, an AI model calculates areas for improvement in operations.

[0119] Step 5:

[0120] Based on the areas for improvement calculated by the server, it generates specific advice and notifies the store operations manager.

[0121] (Example 1)

[0122] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0123] Traditional systems made it difficult to effectively collect and analyze customer behavior data from store visitors and propose improvements to store operations. Furthermore, strategic marketing measures based on individual customer attributes and interests were not adequately implemented. In particular, identifying repeat customers, optimizing product placement, and evaluating interest in products that were not purchased were challenging, resulting in a lack of concrete, data-driven advice for efficient store operations and increased sales.

[0124] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0125] In this invention, the server includes means for analyzing video data acquired from a monitoring device to detect individual people, means for recognizing the faces of detected people and estimating their age and gender, means for identifying repeat customers by comparing them with an existing database based on characteristic information, means for aggregating visitor attributes based on collected data and generating reports, means for providing improvement advice to each store, means for analyzing customer movement data to calculate the optimal placement of products and notifying each store, means for collecting data on products that customers picked up but did not purchase and evaluating their level of interest, means for acquiring video data from installed monitoring devices at different times of the day and analyzing customer behavior patterns, and means for calculating and notifying improvements to store operations by correlating the analysis results with sales data. This makes it possible to efficiently collect and analyze customer behavior data and optimize store operations.

[0126] A "surveillance device" is a device installed at the entrance or inside a store to acquire video data.

[0127] "Video data" refers to image and video information acquired by surveillance equipment.

[0128] "Analysis" refers to techniques that process information such as video data to detect individual people or analyze movement patterns.

[0129] "Person detection" is a method of identifying individual people from video data.

[0130] "Facial recognition" is a method of identifying a specific person by analyzing the face of a detected individual.

[0131] "Age and gender estimation" refers to a method of inferring a person's age group and gender based on person detection and facial recognition.

[0132] "Characteristic information" refers to information used to identify individual people, such as facial features and movement data.

[0133] A "database" is a system for organizing and storing collected data.

[0134] A "repeat customer" refers to a person who has visited the store before and is recorded in the existing database.

[0135] "Visitor attributes" refer to characteristics such as visitor age, gender, movement patterns, and purchase history.

[0136] "Report generation" refers to the process of compiling analysis results based on collected data and creating a report.

[0137] "Improvement advice" is a method of providing instructions on areas for improvement in store operations based on collected and analyzed data.

[0138] "Customer movement data" refers to data that shows the routes and behavioral patterns of customers as they move within a store.

[0139] "Product placement" refers to the location and method of displaying products within a store.

[0140] "Interest level" is a measure that evaluates the degree of interest a customer has in a product they picked up but did not purchase.

[0141] "Video data by time period" refers to video data acquired during a specific time period.

[0142] "Behavioral patterns" refer to data that shows the influx and outflow of customers and the trends of popular products during specific time periods.

[0143] This invention relates to a system that analyzes video data acquired from a surveillance device, detects individual people, performs facial recognition and attribute estimation, and analyzes customer behavior based on the results. The program and processing of this system are described below.

[0144] This system consists of monitoring devices, a server, and terminals. The monitoring devices are installed at the entrance and inside the store and acquire customer video data in real time. The acquired video data is transmitted to the server via the terminals.

[0145] The server uses an AI model to analyze the received video data. The AI ​​model utilizes generative AI technology to detect people in the video with high accuracy. At this stage, the server uses a facial recognition algorithm to identify the faces of detected individuals and estimate their age and gender. For example, it might store this information in a database as "a man in his 30s." The software used for this includes common facial recognition libraries and attribute estimation algorithms.

[0146] Furthermore, the server matches facial features against an existing database, counting repeat customers if a match is found. If there is no match, the customer is registered in the database as a new visitor. Finally, visitor attribute data is aggregated, and daily and monthly reports are generated. These reports are then communicated to store operations staff.

[0147] Next, the server analyzes customer movement data transmitted from multiple monitoring devices within the store to track customer behavior. It then correlates this movement data with product placement data to identify which products are attracting the most attention. Using an AI model, the server calculates the optimal product placement based on the movement data and provides specific advice to store managers.

[0148] The server also analyzes data on products that customers picked up but did not purchase, and evaluates the customer's level of interest. Specifically, it correlates customer movement with data on the target products and calculates an interest score. This data is then shared with marketing personnel and used for sales promotion activities.

[0149] Furthermore, monitoring devices capture video data of visitors at specific time intervals and transmit it to a server. The server organizes this data by time period and analyzes customer behavior patterns. Based on customer inflow and outflow, product attention levels, and sales data for specific time periods, an AI model calculates areas for improvement in store operations. This advice is then communicated to store managers for implementation.

[0150] For example, the server might analyze that "customer traffic increased during the 2 PM timeframe" and notify the store manager of the need to allocate additional staff.

[0151] Example of a prompt

[0152] "Analyze customer movement data recorded by security cameras to identify which products are attracting the most attention. Based on these results, propose the optimal product placement and notify the store operations manager."

[0153] The above outlines a specific implementation of this system. This makes it possible to efficiently analyze customer behavior data and optimize store operations.

[0154] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0155] Step 1:

[0156] The terminal acquires video data in real time from surveillance devices installed at the entrance and inside the store.

[0157] Specific operation: The monitoring device captures video data using a camera sensor, and the terminal receives this data.

[0158] Input: Video data from a monitoring device.

[0159] Output: Video data received by the terminal.

[0160] Step 2:

[0161] The device sends the acquired video data to the server.

[0162] Specific operation: The device uploads the received video data to the server via the network.

[0163] Input: Video data received by the device.

[0164] Output: Video data received by the server.

[0165] Step 3:

[0166] The server inputs the received video data into an AI model to detect people in the video.

[0167] Specific operation: The server inputs video data into the AI ​​model, and the model detects people in each video frame.

[0168] Input: Video data received by the server.

[0169] Output: Person detection results by the AI ​​model.

[0170] Step 4:

[0171] The server performs facial recognition on detected individuals and estimates their age and gender.

[0172] Specific operation: The server uses a facial recognition algorithm to analyze the detected face and estimate its age and gender. For example, it might save it in the database as "male in his 30s."

[0173] Input: Person detection results from an AI model.

[0174] Output: Person data with estimated age and gender.

[0175] Step 5:

[0176] The server matches the facial features against an existing database. If there is a match, the person is counted as a repeat customer. If there is no match, they are registered in the database as a new visitor.

[0177] Specific operation: The server compares the identified facial feature data with the database to determine whether it is a new or repeat customer.

[0178] Input: Individual data with estimated age and gender, and existing databases.

[0179] Output: Repeat customer list, New visitor list.

[0180] Step 6:

[0181] The server aggregates visitor attribute data and generates daily and monthly reports.

[0182] Specific operation: The server aggregates data on new and repeat customers, and a report generation program creates daily and monthly reports.

[0183] Input: Repeat customer list, New visitor list.

[0184] Output: Daily and monthly reports.

[0185] Step 7:

[0186] The server notifies the store operations manager of the report.

[0187] Specific operation: The server sends the generated report to the responsible person via email or dashboard.

[0188] Input: Daily and monthly reports.

[0189] Output: Report sent to store operations staff.

[0190] Step 8:

[0191] The terminal acquires customer movement data in real time from multiple monitoring devices within the store.

[0192] Specific operation: The monitoring device tracks customer movements within the store and captures movement data in real time.

[0193] Input: Customer traffic data from monitoring devices within the store.

[0194] Output: Movement data received by the terminal.

[0195] Step 9:

[0196] The terminal sends the movement data it acquires to the server.

[0197] Specific operation: The terminal receives movement data and sends it to the server via the network.

[0198] Input: Movement data received by the terminal.

[0199] Output: Movement data received by the server.

[0200] Step 10:

[0201] The server analyzes the movement data it receives to track customer movements.

[0202] Specific operation: The server processes the movement data using an analysis algorithm to analyze customer movement patterns.

[0203] Input: Movement data received by the server.

[0204] Output: Customer movement analysis results.

[0205] Step 11:

[0206] The server correlates customer flow data with product placement data to identify which products are attracting the most attention.

[0207] Specific operation: The server matches customer flow data with product placement data to identify products of high interest.

[0208] Input: Customer movement analysis results, product placement data.

[0209] Output: A list of products the customer has shown interest in.

[0210] Step 12:

[0211] The server calculates the optimal placement of products and notifies store managers with specific advice.

[0212] Specific operation: The server uses an optimal placement algorithm to generate product placement advice and notifies the person in charge.

[0213] Input: A list of products that the customer has shown interest in.

[0214] Output: Placement advice report.

[0215] Step 13:

[0216] The terminal extracts data from in-store monitoring devices about items that customers picked up but did not purchase, and sends it to a server.

[0217] Specific operation: The monitoring device tracks customer behavior, collects data on items picked up but not purchased, and transmits it to the terminal.

[0218] Input: Data on products that were not purchased.

[0219] Output: Data on unpurchased items received by the server.

[0220] Step 14:

[0221] The server analyzes data on unpurchased items to assess the customer's level of interest.

[0222] Specific operation: The server analyzes data on unpurchased items and calculates an interest score.

[0223] Input: Data on unpurchased items received by the server.

[0224] Output: A list of products with interest scores.

[0225] Step 15:

[0226] The server notifies marketing personnel of interest levels, who then use the data for sales promotion activities.

[0227] Specific operation: The server creates a marketing report based on interest level data and sends it to the person in charge.

[0228] Input: A list of products with interest scores.

[0229] Output: Marketing report.

[0230] Step 16:

[0231] The terminal acquires video data of visitors from the monitoring device at specific time intervals and sends it to the server.

[0232] Specific operation: The device captures video data at specified time intervals and sends it to the server.

[0233] Input: Time-series video data from monitoring equipment.

[0234] Output: Video data received by the server for each time period.

[0235] Step 17:

[0236] The server organizes video data by time period and analyzes customer behavior patterns.

[0237] Specific operation: The server collects data for each time period and analyzes behavioral patterns.

[0238] Input: Video data received by the server for each time period.

[0239] Output: Customer behavior patterns by time of day.

[0240] Step 18:

[0241] The server correlates the analysis results with sales data to calculate areas for improvement in store operations.

[0242] Specific operation: The server analyzes the correlation between behavioral patterns and sales data to identify areas for improvement.

[0243] Input: Customer behavior patterns by time of day, sales data.

[0244] Output: A report identifying areas for improvement.

[0245] Step 19:

[0246] The server notifies the store operations manager with improvement advice.

[0247] Specific action: The server sends a report indicating areas for improvement to the responsible person and notifies them.

[0248] Input: A report identifying areas for improvement.

[0249] Output: Improvement advice notified to store operations staff.

[0250] (Application Example 1)

[0251] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0252] Traditional store operations have made it difficult to adequately analyze customer behavior data, particularly regarding customer movement patterns, product placement, and analysis of unpurchased items, without utilizing real-time data. Furthermore, there was a lack of means to observe customer behavior at different times of the day and to obtain concrete suggestions for operational improvements based on that observation. As a result, optimizing store operations was hindered, making it difficult to contribute to increased sales and improved customer satisfaction.

[0253] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0254] In this invention, the server includes means for analyzing image data acquired from a monitoring device to detect individual people, means for recognizing the faces of detected people and estimating their age and gender, means for identifying repeat customers by comparing them with an existing information base based on specific features, means for aggregating visitor attributes based on collected data and generating reports, means for providing improvement suggestions to each facility, means for analyzing customer movement data and calculating the appropriate placement of products, means for notifying facility operators of the suggestions based on the calculated placement, means for acquiring data on products that customers showed interest in but did not purchase from collected image data, means for evaluating the level of interest and analyzing it as marketing data, means for organizing visitor data by time of day and analyzing customer behavior patterns, and means for calculating points for operational improvement in each time period and notifying operators. This makes it possible to efficiently collect and analyze customer behavior data within stores and to make operational improvement suggestions in real time.

[0255] A "surveillance device" refers to a device that acquires video data and performs tasks such as person detection and movement analysis.

[0256] "Image data" refers to video information acquired from surveillance devices and is used for person detection and movement analysis.

[0257] "Person detection" refers to a method of identifying individual people from image data.

[0258] "Facial recognition" refers to a method of identifying a person by analyzing the facial features of a detected individual.

[0259] "Age and gender estimation" refers to the act of inferring a person's age and gender from the results of facial recognition.

[0260] "Specific features" refer to facial characteristic information obtained through facial recognition, and are data used to compare this information with existing information bases.

[0261] "Existing information base" refers to a database containing customer data collected previously.

[0262] A "repeat customer" refers to a customer who has a history of visiting the store in the past and has been determined to be a returning customer based on matching specific characteristics.

[0263] "Collected data" refers to all data obtained from monitoring devices and analysis means.

[0264] "Visitor attributes" refer to characteristics such as the age, gender, and number of visits of customers who visit the store.

[0265] "Report generation" refers to the act of compiling collected data and creating a document that summarizes it in a specific format.

[0266] "Each facility" refers to the stores or business locations that are the subject of data analysis.

[0267] An "improvement suggestion" is a proposal that outlines specific measures to improve store operations based on collected data.

[0268] "Customer movement data" refers to information that shows the paths customers take within a store.

[0269] "Appropriate product placement" refers to advice on optimizing product placement based on customer traffic data.

[0270] A "facility operations manager" refers to a manager responsible for the operation of a store or business location.

[0271] "Data acquisition" refers to the act of extracting necessary information from image data.

[0272] "Interest level assessment" is the act of analyzing and evaluating the level of interest shown by customers.

[0273] "Marketing data" refers to data about customer interests and behavior extracted from collected data, and is used in marketing strategies.

[0274] "Visitor data by time slot" refers to customer visit information organized by specific time periods.

[0275] "Customer behavior patterns" refer to pattern information that shows how customers move around within a store and what actions they take.

[0276] "Areas for operational improvement" refers to the points necessary to improve store operations based on collected data.

[0277] This invention relates to a system that analyzes image data acquired from monitoring devices to track and analyze customer behavior in real time. This system collects customer attribute data, movement data, and purchasing behavior information, contributing to the optimization of facility operations. Specific embodiments of this system are described below.

[0278] The server uses multiple AI models to process image data. For example, it uses OpenCV to acquire image data and a FaceRecognitionModel to recognize and analyze people's faces. It also uses an ActionDetectionModel to analyze customer movement and behavior. Furthermore, it uses a SpatialAnalysisModel to analyze the collected movement data and calculate the appropriate placement of products. All of this data is then centrally managed and stored by DatabaseConnection.

[0279] Regarding the hardware and software to be used

[0280] Hardware: Surveillance equipment (cameras), servers, database servers, smartphones, head-mounted displays

[0281] Software: OpenCV, FaceRecognitionModel, ActionDetectionModel, SpatialAnalysisModel, DatabaseConnection

[0282] The server first acquires video data from the monitoring device. Next, it analyzes the video frames using OpenCV to detect faces. It extracts the facial features detected using FaceRecognitionModel and estimates age and gender based on that data. It also compares specific features with an existing information base to determine whether the person is a repeat customer or a new visitor. This information is stored in a database via DatabaseConnection and compiled into a report.

[0283] Next, the ActionDetectionModel is used to analyze customer movement data. For each frame, customer movement is analyzed and movement data is collected. This data is then analyzed by the SpatialAnalysisModel and used to optimize product placement. Based on these analysis results, the server provides improvement advice to facility managers.

[0284] Data on products that customers picked up but did not purchase is obtained by the Action Detection Model, and its degree of interest is evaluated. This evaluation is analyzed as marketing data, and a specific marketing strategy is formulated from the collected results.

[0285] Specific Example

[0286] For example, when a certain customer shows interest in a specific product but does not purchase it, the server analyzes their movement path and evaluates whether other customers also tend to be interested in the same product. Also, when there is a sudden increase in the influx of customers during a specific time period, the server analyzes the cause and makes specific proposals for operational improvement. This enables the operations staff to make decisions based on evidence.

[0287] As an example of a prompt sentence for the generative AI model, "Please recognize the faces of customers in the video obtained from the surveillance camera, estimate their age and gender, determine repeat customers, and also analyze the movement paths of customers and provide advice on the optimal placement of products." can be cited.

[0288] The flow of the specific process in Application Example 1 will be described using FIG. 12.

[0289] Step 1:

[0290] The server receives video data from the surveillance device in real time. This video data records the actions of customers from entering the store to leaving the store. Specifically, the surveillance device continuously captures frame images from the cameras in the store and transmits the data to the server. The input is the video data, and the output is the video information stored in the server.

[0291] Step 2:

[0292] The server analyzes the received video data using OpenCV to detect the faces of individual customers. The input is the video data received in step 1, and the output is the coordinate information of the detected faces. Specifically, the server applies a face detection algorithm to each frame to identify the face region within the image.

[0293] Step 3:

[0294] The server extracts facial features detected using a FaceRecognitionModel and estimates age and sex. The input is the coordinate information of the face detected in step 2, and the output is the estimated age and sex information. Specifically, it applies a facial feature extraction algorithm and performs estimation using the age and sex model.

[0295] Step 4:

[0296] The server determines whether a customer is a repeat customer or a new visitor by comparing specific features with an existing information base. The input is the facial features extracted in step 3, and the output is the repeat customer determination result. Specifically, the server compares the features with existing data in the information base, calculates the degree of match, and makes the determination.

[0297] Step 5:

[0298] The server aggregates visitor attributes based on the collected data and generates a report. The input is all output data from steps 1 through 4, and the output is the aggregated report. Specifically, it applies a data analysis algorithm to create a report document in a specific format.

[0299] Step 6:

[0300] The server analyzes customer movement data using an ActionDetectionModel. The input is the video data received in step 1, and the output is the movement data. Specifically, it tracks the customer's position in each frame and stores the path information.

[0301] Step 7:

[0302] The server analyzes the flow line data with the SpatialAnalysisModel and calculates the appropriate arrangement of products. The input is the flow line data analyzed in Step 6, and the output is a specific plan for the optimal product arrangement. As a specific operation, an optimization algorithm is applied based on the flow line pattern and customer interest.

[0303] Step 8:

[0304] Based on the calculated arrangement, the server notifies the advice to the facility operator. The input is the arrangement plan output in Step 7, and the output is the content of the notified advice. As a specific operation, recommendations for arrangement changes are sent to the operator through the notification system.

[0305] Step 9:

[0306] The server obtains data on products that customers were interested in but did not purchase from the collected image data. The input is the video data received in Step 1, and the output is the data on interested products. As a specific operation, the hand - taking and putting - back operations of products are analyzed and the data is extracted.

[0307] Step 10:

[0308] The server evaluates the degree of interest and analyzes it as marketing data. The input is the data on interested products obtained in Step 9, and the output is the data on the evaluation of the degree of interest. As a specific operation, a degree - of - interest score is calculated, and the results are analyzed and sorted as marketing data.

[0309] Step 11:

[0310] The server sorts the visitor data for each time period and analyzes the customer behavior patterns. The input is the video data received in Step 1 and all the analysis results, and the output is the result of the analysis of the behavior patterns. As a specific operation, it is segmented by time period and the customer inflow pattern is analyzed.

[0311] Step 12:

[0312] The server calculates areas for operational improvement for each time period and notifies the operations staff. The input is the behavioral pattern data analyzed in step 11, and the output is the operational improvement suggestions that are notified. Specifically, it calculates a particular improvement action based on the analysis results and notifies the person in charge.

[0313] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0314] This invention is a system that analyzes video data acquired from a monitoring device, detects individual people, performs facial recognition and attribute estimation, and analyzes customer behavior based on the results. Furthermore, by combining it with an emotion engine, it also analyzes visitor emotional data to evaluate customer satisfaction and emotional responses to products of interest. A specific embodiment of this system is described below.

[0315] Explain the program's processing in natural language.

[0316] 1. Automatic counting of customer visitors

[0317] Surveillance devices are installed at the store entrance and inside the store to capture video of visitors in real time. The video data is sent to a server, which uses an AI model to detect people in the video. It performs facial recognition on each person and estimates their age and gender. This information is temporarily stored in a database to determine if the person is a repeat customer. Specifically, the server compares the facial features with an existing database, and if there is a match, the person is registered as a repeat customer. If there is no match, the person is registered as a new visitor in the database. Finally, visitor attribute data is compiled, and daily and monthly reports are generated. These reports are notified to the store operations manager. In addition, the server uses an emotion engine to analyze the facial expressions of people in the video data and recognize the customer's emotions. The recognized emotion data is also stored as attribute data.

[0318] Specific example

[0319] server

[0320] The server receives the video data and uses an AI model to detect people.

[0321] The server performs facial recognition and estimates age and gender. For example, it might save the information in the database as "male in his 30s."

[0322] The server matches the facial features against the database, counts the customer as a "repeat customer," and updates the database.

[0323] The server uses an emotion engine to recognize emotions such as "smile" or "confusion" and store them in a database.

[0324] 2. Analysis of in-store movement and layout advice

[0325] Multiple monitoring devices within the store capture customer movement data in real time and transmit it to a server. The server analyzes this data to track customer movements. It correlates the movement data with product placement data to identify which products are attracting the most interest. Using an AI model, the server calculates the optimal product placement based on the movement data and notifies store managers with specific advice. It also integrates emotional data with the movement data to analyze customers' emotional responses to products and store layouts.

[0326] Specific example

[0327] server

[0328] The server analyzes customer movement patterns and determines that "Customer A has shown interest in product X."

[0329] Based on the customer flow data, the server advises and notifies the user to place product Y near the entrance.

[0330] The server integrates emotional data and evaluates, for example, "Customer A was pleased when they saw product X."

[0331] 3. Analysis of interest in products that were not purchased.

[0332] Data on products that customers picked up but did not purchase is extracted from the video and sent to a server. The server analyzes this data to evaluate the customer's level of interest. Specifically, it correlates the customer's movement with the data of the target products and calculates an interest score. This data is notified to marketing personnel and used for sales promotion activities. In addition, an emotion engine is used to evaluate the emotional response of customers to products they picked up but did not purchase.

[0333] Specific example

[0334] server

[0335] The server identifies product B, which the user picked up but did not purchase, and rates its level of interest as "high".

[0336] The server generates a marketing report based on the evaluation data and notifies the client of the marketing measures that should be adopted.

[0337] The server saves the emotional data to the database, indicating "interested but undecided."

[0338] 4. Observation of customer behavior by time of day and operational advice

[0339] Monitoring devices capture video data of visitors at specific time intervals and transmit it to a server. The server organizes this data by time period and analyzes customer behavior patterns. Based on customer inflow and outflow, product attention, and sales data for specific time periods, an AI model calculates areas for improvement in store operations. This advice is notified and implemented by store managers. Sentimental data for each time period is also analyzed to evaluate customer satisfaction for each time slot.

[0340] Specific example

[0341] server

[0342] The server analyzes customer data by time of day and identifies that "customer traffic increases around 2 PM."

[0343] Based on the analysis results, the server will notify you of the advice to "deploy additional staff by 2 PM."

[0344] The server analyzes sentiment data for each time period and evaluates that "customer satisfaction is high at 2 PM."

[0345] The above describes the operation and specific examples of the system in an embodiment for carrying out the present invention. This makes it possible to efficiently analyze customer behavior data and emotional data, thereby optimizing store operations and improving customer satisfaction.

[0346] The following describes the processing flow.

[0347] Program processing flow

[0348] Automatic customer visitor counting and emotion recognition

[0349] Step 1:

[0350] The terminal acquires video data from the monitoring device and transmits it to the server in real time.

[0351] Step 2:

[0352] The server analyzes the received video data using an AI model to detect people in the video.

[0353] Step 3:

[0354] The server performs facial recognition on detected individuals and estimates their age and gender.

[0355] Step 4:

[0356] The server temporarily stores the estimated facial data and compares it with an existing database based on specific features.

[0357] Step 5:

[0358] The server identifies repeat customers and new visitors based on the results of matching them with the database, and updates and registers their respective data.

[0359] Step 6:

[0360] The server uses an emotion engine to analyze the facial expressions of people in the video data and recognize their emotions.

[0361] Step 7:

[0362] The server saves the recognized emotion data as attribute data.

[0363] Step 8:

[0364] The server aggregates visitor attribute data, generates daily and monthly reports, and notifies store operations staff.

[0365] Analysis of in-store movement and layout advice

[0366] Step 1:

[0367] The terminal acquires video data from multiple monitoring devices within the store and transmits it to the server in real time.

[0368] Step 2:

[0369] The server analyzes the received video data using an AI model to track customer movement.

[0370] Step 3:

[0371] The server correlates customer movement data with product placement data to identify which products customers showed interest in.

[0372] Step 4:

[0373] The server calculates the optimal placement of products based on customer movement data.

[0374] Step 5:

[0375] Based on the calculated server placement, the server generates specific advice and notifies the store operations manager.

[0376] Step 6:

[0377] The server integrates emotional data with movement data to analyze customers' emotional responses to products of interest and store layouts.

[0378] Analysis of interest in products that were not purchased.

[0379] Step 1:

[0380] The terminal acquires video data from the monitoring device and transmits it to the server in real time.

[0381] Step 2:

[0382] The server analyzes the video data to identify products that customers picked up but did not purchase.

[0383] Step 3:

[0384] The server tracks customer movement and evaluates their level of interest in products they did not purchase.

[0385] Step 4:

[0386] The server stores interest data in a database and organizes it as marketing data.

[0387] Step 5:

[0388] The server uses an emotion engine to evaluate the emotional response of customers to products they picked up but did not purchase.

[0389] Step 6:

[0390] The server generates marketing reports and notifies marketing personnel.

[0391] Observation of customer behavior by time of day and operational advice

[0392] Step 1:

[0393] The terminal captures video data from the monitoring device, categorized by time period, and transmits it to the server in real time.

[0394] Step 2:

[0395] The server organizes the video data received by time period and compiles data on customer inflow and outflow.

[0396] Step 3:

[0397] The server analyzes data for each time period to identify customer behavior patterns.

[0398] Step 4:

[0399] Based on the data analyzed by the server, an AI model calculates areas for improvement in operations.

[0400] Step 5:

[0401] The server uses an emotion engine to analyze emotional data for each time period and evaluate customer satisfaction.

[0402] Step 6:

[0403] Based on the areas for improvement calculated by the server, it generates specific advice and notifies the store operations manager.

[0404] (Example 2)

[0405] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0406] Traditional store management systems had limitations in analyzing customer behavior and collecting attribute data, making it difficult to identify repeat customers or analyze their sentiment. Furthermore, data analysis for optimizing in-store product placement and providing appropriate marketing for customer interests was insufficient. This made optimizing store operations and improving customer satisfaction difficult.

[0407] The identification processing by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for analyzing video data acquired from a monitoring device to detect individual people, means for recognizing the faces of the detected people and estimating their age and gender, means for identifying repeat customers by comparing them with an existing database based on specific features, means for analyzing the facial expressions of people in the video data using an emotion engine and recognizing emotion data, means for aggregating visitor attributes and generating reports based on the collected data, means for providing improvement advice to each store, means for analyzing customer movement data within the store and calculating the appropriate placement of products, means for notifying store managers of advice based on the calculated placement, means for collecting data on products that customers showed interest in but did not purchase, and means for evaluating the level of interest and analyzing it as marketing data. This enables efficient analysis of customer behavior data and emotion data, optimizing store operations and improving customer satisfaction.

[0408] A "surveillance device" is a device installed at the entrance or inside a store to acquire video data in real time.

[0409] "Video data" refers to digital data acquired from surveillance devices that records the movements and actions of customers inside and outside the store.

[0410] "Person detection" is the process of identifying each individual within video data and confirming their presence.

[0411] "Facial recognition" is a technology that analyzes the faces of detected individuals and identifies the unique characteristics of each person.

[0412] "Age estimation" is the process of calculating a person's age based on data obtained through facial recognition.

[0413] "Gender estimation" is the process of determining a person's gender based on data obtained through facial recognition.

[0414] "Specific features" refer to unique facial feature data extracted through facial recognition to identify individual people.

[0415] "Existing database" refers to a database where past visitor data is stored.

[0416] A "repeat customer" is a customer who has visited the store before and is returning to the store.

[0417] An "emotion engine" is software that analyzes a person's facial expressions in video data and recognizes emotional data.

[0418] "Attribute data" is a general term for various types of customer-related data, such as age, gender, visit history, and emotions.

[0419] "Report generation" is the process of creating a report that includes statistics and analysis results based on collected data.

[0420] "Improvement advice" means suggesting areas for improvement in store operations based on the results of data analysis.

[0421] "Customer flow data" refers to data that records the routes and movements of customers as they move within a store.

[0422] "Appropriate product placement" refers to a placement method that optimizes the location of products based on customer movement data.

[0423] "Interest level assessment" is a process of measuring the degree of interest a customer has in a particular product.

[0424] "Marketing data" refers to sales promotion-related data analyzed based on customer behavior and interests.

[0425] "Analysis tools" refer to software and algorithms used to analyze collected data and extract useful information.

[0426] This invention is a system that analyzes customer behavior, performs facial recognition, estimates attributes, and analyzes emotions in store operations, and uses this information to support improvements in store operations. The following describes how this system is specifically implemented.

[0427] Automatic counting of customer visitors

[0428] terminal

[0429] The terminal uses surveillance equipment to capture real-time images of the store's entrance and interior, acquiring video data. This terminal includes a camera and video processing hardware.

[0430] server

[0431] The server inputs the acquired video data into the AI ​​model "PersonDetect_v2" to detect people in the video.

[0432] The server uses "FaceRecog_v3" to perform facial recognition on detected individuals and estimate their age and gender.

[0433] The server compares facial features with the database. If there is a match, the data is updated to indicate a repeat customer; otherwise, the person is registered in the database as a new visitor.

[0434] The server uses the emotion engine "EmotionEngine_v1" to analyze the facial expressions of people in the video, recognize emotion data, and save it to a database.

[0435] Based on the collected data, visitor attributes are compiled, and daily and monthly reports are generated. These reports are stored in a database and notifications are sent.

[0436] Analysis of in-store movement and layout advice

[0437] terminal

[0438] The terminal uses multiple monitoring devices to capture real-time data on customer movement within the store and transmits it to the server.

[0439] server

[0440] The server analyzes the received movement data using the AI ​​model "MovTrack_v2" to track customer movements.

[0441] The server correlates customer movement data with product placement data to identify products of high interest.

[0442] The server uses the AI ​​model "RecommendConfig_v1" to calculate the optimal placement of products and notifies store managers with specific placement advice.

[0443] Analysis of interest in products that were not purchased.

[0444] terminal

[0445] The terminal extracts data from the video of products that customers picked up but did not purchase, and sends it to the server.

[0446] server

[0447] The server analyzes the video data to identify products that were not purchased.

[0448] The server associates user flow data with data on the target product and calculates an interest score using the "InterestRate_v1" model.

[0449] The server generates a marketing report based on the evaluation data and notifies the marketing team.

[0450] Observation of customer behavior by time of day and operational advice

[0451] terminal

[0452] The terminal captures video data of visitors at set intervals and sends it to the server.

[0453] server

[0454] The server organizes video data by time period and uses "TimeSlotAnalyzer_v1" to analyze customer behavior patterns.

[0455] The server uses data on visitor inflow and outflow, product popularity, and sales figures for specific time periods to generate an AI model that calculates areas for improvement in store operations.

[0456] The server notifies the store operations manager of advice based on the analysis results.

[0457] Specific example

[0458] Example of a prompt

[0459] The following are specific examples of each process.

[0460] Automatic counting of customer visitors

[0461] "Please generate an AI model program that estimates a customer's age and gender and determines whether they are a repeat customer or a new customer."

[0462] Analysis of in-store movement and layout advice

[0463] "Please generate a program for a system that analyzes customer movement data and product placement data within a store to propose the optimal product layout."

[0464] Analysis of interest in products that were not purchased.

[0465] "Please create a program that analyzes data on products customers picked up but did not purchase, and evaluates their level of interest."

[0466] Observation of customer behavior by time of day and operational advice

[0467] "Please generate a program for a system that observes customer behavior at different times of the day and provides advice for store operations."

[0468] As described above, the system of the present invention efficiently analyzes customer behavior data and emotional data to optimize store operations and improve customer satisfaction.

[0469] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0470] Automatic counting of customer visitors

[0471] Step 1: Acquire video data

[0472] terminal

[0473] Using surveillance equipment, the store entrance and interior are photographed in real time, and video data is acquired.

[0474] Input: Video of the store entrance and interior.

[0475] Output: Video data

[0476] Send the video data to the server.

[0477] Step 2: Receiving video data and detecting people

[0478] server

[0479] The video data received at the time of posting (10 minutes ago) is input into the AI ​​model "PersonDetect_v2" to detect people in the video.

[0480] Input: Video data

[0481] Output: Detected person data

[0482] The detected person data is temporarily stored in memory.

[0483] Step 3: Face Recognition and Attribute Estimation

[0484] server

[0485] Using the detected person data, the face recognition algorithm "FaceRecog_v3" is executed to identify each person's face.

[0486] Input: Detected person data

[0487] Output: Face recognition data

[0488] Age and gender are estimated based on the results of facial recognition.

[0489] Input: Face recognition data

[0490] Output: Age and gender data

[0491] Step 4: Matching with the database

[0492] server

[0493] The estimated facial features are compared with data from an existing database.

[0494] Input: Age and gender data, facial feature data

[0495] Output: Determination result for whether the visitor is a repeat customer or a new visitor.

[0496] If a match is found, the person is recognized as a repeat customer and the database is updated. If there is no match, the person is registered in the database as a new visitor.

[0497] Step 5: Sentiment analysis and data storage

[0498] server

[0499] The emotion engine "EmotionEngine_v1" is used to analyze the facial expressions of people in video data and recognize emotional data.

[0500] Input: Face recognition data

[0501] Output: Sentiment data

[0502] The recognized emotion data is stored in the database as attribute data.

[0503] Based on the collected data, visitor attributes are compiled, and daily and monthly reports are generated.

[0504] Input: Attribute data

[0505] Output: Daily and monthly reports

[0506] Analysis of in-store movement and layout advice

[0507] Step 1: Obtaining movement data

[0508] terminal

[0509] Multiple monitoring devices are used to capture real-time data on customer movement within the store and transmit it to a server.

[0510] Input: Video of customer movement within the store

[0511] Output: Movement data

[0512] Step 2: Receiving and analyzing movement data

[0513] server

[0514] The received movement data is analyzed using the AI ​​model "MovTrack_v2" to track customer movements.

[0515] Input: Movement data

[0516] Output: Tracking data

[0517] Step 3: Identifying the level of interest in the product

[0518] server

[0519] By linking customer traffic data with product placement data, we can identify products of high interest.

[0520] Input: Tracking data, product placement data

[0521] Output: List of products of high interest

[0522] Step 4: Calculation and notification of the optimal placement

[0523] server

[0524] The optimal placement of products is calculated using the AI ​​model "RecommendConfig_v1".

[0525] Input: List of products of high interest

[0526] Output: Optimal product placement plan

[0527] The calculated layout plan will be provided to the store operations manager as advice.

[0528] Analysis of interest in products that were not purchased.

[0529] Step 1: Data acquisition for the product picked up

[0530] terminal

[0531] The system captures video data of products that customers picked up but did not purchase, and sends it to a server.

[0532] Input: Customer behavior video

[0533] Output: Data of the product picked up

[0534] Step 2: Receiving and analyzing product data

[0535] server

[0536] The received video data is analyzed to identify products that were not purchased.

[0537] Input: Data of the product picked up

[0538] Output: List of items that were not purchased

[0539] Step 3: Evaluate and notify of interest level

[0540] server

[0541] The system correlates customer movement data with data on unpurchased items and calculates an interest score using "InterestRate_v1".

[0542] Input: List of unpurchased items, customer flow data

[0543] Output: Interest score

[0544] A marketing report is generated based on the interest score, and the marketing team is notified.

[0545] Observation of customer behavior by time of day and operational advice

[0546] Step 1: Obtain customer data by time of day

[0547] terminal

[0548] Video data of visitors is captured at each time slot and sent to the server.

[0549] Input: Video footage by time period

[0550] Output: Customer data by time period

[0551] Step 2: Data analysis by time period

[0552] server

[0553] The video data received for each time period is organized, and customer behavior patterns are analyzed using the "TimeSlotAnalyzer_v1" model.

[0554] Input: Customer data by time period

[0555] Output: Behavioral patterns by time of day

[0556] Step 3: Calculation and notification of areas for operational improvement

[0557] server

[0558] Based on visitor inflow and outflow during specific time periods, product popularity, and sales data, an AI model calculates areas for improvement in store operations.

[0559] Input: Behavioral patterns by time of day, attention level data, sales data

[0560] Output: Points for improvement in operations

[0561] Notify the store operations manager of the areas for improvement.

[0562] The above outlines the processing steps and specific operations of this system's program. This enables efficient analysis of customer behavior and emotional data, leading to optimized store operations and improved customer satisfaction.

[0563] (Application Example 2)

[0564] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0565] The problem that this invention aims to solve is to efficiently monitor customer behavior and emotions in physical stores and improve customer satisfaction. By combining real-time customer behavior data and emotional data, the aim is to provide more appropriate services, improve store operations, and ultimately increase sales.

[0566] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0567] In this invention, the server includes means for analyzing video data acquired from a monitoring device to detect individual people, means for recognizing the faces of detected people and estimating their age and gender, means for identifying repeat customers by comparing them with an existing database based on specific characteristics, means for aggregating visitor attributes based on collected data and generating reports, means for providing improvement advice to each store, means for analyzing customer emotion data and evaluating customer satisfaction, means for analyzing customer movement data within the store and calculating the optimal placement of products, means for collecting data on products that customers showed interest in but did not purchase and evaluating their level of interest, and means for analyzing customer behavior data by time of day and suggesting areas for improvement in store operations. This makes it possible to analyze customer behavior and emotions in real time and immediately implement measures to improve store operations.

[0568] A "surveillance device" refers to devices such as cameras and sensors that are installed to acquire video data.

[0569] "Video data" refers to data containing visual information acquired by surveillance equipment.

[0570] "Person detection means" refers to algorithms and technologies for identifying individual people from video data.

[0571] "Facial recognition means" refers to technology for identifying and distinguishing the faces of individuals detected from video data.

[0572] "Age estimation methods" refer to technologies that estimate the age of a person whose face has been recognized.

[0573] "Gender estimation means" refers to technology that estimates the gender of a person whose face has been recognized.

[0574] "Specific characteristics" refer to unique attributes such as facial features or behavioral patterns.

[0575] A "database" is an information system that stores information such as that of repeat customers.

[0576] A "repeat customer identification method" is a technology that determines whether a customer is a repeat customer by comparing them with an existing database based on specific characteristics.

[0577] "Visitor attributes" refer to characteristics such as the age, gender, and emotions of visitors.

[0578] A "report generation method" is a technology that compiles analysis results in report format based on collected data.

[0579] A "method for providing improvement advice" is a technology that notifies store managers of areas for improvement based on data analysis results.

[0580] "Emotional data" refers to data that represents the emotions estimated from the facial expressions of customers.

[0581] A "customer satisfaction evaluation method" is a technology that evaluates customer satisfaction based on emotional data.

[0582] "Customer movement data" refers to data that represents the paths customers take within a store.

[0583] The "optimal placement calculation method" is a technology that calculates the optimal placement of products based on traffic flow data.

[0584] "Products of interest" are products that customers showed interest in but did not purchase.

[0585] An "interest level evaluation method" is a technology that evaluates how interested a customer is in a product they are interested in.

[0586] "Customer behavior data by time of day" refers to data that represents customer behavior patterns during specific time periods.

[0587] "Store operation improvement methods" refer to technologies that analyze customer behavior data for each time period and identify areas for improvement in store operations.

[0588] The embodiments for carrying out this invention are described in detail below. This system analyzes video data acquired from a monitoring device to detect individual people, perform facial recognition, estimate attributes, and analyze emotion data, thereby understanding customer behavior and emotions in physical stores in real time, and aiming to optimize store operations and improve customer satisfaction.

[0589] System Configuration

[0590] This system consists of monitoring devices, a cloud server, and smart glasses or smartphones. The monitoring devices include standard surveillance cameras and sensors, while the cloud server implements AI models and emotion engines. The smart glasses and smartphones are terminals for store employees to receive and respond to information in real time.

[0591] Operation flow

[0592] 1. Automatic counting of customer visitors

[0593] server:

[0594] The server receives video data transmitted from the monitoring device and uses an AI model to detect people in the video. It recognizes the faces of the detected people and estimates their age and gender. For example, it might store them in the database as "a man in his 30s." Based on the facial features, it compares them with existing databases to determine whether they are repeat customers or new visitors. An emotion engine is used to analyze facial expressions and add emotional data such as "smiling" or "confused."

[0595] 2. Analysis of in-store movement and layout advice

[0596] server:

[0597] The system receives video data transmitted from multiple monitoring devices within the store and analyzes customer movement data in real time. Based on this, it correlates it with product placement data to identify which products are attracting the most attention. The server uses an AI model to calculate the optimal product placement based on the movement data and notifies store managers with specific advice. In addition, emotional data is integrated with the movement data to analyze customers' emotional reactions to products of interest and to the sales floor.

[0598] 3. Analysis of interest in products that were not purchased.

[0599] server:

[0600] Data on products that customers picked up but did not purchase is extracted from the video and sent to a server. The server analyzes this data to evaluate the customer's level of interest. It correlates the movement data with the data on the target products and calculates an interest score. This data is notified to marketing personnel and used for sales promotion activities. In addition, an emotion engine is used to evaluate the emotional response of customers to products they picked up but did not purchase.

[0601] 4. Observation of customer behavior by time of day and operational advice

[0602] server:

[0603] Monitoring devices capture video data of visitors at specific time intervals and transmit it to a server. The server organizes this data by time period and analyzes customer behavior patterns. Based on customer inflow and outflow, product attention, and sales data for specific time periods, an AI model calculates areas for improvement in store operations. This advice is notified and implemented by store managers. Sentimental data for each time period is also analyzed to evaluate customer satisfaction for each time slot.

[0604] Examples and prompts for generative AI models

[0605] For example, if customers tend to gather in a particular product section around 2 PM, and many of them smile while looking at the products but do not make a purchase, the system will determine that "they are highly interested in this product but are hesitant," and will either place related products nearby or notify staff to follow up.

[0606] Example of a prompt

[0607] Prompt example:

[0608] "Around 2 PM, customers tend to pick up product X but don't end up purchasing it. Customer sentiment data indicates they are interested but hesitant. Please propose effective marketing strategies to address this situation."

[0609] This system enables store managers to respond appropriately in real time, leading to improved customer satisfaction and increased sales.

[0610] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0611] Step 1:

[0612] The surveillance system captures real-time video data of the store's entrance and interior.

[0613] Input: Video data acquired from a monitoring device.

[0614] Output: Sends a stream of video data to the cloud server.

[0615] Operation: The surveillance device's camera captures video data and sends it to a cloud server via the internet.

[0616] Step 2:

[0617] The server receives video data and uses an AI model to detect people in the video.

[0618] Input: Video data sent in Step 1.

[0619] Output: Location information of people detected in the video.

[0620] Operation: The AI ​​model analyzes the video data and executes an algorithm to identify the contours and locations of people.

[0621] Step 3:

[0622] The server recognizes the face of the detected person and estimates their age and gender.

[0623] Input: Location information of the person detected in Step 2.

[0624] Output: Recognized facial feature data, estimated age and sex data.

[0625] Operation: A facial recognition algorithm identifies a person's face and performs age and gender estimation.

[0626] Step 4:

[0627] The server matches facial features against an existing database to determine whether the person is a repeat customer or a new visitor.

[0628] Input: Facial feature data obtained in Step 3.

[0629] Output: Determination result of whether the visitor is a repeat customer or a new visitor.

[0630] Operation: Executes an algorithm to match feature data against a database and check if any matching data exists.

[0631] Step 5:

[0632] The server uses an emotion engine to analyze facial expressions and extract emotional data.

[0633] Input: Video data of the face recognized in Step 3.

[0634] Output: Extracted emotion data (e.g., smile, confusion, etc.).

[0635] Operation: The emotion engine analyzes facial features and classifies the emotional state.

[0636] Step 6:

[0637] The server collects visitor attributes and generates a report.

[0638] Input: Visitor data and sentiment data obtained in Steps 4 and 5.

[0639] Output: Summary report of visitor attributes.

[0640] Operation: Aggregates individual data points stored in the database and automatically generates daily and monthly reports.

[0641] Step 7:

[0642] A server receives video data transmitted from multiple monitoring devices within the store and analyzes customer movement data in real time.

[0643] Input: Video data stream from the monitoring device.

[0644] Output: Customer movement data.

[0645] Operation: Video data obtained from multiple cameras is analyzed using a tracking algorithm to identify customer movement patterns.

[0646] Step 8:

[0647] The server calculates the optimal placement of products based on customer movement data.

[0648] Input: Movement data obtained in Step 7.

[0649] Output: Optimal placement plan for products.

[0650] Operation: Integrates customer flow data with product placement data and calculates the optimal product placement using an AI model.

[0651] Step 9:

[0652] The server collects data on products that were not purchased and evaluates the level of interest.

[0653] Input: Video data and movement data from monitoring devices.

[0654] Output: Interest score for products that were not purchased.

[0655] Operation: The system determines whether a user picked up a specific product from the video, and if they did not purchase it, it analyzes their behavior and emotional data to evaluate their level of interest.

[0656] Step 10:

[0657] The server analyzes customer behavior data for each time period and suggests areas for improvement in store operations.

[0658] Inputs: Video data by time of day, sales data, customer flow data, and sentiment data.

[0659] Output: Customer behavior report by time of day and operational improvement advice.

[0660] Function: It organizes and analyzes data for each time period, evaluates customer behavior patterns and emotions during specific time periods, and generates operational improvement advice based on that.

[0661] Through the above processing steps, it is possible to analyze customer behavior and emotions in physical stores in detail and achieve optimal store operations in real time.

[0662] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0663] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include those described above. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions shown by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0664] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0665] [Second Embodiment]

[0666] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0667] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0668] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0669] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0670] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0671] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0672] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0673] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0674] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0675] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0676] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0677] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0678] This invention relates to a system that analyzes video data acquired from a surveillance device, detects individual people, performs facial recognition and attribute estimation, and analyzes customer behavior based on the results. The program and processing of this system are described in detail below.

[0679] Explain the program's processing in natural language.

[0680] 1. Automatic counting of customer visitors

[0681] Surveillance devices are installed at the store entrance and inside the store to capture video of visitors in real time. The video data is sent to a server, which uses an AI model to detect people in the video. It performs facial recognition on each person and estimates their age and gender. This information is temporarily stored in a database to determine if a customer is a repeat customer. Specifically, the server compares the facial features with an existing database, and if there is a match, the person is registered as a repeat customer. If there is no match, the person is registered as a new visitor in the database. Finally, visitor attribute data is compiled, and daily and monthly reports are generated. These reports are notified to the store operations manager.

[0682] Specific example

[0683] server

[0684] The server receives the video data and uses an AI model to detect people.

[0685] The server performs facial recognition and estimates age and gender. For example, it might save the information in the database as "male in his 30s."

[0686] The server matches the facial features against the database, counts the customer as a "repeat customer," and updates the database.

[0687] 2. Analysis of in-store movement and layout advice

[0688] Multiple monitoring devices within the store capture customer movement data in real time and transmit it to a server. The server analyzes this data to track customer movements. It correlates the movement data with product placement data to identify which products are attracting the most interest. Using an AI model, the server calculates the optimal product placement based on the movement data and notifies store managers with specific advice.

[0689] Specific example

[0690] server

[0691] The server analyzes customer movement patterns and determines that "Customer A has shown interest in product X."

[0692] Based on the customer flow data, the server advises and notifies the user to place product Y near the entrance.

[0693] 3. Analysis of interest in products that were not purchased.

[0694] Data on products that customers picked up but did not purchase is extracted from the video and sent to a server. The server analyzes this data to evaluate the customer's level of interest. Specifically, it correlates the customer's movement with the data on the target products and calculates an interest score. This data is then notified to marketing personnel and used for sales promotion activities.

[0695] Specific example

[0696] server

[0697] The server identifies product B, which the user picked up but did not purchase, and rates its level of interest as "high".

[0698] The server generates a marketing report based on the evaluation data and notifies the client of the marketing measures that should be adopted.

[0699] 4. Observation of customer behavior by time of day and operational advice

[0700] Monitoring devices capture video data of visitors at specific time intervals and transmit it to a server. The server organizes this data by time period and analyzes customer behavior patterns. Based on customer inflow and outflow, product attention levels, and sales data for specific time periods, an AI model calculates areas for improvement in store operations. This advice is communicated to store managers for their implementation.

[0701] Specific example

[0702] server

[0703] The server analyzes customer data by time of day and identifies that "customer traffic increases around 2 PM."

[0704] Based on the analysis results, the server will notify you of the advice to "deploy additional staff by 2 PM."

[0705] The above describes the operation and specific examples of the system in an embodiment for carrying out the present invention. This makes it possible to efficiently analyze customer behavior data and optimize store operations.

[0706] The following describes the processing flow.

[0707] Program processing flow

[0708] Automatic counting of customer visitors

[0709] Step 1:

[0710] The terminal acquires video data from the monitoring device and transmits it to the server in real time.

[0711] Step 2:

[0712] The server analyzes the received video data using an AI model to detect people in the video.

[0713] Step 3:

[0714] The server performs facial recognition on detected individuals and estimates their age and gender.

[0715] Step 4:

[0716] The server temporarily stores the estimated facial data and compares it with an existing database based on specific features.

[0717] Step 5:

[0718] The server identifies repeat customers and new visitors based on the results of matching them with the database, and updates and registers their respective data.

[0719] Step 6:

[0720] The server aggregates visitor attribute data, generates daily and monthly reports, and notifies store operations staff.

[0721] Analysis of in-store movement and layout advice

[0722] Step 1:

[0723] The terminal acquires video data from multiple monitoring devices within the store and transmits it to the server in real time.

[0724] Step 2:

[0725] The server analyzes the received video data using an AI model to track customer movement.

[0726] Step 3:

[0727] The server correlates customer movement data with product placement data to identify which products customers showed interest in.

[0728] Step 4:

[0729] The server calculates the optimal placement of products based on customer movement data.

[0730] Step 5:

[0731] Based on the calculated server placement, the server generates specific advice and notifies the store operations manager.

[0732] Analysis of interest in products that were not purchased.

[0733] Step 1:

[0734] The terminal acquires video data from the monitoring device and transmits it to the server in real time.

[0735] Step 2:

[0736] The server analyzes the video data to identify products that customers picked up but did not purchase.

[0737] Step 3:

[0738] The server tracks customer movement and evaluates their level of interest in products they did not purchase.

[0739] Step 4:

[0740] The server stores interest data in a database and organizes it as marketing data.

[0741] Step 5:

[0742] The server generates marketing reports and notifies marketing personnel.

[0743] Observation of customer behavior by time of day and operational advice

[0744] Step 1:

[0745] The terminal captures video data from the monitoring device, categorized by time period, and transmits it to the server in real time.

[0746] Step 2:

[0747] The server organizes the video data received by time period and compiles data on customer inflow and outflow.

[0748] Step 3:

[0749] The server analyzes data for each time period to identify customer behavior patterns.

[0750] Step 4:

[0751] Based on the data analyzed by the server, an AI model calculates areas for improvement in operations.

[0752] Step 5:

[0753] Based on the areas for improvement calculated by the server, it generates specific advice and notifies the store operations manager.

[0754] (Example 1)

[0755] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0756] Traditional systems made it difficult to effectively collect and analyze customer behavior data from store visitors and propose improvements to store operations. Furthermore, strategic marketing measures based on individual customer attributes and interests were not adequately implemented. In particular, identifying repeat customers, optimizing product placement, and evaluating interest in products that were not purchased were challenging, resulting in a lack of concrete, data-driven advice for efficient store operations and increased sales.

[0757] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0758] In this invention, the server includes means for analyzing video data acquired from a monitoring device to detect individual people, means for recognizing the faces of detected people and estimating their age and gender, means for identifying repeat customers by comparing them with an existing database based on characteristic information, means for aggregating visitor attributes based on collected data and generating reports, means for providing improvement advice to each store, means for analyzing customer movement data to calculate the optimal placement of products and notifying each store, means for collecting data on products that customers picked up but did not purchase and evaluating their level of interest, means for acquiring video data from installed monitoring devices at different times of the day and analyzing customer behavior patterns, and means for calculating and notifying improvements to store operations by correlating the analysis results with sales data. This makes it possible to efficiently collect and analyze customer behavior data and optimize store operations.

[0759] A "surveillance device" is a device installed at the entrance or inside a store to acquire video data.

[0760] "Video data" refers to image and video information acquired by surveillance equipment.

[0761] "Analysis" refers to techniques that process information such as video data to detect individual people or analyze movement patterns.

[0762] "Person detection" is a method of identifying individual people from video data.

[0763] "Facial recognition" is a method of identifying a specific person by analyzing the face of a detected individual.

[0764] "Age and gender estimation" refers to a method of inferring a person's age group and gender based on person detection and facial recognition.

[0765] "Characteristic information" refers to information used to identify individual people, such as facial features and movement data.

[0766] A "database" is a system for organizing and storing collected data.

[0767] A "repeat customer" refers to a person who has visited the store before and is recorded in the existing database.

[0768] "Visitor attributes" refer to characteristics such as visitor age, gender, movement patterns, and purchase history.

[0769] "Report generation" refers to the process of compiling analysis results based on collected data and creating a report.

[0770] "Improvement advice" is a method of providing instructions on areas for improvement in store operations based on collected and analyzed data.

[0771] "Customer movement data" refers to data that shows the routes and behavioral patterns of customers as they move within a store.

[0772] "Product placement" refers to the location and method of displaying products within a store.

[0773] "Interest level" is a measure that evaluates the degree of interest a customer has in a product they picked up but did not purchase.

[0774] "Video data by time period" refers to video data acquired during a specific time period.

[0775] "Behavioral patterns" refer to data that shows the influx and outflow of customers and the trends of popular products during specific time periods.

[0776] This invention relates to a system that analyzes video data acquired from a surveillance device, detects individual people, performs facial recognition and attribute estimation, and analyzes customer behavior based on the results. The program and processing of this system are described below.

[0777] This system consists of monitoring devices, a server, and terminals. The monitoring devices are installed at the entrance and inside the store and acquire customer video data in real time. The acquired video data is transmitted to the server via the terminals.

[0778] The server uses an AI model to analyze the received video data. The AI ​​model utilizes generative AI technology to detect people in the video with high accuracy. At this stage, the server uses a facial recognition algorithm to identify the faces of detected individuals and estimate their age and gender. For example, it might store this information in a database as "a man in his 30s." The software used for this includes common facial recognition libraries and attribute estimation algorithms.

[0779] Furthermore, the server matches facial features against an existing database, counting repeat customers if a match is found. If there is no match, the customer is registered in the database as a new visitor. Finally, visitor attribute data is aggregated, and daily and monthly reports are generated. These reports are then communicated to store operations staff.

[0780] Next, the server analyzes customer movement data transmitted from multiple monitoring devices within the store to track customer behavior. It then correlates this movement data with product placement data to identify which products are attracting the most attention. Using an AI model, the server calculates the optimal product placement based on the movement data and provides specific advice to store managers.

[0781] The server also analyzes data on products that customers picked up but did not purchase, and evaluates the customer's level of interest. Specifically, it correlates customer movement with data on the target products and calculates an interest score. This data is then shared with marketing personnel and used for sales promotion activities.

[0782] Furthermore, monitoring devices capture video data of visitors at specific time intervals and transmit it to a server. The server organizes this data by time period and analyzes customer behavior patterns. Based on customer inflow and outflow, product attention levels, and sales data for specific time periods, an AI model calculates areas for improvement in store operations. This advice is then communicated to store managers for implementation.

[0783] For example, the server might analyze that "customer traffic increased during the 2 PM timeframe" and notify the store manager of the need to allocate additional staff.

[0784] Example of a prompt

[0785] "Analyze customer movement data recorded by security cameras to identify which products are attracting the most attention. Based on these results, propose the optimal product placement and notify the store operations manager."

[0786] The above outlines a specific implementation of this system. This makes it possible to efficiently analyze customer behavior data and optimize store operations.

[0787] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0788] Step 1:

[0789] The terminal acquires video data in real time from surveillance devices installed at the entrance and inside the store.

[0790] Specific operation: The monitoring device captures video data using a camera sensor, and the terminal receives this data.

[0791] Input: Video data from a monitoring device.

[0792] Output: Video data received by the terminal.

[0793] Step 2:

[0794] The device sends the acquired video data to the server.

[0795] Specific operation: The device uploads the received video data to the server via the network.

[0796] Input: Video data received by the device.

[0797] Output: Video data received by the server.

[0798] Step 3:

[0799] The server inputs the received video data into an AI model to detect people in the video.

[0800] Specific operation: The server inputs video data into the AI ​​model, and the model detects people in each video frame.

[0801] Input: Video data received by the server.

[0802] Output: Person detection results by the AI ​​model.

[0803] Step 4:

[0804] The server performs facial recognition on detected individuals and estimates their age and gender.

[0805] Specific operation: The server uses a facial recognition algorithm to analyze the detected face and estimate its age and gender. For example, it might save it in the database as "male in his 30s."

[0806] Input: Person detection results from an AI model.

[0807] Output: Person data with estimated age and gender.

[0808] Step 5:

[0809] The server matches the facial features against an existing database. If there is a match, the person is counted as a repeat customer. If there is no match, they are registered in the database as a new visitor.

[0810] Specific operation: The server compares the identified facial feature data with the database to determine whether it is a new or repeat customer.

[0811] Input: Individual data with estimated age and gender, and existing databases.

[0812] Output: Repeat customer list, New visitor list.

[0813] Step 6:

[0814] The server aggregates visitor attribute data and generates daily and monthly reports.

[0815] Specific operation: The server aggregates data on new and repeat customers, and a report generation program creates daily and monthly reports.

[0816] Input: Repeat customer list, New visitor list.

[0817] Output: Daily and monthly reports.

[0818] Step 7:

[0819] The server notifies the store operations manager of the report.

[0820] Specific operation: The server sends the generated report to the responsible person via email or dashboard.

[0821] Input: Daily and monthly reports.

[0822] Output: Report sent to store operations staff.

[0823] Step 8:

[0824] The terminal acquires customer movement data in real time from multiple monitoring devices within the store.

[0825] Specific operation: The monitoring device tracks customer movements within the store and captures movement data in real time.

[0826] Input: Customer traffic data from monitoring devices within the store.

[0827] Output: Movement data received by the terminal.

[0828] Step 9:

[0829] The terminal sends the movement data it acquires to the server.

[0830] Specific operation: The terminal receives movement data and sends it to the server via the network.

[0831] Input: Movement data received by the terminal.

[0832] Output: Movement data received by the server.

[0833] Step 10:

[0834] The server analyzes the movement data it receives to track customer movements.

[0835] Specific operation: The server processes the movement data using an analysis algorithm to analyze customer movement patterns.

[0836] Input: Movement data received by the server.

[0837] Output: Customer movement analysis results.

[0838] Step 11:

[0839] The server correlates customer flow data with product placement data to identify which products are attracting the most attention.

[0840] Specific operation: The server matches customer flow data with product placement data to identify products of high interest.

[0841] Input: Customer movement analysis results, product placement data.

[0842] Output: A list of products the customer has shown interest in.

[0843] Step 12:

[0844] The server calculates the optimal placement of products and notifies store managers with specific advice.

[0845] Specific operation: The server uses an optimal placement algorithm to generate product placement advice and notifies the person in charge.

[0846] Input: A list of products that the customer has shown interest in.

[0847] Output: Placement advice report.

[0848] Step 13:

[0849] The terminal extracts data from in-store monitoring devices about items that customers picked up but did not purchase, and sends it to a server.

[0850] Specific operation: The monitoring device tracks customer behavior, collects data on items picked up but not purchased, and transmits it to the terminal.

[0851] Input: Data on products that were not purchased.

[0852] Output: Data on unpurchased items received by the server.

[0853] Step 14:

[0854] The server analyzes data on unpurchased items to assess the customer's level of interest.

[0855] Specific operation: The server analyzes data on unpurchased items and calculates an interest score.

[0856] Input: Data on unpurchased items received by the server.

[0857] Output: A list of products with interest scores.

[0858] Step 15:

[0859] The server notifies marketing personnel of interest levels, who then use the data for sales promotion activities.

[0860] Specific operation: The server creates a marketing report based on interest level data and sends it to the person in charge.

[0861] Input: A list of products with interest scores.

[0862] Output: Marketing report.

[0863] Step 16:

[0864] The terminal acquires video data of visitors from the monitoring device at specific time intervals and sends it to the server.

[0865] Specific operation: The device captures video data at specified time intervals and sends it to the server.

[0866] Input: Time-series video data from monitoring equipment.

[0867] Output: Video data received by the server for each time period.

[0868] Step 17:

[0869] The server organizes video data by time period and analyzes customer behavior patterns.

[0870] Specific operation: The server collects data for each time period and analyzes behavioral patterns.

[0871] Input: Video data received by the server for each time period.

[0872] Output: Customer behavior patterns by time of day.

[0873] Step 18:

[0874] The server correlates the analysis results with sales data to calculate areas for improvement in store operations.

[0875] Specific operation: The server analyzes the correlation between behavioral patterns and sales data to identify areas for improvement.

[0876] Input: Customer behavior patterns by time of day, sales data.

[0877] Output: A report identifying areas for improvement.

[0878] Step 19:

[0879] The server notifies the store operations manager with improvement advice.

[0880] Specific action: The server sends a report indicating areas for improvement to the responsible person and notifies them.

[0881] Input: A report identifying areas for improvement.

[0882] Output: Improvement advice notified to store operations staff.

[0883] (Application Example 1)

[0884] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0885] Traditional store operations have made it difficult to adequately analyze customer behavior data, particularly regarding customer movement patterns, product placement, and analysis of unpurchased items, without utilizing real-time data. Furthermore, there was a lack of means to observe customer behavior at different times of the day and to obtain concrete suggestions for operational improvements based on that observation. As a result, optimizing store operations was hindered, making it difficult to contribute to increased sales and improved customer satisfaction.

[0886] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0887] In this invention, the server includes means for analyzing image data acquired from a monitoring device to detect individual people, means for recognizing the faces of detected people and estimating their age and gender, means for identifying repeat customers by comparing them with an existing information base based on specific features, means for aggregating visitor attributes based on collected data and generating reports, means for providing improvement suggestions to each facility, means for analyzing customer movement data and calculating the appropriate placement of products, means for notifying facility operators of the suggestions based on the calculated placement, means for acquiring data on products that customers showed interest in but did not purchase from collected image data, means for evaluating the level of interest and analyzing it as marketing data, means for organizing visitor data by time of day and analyzing customer behavior patterns, and means for calculating points for operational improvement in each time period and notifying operators. This makes it possible to efficiently collect and analyze customer behavior data within stores and to make operational improvement suggestions in real time.

[0888] A "surveillance device" refers to a device that acquires video data and performs tasks such as person detection and movement analysis.

[0889] "Image data" refers to video information acquired from surveillance devices and is used for person detection and movement analysis.

[0890] "Person detection" refers to a method of identifying individual people from image data.

[0891] "Facial recognition" refers to a method of identifying a person by analyzing the facial features of a detected individual.

[0892] "Age and gender estimation" refers to the act of inferring a person's age and gender from the results of facial recognition.

[0893] "Specific features" refer to facial characteristic information obtained through facial recognition, and are data used to compare this information with existing information bases.

[0894] "Existing information base" refers to a database containing customer data collected previously.

[0895] A "repeat customer" refers to a customer who has a history of visiting the store in the past and has been determined to be a returning customer based on matching specific characteristics.

[0896] "Collected data" refers to all data obtained from monitoring devices and analysis means.

[0897] "Visitor attributes" refer to characteristics such as the age, gender, and number of visits of customers who visit the store.

[0898] "Report generation" refers to the act of compiling collected data and creating a document that summarizes it in a specific format.

[0899] "Each facility" refers to the stores or business locations that are the subject of data analysis.

[0900] An "improvement suggestion" is a proposal that outlines specific measures to improve store operations based on collected data.

[0901] "Customer movement data" refers to information that shows the paths customers take within a store.

[0902] "Appropriate product placement" refers to advice on optimizing product placement based on customer traffic data.

[0903] A "facility operations manager" refers to a manager responsible for the operation of a store or business location.

[0904] "Data acquisition" refers to the act of extracting necessary information from image data.

[0905] "Interest level assessment" is the act of analyzing and evaluating the level of interest shown by customers.

[0906] "Marketing data" refers to data about customer interests and behavior extracted from collected data, and is used in marketing strategies.

[0907] "Visitor data by time slot" refers to customer visit information organized by specific time periods.

[0908] "Customer behavior patterns" refer to pattern information that shows how customers move around within a store and what actions they take.

[0909] "Areas for operational improvement" refers to the points necessary to improve store operations based on collected data.

[0910] This invention relates to a system that analyzes image data acquired from monitoring devices to track and analyze customer behavior in real time. This system collects customer attribute data, movement data, and purchasing behavior information, contributing to the optimization of facility operations. Specific embodiments of this system are described below.

[0911] The server uses multiple AI models to process image data. For example, it uses OpenCV to acquire image data and a FaceRecognitionModel to recognize and analyze people's faces. It also uses an ActionDetectionModel to analyze customer movement and behavior. Furthermore, it uses a SpatialAnalysisModel to analyze the collected movement data and calculate the appropriate placement of products. All of this data is then centrally managed and stored by DatabaseConnection.

[0912] Regarding the hardware and software to be used

[0913] Hardware: Surveillance equipment (cameras), servers, database servers, smartphones, head-mounted displays

[0914] Software: OpenCV, FaceRecognitionModel, ActionDetectionModel, SpatialAnalysisModel, DatabaseConnection

[0915] The server first acquires video data from the monitoring device. Next, it analyzes the video frames using OpenCV to detect faces. It extracts the facial features detected using FaceRecognitionModel and estimates age and gender based on that data. It also compares specific features with an existing information base to determine whether the person is a repeat customer or a new visitor. This information is stored in a database via DatabaseConnection and compiled into a report.

[0916] Next, the ActionDetectionModel is used to analyze customer movement data. For each frame, customer movement is analyzed and movement data is collected. This data is then analyzed by the SpatialAnalysisModel and used to optimize product placement. Based on these analysis results, the server provides improvement advice to facility managers.

[0917] Data on products that customers picked up but did not purchase is collected using the ActionDetectionModel, and their level of interest is evaluated. This evaluation is analyzed as marketing data, and specific marketing strategies are formulated based on the collected results.

[0918] Specific example

[0919] For example, if a customer shows interest in a particular product but doesn't purchase it, the server analyzes their behavior and evaluates whether other customers tend to be interested in similar products. Also, if there's a sudden surge in customer traffic during a specific time period, the server analyzes the cause and provides specific suggestions for operational improvements. This allows operations staff to make evidence-based decisions.

[0920] An example of a prompt for a generated AI model is: "Recognize customer faces from video footage obtained from surveillance cameras, estimate their age and gender, and determine if they are repeat customers. Also, analyze customer movement patterns and provide advice on the optimal placement of products."

[0921] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0922] Step 1:

[0923] The server receives video data in real time from the monitoring device. This video data records the actions of customers from the moment they enter the store until they leave. Specifically, the monitoring device continuously captures frame images from cameras inside the store and sends this data to the server. The input is video data, and the output is video information stored on the server.

[0924] Step 2:

[0925] The server analyzes the received video data using OpenCV to detect the faces of individual customers. The input is the video data received in step 1, and the output is the coordinate information of the detected faces. Specifically, the server applies a face detection algorithm to each frame to identify the face region within the image.

[0926] Step 3:

[0927] The server extracts facial features detected using a FaceRecognitionModel and estimates age and sex. The input is the coordinate information of the face detected in step 2, and the output is the estimated age and sex information. Specifically, it applies a facial feature extraction algorithm and performs estimation using the age and sex model.

[0928] Step 4:

[0929] The server determines whether a customer is a repeat customer or a new visitor by comparing specific features with an existing information base. The input is the facial features extracted in step 3, and the output is the repeat customer determination result. Specifically, the server compares the features with existing data in the information base, calculates the degree of match, and makes the determination.

[0930] Step 5:

[0931] The server aggregates visitor attributes based on the collected data and generates a report. The input is all output data from steps 1 through 4, and the output is the aggregated report. Specifically, it applies a data analysis algorithm to create a report document in a specific format.

[0932] Step 6:

[0933] The server analyzes customer movement data using an ActionDetectionModel. The input is the video data received in step 1, and the output is the movement data. Specifically, it tracks the customer's position in each frame and stores the path information.

[0934] Step 7:

[0935] The server analyzes the movement data using SpatialAnalysisModel to calculate the appropriate placement of products. The input is the movement data analyzed in step 6, and the output is a specific proposal for the optimal product placement. Specifically, it applies an optimization algorithm based on the movement patterns and customer interests.

[0936] Step 8:

[0937] The server notifies the facility manager with advice based on the calculated layout. The input is the layout plan generated in step 7, and the output is the notified advice. Specifically, the server sends a recommendation for layout changes to the manager via the notification system.

[0938] Step 9:

[0939] The server retrieves data on products that customers showed interest in but did not purchase, from the collected image data. The input is the video data received in step 1, and the output is the data on products of interest. Specifically, it analyzes the customer's actions of picking up and putting back products and extracts that data.

[0940] Step 10:

[0941] The server evaluates the level of interest and analyzes it as marketing data. The input is the interest product data obtained in step 9, and the output is the level of interest evaluation data. Specifically, it calculates the level of interest score, analyzes the results, and organizes them as marketing data.

[0942] Step 11:

[0943] The server organizes visitor data by time period and analyzes customer behavior patterns. Input is the video data and all analysis results received in step 1, and output is the behavior pattern analysis results. Specifically, it divides the data by time period and analyzes customer inflow patterns.

[0944] Step 12:

[0945] The server calculates areas for operational improvement for each time period and notifies the operations staff. The input is the behavioral pattern data analyzed in step 11, and the output is the operational improvement suggestions that are notified. Specifically, it calculates a particular improvement action based on the analysis results and notifies the person in charge.

[0946] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0947] This invention is a system that analyzes video data acquired from a monitoring device, detects individual people, performs facial recognition and attribute estimation, and analyzes customer behavior based on the results. Furthermore, by combining it with an emotion engine, it also analyzes visitor emotional data to evaluate customer satisfaction and emotional responses to products of interest. A specific embodiment of this system is described below.

[0948] Explain the program's processing in natural language.

[0949] 1. Automatic counting of customer visitors

[0950] Surveillance devices are installed at the store entrance and inside the store to capture video of visitors in real time. The video data is sent to a server, which uses an AI model to detect people in the video. It performs facial recognition on each person and estimates their age and gender. This information is temporarily stored in a database to determine if the person is a repeat customer. Specifically, the server compares the facial features with an existing database, and if there is a match, the person is registered as a repeat customer. If there is no match, the person is registered as a new visitor in the database. Finally, visitor attribute data is compiled, and daily and monthly reports are generated. These reports are notified to the store operations manager. In addition, the server uses an emotion engine to analyze the facial expressions of people in the video data and recognize the customer's emotions. The recognized emotion data is also stored as attribute data.

[0951] Specific example

[0952] server

[0953] The server receives the video data and uses an AI model to detect people.

[0954] The server performs facial recognition and estimates age and gender. For example, it might save the information in the database as "male in his 30s."

[0955] The server matches the facial features against the database, counts the customer as a "repeat customer," and updates the database.

[0956] The server uses an emotion engine to recognize emotions such as "smile" or "confusion" and store them in a database.

[0957] 2. Analysis of in-store movement and layout advice

[0958] Multiple monitoring devices within the store capture customer movement data in real time and transmit it to a server. The server analyzes this data to track customer movements. It correlates the movement data with product placement data to identify which products are attracting the most interest. Using an AI model, the server calculates the optimal product placement based on the movement data and notifies store managers with specific advice. It also integrates emotional data with the movement data to analyze customers' emotional responses to products and store layouts.

[0959] Specific example

[0960] server

[0961] The server analyzes customer movement patterns and determines that "Customer A has shown interest in product X."

[0962] Based on the customer flow data, the server advises and notifies the user to place product Y near the entrance.

[0963] The server integrates emotional data and evaluates, for example, "Customer A was pleased when they saw product X."

[0964] 3. Analysis of interest in products that were not purchased.

[0965] Data on products that customers picked up but did not purchase is extracted from the video and sent to a server. The server analyzes this data to evaluate the customer's level of interest. Specifically, it correlates the customer's movement with the data of the target products and calculates an interest score. This data is notified to marketing personnel and used for sales promotion activities. In addition, an emotion engine is used to evaluate the emotional response of customers to products they picked up but did not purchase.

[0966] Specific example

[0967] server

[0968] The server identifies product B, which the user picked up but did not purchase, and rates its level of interest as "high".

[0969] The server generates a marketing report based on the evaluation data and notifies the client of the marketing measures that should be adopted.

[0970] The server saves the emotional data to the database, indicating "interested but undecided."

[0971] 4. Observation of customer behavior by time of day and operational advice

[0972] Monitoring devices capture video data of visitors at specific time intervals and transmit it to a server. The server organizes this data by time period and analyzes customer behavior patterns. Based on customer inflow and outflow, product attention, and sales data for specific time periods, an AI model calculates areas for improvement in store operations. This advice is notified and implemented by store managers. Sentimental data for each time period is also analyzed to evaluate customer satisfaction for each time slot.

[0973] Specific example

[0974] server

[0975] The server analyzes customer data by time of day and identifies that "customer traffic increases around 2 PM."

[0976] Based on the analysis results, the server will notify you of the advice to "deploy additional staff by 2 PM."

[0977] The server analyzes sentiment data for each time period and evaluates that "customer satisfaction is high at 2 PM."

[0978] The above describes the operation and specific examples of the system in an embodiment for carrying out the present invention. This makes it possible to efficiently analyze customer behavior data and emotional data, thereby optimizing store operations and improving customer satisfaction.

[0979] The following describes the processing flow.

[0980] Program processing flow

[0981] Automatic customer visitor counting and emotion recognition

[0982] Step 1:

[0983] The terminal acquires video data from the monitoring device and transmits it to the server in real time.

[0984] Step 2:

[0985] The server analyzes the received video data using an AI model to detect people in the video.

[0986] Step 3:

[0987] The server performs facial recognition on detected individuals and estimates their age and gender.

[0988] Step 4:

[0989] The server temporarily stores the estimated facial data and compares it with an existing database based on specific features.

[0990] Step 5:

[0991] The server identifies repeat customers and new visitors based on the results of matching them with the database, and updates and registers their respective data.

[0992] Step 6:

[0993] The server uses an emotion engine to analyze the facial expressions of people in the video data and recognize their emotions.

[0994] Step 7:

[0995] The server saves the recognized emotion data as attribute data.

[0996] Step 8:

[0997] The server aggregates visitor attribute data, generates daily and monthly reports, and notifies store operations staff.

[0998] Analysis of in-store movement and layout advice

[0999] Step 1:

[1000] The terminal acquires video data from multiple monitoring devices within the store and transmits it to the server in real time.

[1001] Step 2:

[1002] The server analyzes the received video data using an AI model to track customer movement.

[1003] Step 3:

[1004] The server correlates customer movement data with product placement data to identify which products customers showed interest in.

[1005] Step 4:

[1006] The server calculates the optimal placement of products based on customer movement data.

[1007] Step 5:

[1008] Based on the calculated server placement, the server generates specific advice and notifies the store operations manager.

[1009] Step 6:

[1010] The server integrates emotional data with movement data to analyze customers' emotional responses to products of interest and store layouts.

[1011] Analysis of interest in products that were not purchased.

[1012] Step 1:

[1013] The terminal acquires video data from the monitoring device and transmits it to the server in real time.

[1014] Step 2:

[1015] The server analyzes the video data to identify products that customers picked up but did not purchase.

[1016] Step 3:

[1017] The server tracks customer movement and evaluates their level of interest in products they did not purchase.

[1018] Step 4:

[1019] The server stores interest data in a database and organizes it as marketing data.

[1020] Step 5:

[1021] The server uses an emotion engine to evaluate the emotional response of customers to products they picked up but did not purchase.

[1022] Step 6:

[1023] The server generates marketing reports and notifies marketing personnel.

[1024] Observation of customer behavior by time of day and operational advice

[1025] Step 1:

[1026] The terminal captures video data from the monitoring device, categorized by time period, and transmits it to the server in real time.

[1027] Step 2:

[1028] The server organizes the video data received by time period and compiles data on customer inflow and outflow.

[1029] Step 3:

[1030] The server analyzes data for each time period to identify customer behavior patterns.

[1031] Step 4:

[1032] Based on the data analyzed by the server, an AI model calculates areas for improvement in operations.

[1033] Step 5:

[1034] The server uses an emotion engine to analyze emotional data for each time period and evaluate customer satisfaction.

[1035] Step 6:

[1036] Based on the areas for improvement calculated by the server, it generates specific advice and notifies the store operations manager.

[1037] (Example 2)

[1038] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[1039] Traditional store management systems had limitations in analyzing customer behavior and collecting attribute data, making it difficult to identify repeat customers or analyze their sentiment. Furthermore, data analysis for optimizing in-store product placement and providing appropriate marketing for customer interests was insufficient. This made optimizing store operations and improving customer satisfaction difficult.

[1040] The identification processing by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for analyzing video data acquired from a monitoring device to detect individual people, means for recognizing the faces of the detected people and estimating their age and gender, means for identifying repeat customers by comparing them with an existing database based on specific features, means for analyzing the facial expressions of people in the video data using an emotion engine and recognizing emotion data, means for aggregating visitor attributes and generating reports based on the collected data, means for providing improvement advice to each store, means for analyzing customer movement data within the store and calculating the appropriate placement of products, means for notifying store managers of advice based on the calculated placement, means for collecting data on products that customers showed interest in but did not purchase, and means for evaluating the level of interest and analyzing it as marketing data. This enables efficient analysis of customer behavior data and emotion data, optimizing store operations and improving customer satisfaction.

[1041] A "surveillance device" is a device installed at the entrance or inside a store to acquire video data in real time.

[1042] "Video data" refers to digital data acquired from surveillance devices that records the movements and actions of customers inside and outside the store.

[1043] "Person detection" is the process of identifying each individual within video data and confirming their presence.

[1044] "Facial recognition" is a technology that analyzes the faces of detected individuals and identifies the unique characteristics of each person.

[1045] "Age estimation" is the process of calculating a person's age based on data obtained through facial recognition.

[1046] "Gender estimation" is the process of determining a person's gender based on data obtained through facial recognition.

[1047] "Specific features" refer to unique facial feature data extracted through facial recognition to identify individual people.

[1048] "Existing database" refers to a database where past visitor data is stored.

[1049] A "repeat customer" is a customer who has visited the store before and is returning to the store.

[1050] An "emotion engine" is software that analyzes a person's facial expressions in video data and recognizes emotional data.

[1051] "Attribute data" is a general term for various types of customer-related data, such as age, gender, visit history, and emotions.

[1052] "Report generation" is the process of creating a report that includes statistics and analysis results based on collected data.

[1053] "Improvement advice" means suggesting areas for improvement in store operations based on the results of data analysis.

[1054] "Customer flow data" refers to data that records the routes and movements of customers as they move within a store.

[1055] "Appropriate product placement" refers to a placement method that optimizes the location of products based on customer movement data.

[1056] "Interest level assessment" is a process of measuring the degree of interest a customer has in a particular product.

[1057] "Marketing data" refers to sales promotion-related data analyzed based on customer behavior and interests.

[1058] "Analysis tools" refer to software and algorithms used to analyze collected data and extract useful information.

[1059] This invention is a system that analyzes customer behavior, performs facial recognition, estimates attributes, and analyzes emotions in store operations, and uses this information to support improvements in store operations. The following describes how this system is specifically implemented.

[1060] Automatic counting of customer visitors

[1061] terminal

[1062] The terminal uses surveillance equipment to capture real-time images of the store's entrance and interior, acquiring video data. This terminal includes a camera and video processing hardware.

[1063] server

[1064] The server inputs the acquired video data into the AI ​​model "PersonDetect_v2" to detect people in the video.

[1065] The server uses "FaceRecog_v3" to perform facial recognition on detected individuals and estimate their age and gender.

[1066] The server compares facial features with the database. If there is a match, the data is updated to indicate a repeat customer; otherwise, the person is registered in the database as a new visitor.

[1067] The server uses the emotion engine "EmotionEngine_v1" to analyze the facial expressions of people in the video, recognize emotion data, and save it to a database.

[1068] Based on the collected data, visitor attributes are compiled, and daily and monthly reports are generated. These reports are stored in a database and notifications are sent.

[1069] Analysis of in-store movement and layout advice

[1070] terminal

[1071] The terminal uses multiple monitoring devices to capture real-time data on customer movement within the store and transmits it to the server.

[1072] server

[1073] The server analyzes the received movement data using the AI ​​model "MovTrack_v2" to track customer movements.

[1074] The server correlates customer movement data with product placement data to identify products of high interest.

[1075] The server uses the AI ​​model "RecommendConfig_v1" to calculate the optimal placement of products and notifies store managers with specific placement advice.

[1076] Analysis of interest in products that were not purchased.

[1077] terminal

[1078] The terminal extracts data from the video of products that customers picked up but did not purchase, and sends it to the server.

[1079] server

[1080] The server analyzes the video data to identify products that were not purchased.

[1081] The server associates user flow data with data on the target product and calculates an interest score using the "InterestRate_v1" model.

[1082] The server generates a marketing report based on the evaluation data and notifies the marketing team.

[1083] Observation of customer behavior by time of day and operational advice

[1084] terminal

[1085] The terminal captures video data of visitors at set intervals and sends it to the server.

[1086] server

[1087] The server organizes video data by time period and uses "TimeSlotAnalyzer_v1" to analyze customer behavior patterns.

[1088] The server uses data on visitor inflow and outflow, product popularity, and sales figures for specific time periods to generate an AI model that calculates areas for improvement in store operations.

[1089] The server notifies the store operations manager of advice based on the analysis results.

[1090] Specific example

[1091] Example of a prompt

[1092] The following are specific examples of each process.

[1093] Automatic counting of customer visitors

[1094] "Please generate an AI model program that estimates a customer's age and gender and determines whether they are a repeat customer or a new customer."

[1095] Analysis of in-store movement and layout advice

[1096] "Please generate a program for a system that analyzes customer movement data and product placement data within a store to propose the optimal product layout."

[1097] Analysis of interest in products that were not purchased.

[1098] "Please create a program that analyzes data on products customers picked up but did not purchase, and evaluates their level of interest."

[1099] Observation of customer behavior by time of day and operational advice

[1100] "Please generate a program for a system that observes customer behavior at different times of the day and provides advice for store operations."

[1101] As described above, the system of the present invention efficiently analyzes customer behavior data and emotional data to optimize store operations and improve customer satisfaction.

[1102] The flow of the specific processing in Example 2 will be explained using Figure 13.

[1103] Automatic counting of customer visitors

[1104] Step 1: Acquire video data

[1105] terminal

[1106] Using surveillance equipment, the store entrance and interior are photographed in real time, and video data is acquired.

[1107] Input: Video of the store entrance and interior.

[1108] Output: Video data

[1109] Send the video data to the server.

[1110] Step 2: Receiving video data and detecting people

[1111] server

[1112] The video data received at the time of posting (10 minutes ago) is input into the AI ​​model "PersonDetect_v2" to detect people in the video.

[1113] Input: Video data

[1114] Output: Detected person data

[1115] The detected person data is temporarily stored in memory.

[1116] Step 3: Face Recognition and Attribute Estimation

[1117] server

[1118] Using the detected person data, the face recognition algorithm "FaceRecog_v3" is executed to identify each person's face.

[1119] Input: Detected person data

[1120] Output: Face recognition data

[1121] Age and gender are estimated based on the results of facial recognition.

[1122] Input: Face recognition data

[1123] Output: Age and gender data

[1124] Step 4: Matching with the database

[1125] server

[1126] The estimated facial features are compared with data from an existing database.

[1127] Input: Age and gender data, facial feature data

[1128] Output: Determination result for whether the visitor is a repeat customer or a new visitor.

[1129] If a match is found, the person is recognized as a repeat customer and the database is updated. If there is no match, the person is registered in the database as a new visitor.

[1130] Step 5: Sentiment analysis and data storage

[1131] server

[1132] The emotion engine "EmotionEngine_v1" is used to analyze the facial expressions of people in video data and recognize emotional data.

[1133] Input: Face recognition data

[1134] Output: Sentiment data

[1135] The recognized emotion data is stored in the database as attribute data.

[1136] Based on the collected data, visitor attributes are compiled, and daily and monthly reports are generated.

[1137] Input: Attribute data

[1138] Output: Daily and monthly reports

[1139] Analysis of in-store movement and layout advice

[1140] Step 1: Obtaining movement data

[1141] terminal

[1142] Multiple monitoring devices are used to capture real-time data on customer movement within the store and transmit it to a server.

[1143] Input: Video of customer movement within the store

[1144] Output: Movement data

[1145] Step 2: Receiving and analyzing movement data

[1146] server

[1147] The received movement data is analyzed using the AI ​​model "MovTrack_v2" to track customer movements.

[1148] Input: Movement data

[1149] Output: Tracking data

[1150] Step 3: Identifying the level of interest in the product

[1151] server

[1152] By linking customer traffic data with product placement data, we can identify products of high interest.

[1153] Input: Tracking data, product placement data

[1154] Output: List of products of high interest

[1155] Step 4: Calculation and notification of the optimal placement

[1156] server

[1157] The optimal placement of products is calculated using the AI ​​model "RecommendConfig_v1".

[1158] Input: List of products of high interest

[1159] Output: Optimal product placement plan

[1160] The calculated layout plan will be provided to the store operations manager as advice.

[1161] Analysis of interest in products that were not purchased.

[1162] Step 1: Data acquisition for the product picked up

[1163] terminal

[1164] The system captures video data of products that customers picked up but did not purchase, and sends it to a server.

[1165] Input: Customer behavior video

[1166] Output: Data of the product picked up

[1167] Step 2: Receiving and analyzing product data

[1168] server

[1169] The received video data is analyzed to identify products that were not purchased.

[1170] Input: Data of the product picked up

[1171] Output: List of items that were not purchased

[1172] Step 3: Evaluate and notify of interest level

[1173] server

[1174] The system correlates customer movement data with data on unpurchased items and calculates an interest score using "InterestRate_v1".

[1175] Input: List of unpurchased items, customer flow data

[1176] Output: Interest score

[1177] A marketing report is generated based on the interest score, and the marketing team is notified.

[1178] Observation of customer behavior by time of day and operational advice

[1179] Step 1: Obtain customer data by time of day

[1180] terminal

[1181] Video data of visitors is captured at each time slot and sent to the server.

[1182] Input: Video footage by time period

[1183] Output: Customer data by time period

[1184] Step 2: Data analysis by time period

[1185] server

[1186] The video data received for each time period is organized, and customer behavior patterns are analyzed using the "TimeSlotAnalyzer_v1" model.

[1187] Input: Customer data by time period

[1188] Output: Behavioral patterns by time of day

[1189] Step 3: Calculation and notification of areas for operational improvement

[1190] server

[1191] Based on visitor inflow and outflow during specific time periods, product popularity, and sales data, an AI model calculates areas for improvement in store operations.

[1192] Input: Behavioral patterns by time of day, attention level data, sales data

[1193] Output: Points for improvement in operations

[1194] Notify the store operations manager of the areas for improvement.

[1195] The above outlines the processing steps and specific operations of this system's program. This enables efficient analysis of customer behavior and emotional data, leading to optimized store operations and improved customer satisfaction.

[1196] (Application Example 2)

[1197] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[1198] The problem that this invention aims to solve is to efficiently monitor customer behavior and emotions in physical stores and improve customer satisfaction. By combining real-time customer behavior data and emotional data, the aim is to provide more appropriate services, improve store operations, and ultimately increase sales.

[1199] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[1200] In this invention, the server includes means for analyzing video data acquired from a monitoring device to detect individual people, means for recognizing the faces of detected people and estimating their age and gender, means for identifying repeat customers by comparing them with an existing database based on specific characteristics, means for aggregating visitor attributes based on collected data and generating reports, means for providing improvement advice to each store, means for analyzing customer emotion data and evaluating customer satisfaction, means for analyzing customer movement data within the store and calculating the optimal placement of products, means for collecting data on products that customers showed interest in but did not purchase and evaluating their level of interest, and means for analyzing customer behavior data by time of day and suggesting areas for improvement in store operations. This makes it possible to analyze customer behavior and emotions in real time and immediately implement measures to improve store operations.

[1201] A "surveillance device" refers to devices such as cameras and sensors that are installed to acquire video data.

[1202] "Video data" refers to data containing visual information acquired by surveillance equipment.

[1203] "Person detection means" refers to algorithms and technologies for identifying individual people from video data.

[1204] "Facial recognition means" refers to technology for identifying and distinguishing the faces of individuals detected from video data.

[1205] "Age estimation methods" refer to technologies that estimate the age of a person whose face has been recognized.

[1206] "Gender estimation means" refers to technology that estimates the gender of a person whose face has been recognized.

[1207] "Specific characteristics" refer to unique attributes such as facial features or behavioral patterns.

[1208] A "database" is an information system that stores information such as that of repeat customers.

[1209] A "repeat customer identification method" is a technology that determines whether a customer is a repeat customer by comparing them with an existing database based on specific characteristics.

[1210] "Visitor attributes" refer to characteristics such as the age, gender, and emotions of visitors.

[1211] A "report generation method" is a technology that compiles analysis results in report format based on collected data.

[1212] A "method for providing improvement advice" is a technology that notifies store managers of areas for improvement based on data analysis results.

[1213] "Emotional data" refers to data that represents the emotions estimated from the facial expressions of customers.

[1214] A "customer satisfaction evaluation method" is a technology that evaluates customer satisfaction based on emotional data.

[1215] "Customer movement data" refers to data that represents the paths customers take within a store.

[1216] The "optimal placement calculation method" is a technology that calculates the optimal placement of products based on traffic flow data.

[1217] "Products of interest" are products that customers showed interest in but did not purchase.

[1218] An "interest level evaluation method" is a technology that evaluates how interested a customer is in a product they are interested in.

[1219] "Customer behavior data by time of day" refers to data that represents customer behavior patterns during specific time periods.

[1220] "Store operation improvement methods" refer to technologies that analyze customer behavior data for each time period and identify areas for improvement in store operations.

[1221] The embodiments for carrying out this invention are described in detail below. This system analyzes video data acquired from a monitoring device to detect individual people, perform facial recognition, estimate attributes, and analyze emotion data, thereby understanding customer behavior and emotions in physical stores in real time, and aiming to optimize store operations and improve customer satisfaction.

[1222] System Configuration

[1223] This system consists of monitoring devices, a cloud server, and smart glasses or smartphones. The monitoring devices include standard surveillance cameras and sensors, while the cloud server implements AI models and emotion engines. The smart glasses and smartphones are terminals for store employees to receive and respond to information in real time.

[1224] Operation flow

[1225] 1. Automatic counting of customer visitors

[1226] server:

[1227] The server receives video data transmitted from the monitoring device and uses an AI model to detect people in the video. It recognizes the faces of the detected people and estimates their age and gender. For example, it might store them in the database as "a man in his 30s." Based on the facial features, it compares them with existing databases to determine whether they are repeat customers or new visitors. An emotion engine is used to analyze facial expressions and add emotional data such as "smiling" or "confused."

[1228] 2. Analysis of in-store movement and layout advice

[1229] server:

[1230] The system receives video data transmitted from multiple monitoring devices within the store and analyzes customer movement data in real time. Based on this, it correlates it with product placement data to identify which products are attracting the most attention. The server uses an AI model to calculate the optimal product placement based on the movement data and notifies store managers with specific advice. In addition, emotional data is integrated with the movement data to analyze customers' emotional reactions to products of interest and to the sales floor.

[1231] 3. Analysis of interest in products that were not purchased.

[1232] server:

[1233] Data on products that customers picked up but did not purchase is extracted from the video and sent to a server. The server analyzes this data to evaluate the customer's level of interest. It correlates the movement data with the data on the target products and calculates an interest score. This data is notified to marketing personnel and used for sales promotion activities. In addition, an emotion engine is used to evaluate the emotional response of customers to products they picked up but did not purchase.

[1234] 4. Observation of customer behavior by time of day and operational advice

[1235] server:

[1236] Monitoring devices capture video data of visitors at specific time intervals and transmit it to a server. The server organizes this data by time period and analyzes customer behavior patterns. Based on customer inflow and outflow, product attention, and sales data for specific time periods, an AI model calculates areas for improvement in store operations. This advice is notified and implemented by store managers. Sentimental data for each time period is also analyzed to evaluate customer satisfaction for each time slot.

[1237] Examples and prompts for generative AI models

[1238] For example, if customers tend to gather in a particular product section around 2 PM, and many of them smile while looking at the products but do not make a purchase, the system will determine that "they are highly interested in this product but are hesitant," and will either place related products nearby or notify staff to follow up.

[1239] Example of a prompt

[1240] Prompt example:

[1241] "Around 2 PM, customers tend to pick up product X but don't end up purchasing it. Customer sentiment data indicates they are interested but hesitant. Please propose effective marketing strategies to address this situation."

[1242] This system enables store managers to respond appropriately in real time, leading to improved customer satisfaction and increased sales.

[1243] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[1244] Step 1:

[1245] The surveillance system captures real-time video data of the store's entrance and interior.

[1246] Input: Video data acquired from a monitoring device.

[1247] Output: Sends a stream of video data to the cloud server.

[1248] Operation: The surveillance device's camera captures video data and sends it to a cloud server via the internet.

[1249] Step 2:

[1250] The server receives video data and uses an AI model to detect people in the video.

[1251] Input: Video data sent in Step 1.

[1252] Output: Location information of people detected in the video.

[1253] Operation: The AI ​​model analyzes the video data and executes an algorithm to identify the contours and locations of people.

[1254] Step 3:

[1255] The server recognizes the face of the detected person and estimates their age and gender.

[1256] Input: Location information of the person detected in Step 2.

[1257] Output: Recognized facial feature data, estimated age and sex data.

[1258] Operation: A facial recognition algorithm identifies a person's face and performs age and gender estimation.

[1259] Step 4:

[1260] The server matches facial features against an existing database to determine whether the person is a repeat customer or a new visitor.

[1261] Input: Facial feature data obtained in Step 3.

[1262] Output: Determination result of whether the visitor is a repeat customer or a new visitor.

[1263] Operation: Executes an algorithm to match feature data against a database and check if any matching data exists.

[1264] Step 5:

[1265] The server uses an emotion engine to analyze facial expressions and extract emotional data.

[1266] Input: Video data of the face recognized in Step 3.

[1267] Output: Extracted emotion data (e.g., smile, confusion, etc.).

[1268] Operation: The emotion engine analyzes facial features and classifies the emotional state.

[1269] Step 6:

[1270] The server collects visitor attributes and generates a report.

[1271] Input: Visitor data and sentiment data obtained in Steps 4 and 5.

[1272] Output: Summary report of visitor attributes.

[1273] Operation: Aggregates individual data points stored in the database and automatically generates daily and monthly reports.

[1274] Step 7:

[1275] A server receives video data transmitted from multiple monitoring devices within the store and analyzes customer movement data in real time.

[1276] Input: Video data stream from the monitoring device.

[1277] Output: Customer movement data.

[1278] Operation: Video data obtained from multiple cameras is analyzed using a tracking algorithm to identify customer movement patterns.

[1279] Step 8:

[1280] The server calculates the optimal placement of products based on customer movement data.

[1281] Input: Movement data obtained in Step 7.

[1282] Output: Optimal placement plan for products.

[1283] Operation: Integrates customer flow data with product placement data and calculates the optimal product placement using an AI model.

[1284] Step 9:

[1285] The server collects data on products that were not purchased and evaluates the level of interest.

[1286] Input: Video data and movement data from monitoring devices.

[1287] Output: Interest score for products that were not purchased.

[1288] Operation: The system determines whether a user picked up a specific product from the video, and if they did not purchase it, it analyzes their behavior and emotional data to evaluate their level of interest.

[1289] Step 10:

[1290] The server analyzes customer behavior data for each time period and suggests areas for improvement in store operations.

[1291] Inputs: Video data by time of day, sales data, customer flow data, and sentiment data.

[1292] Output: Customer behavior report by time of day and operational improvement advice.

[1293] Function: It organizes and analyzes data for each time period, evaluates customer behavior patterns and emotions during specific time periods, and generates operational improvement advice based on that.

[1294] Through the above processing steps, it is possible to analyze customer behavior and emotions in physical stores in detail and achieve optimal store operations in real time.

[1295] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[1296] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include those described above. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions shown by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1297] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[1298] [Third Embodiment]

[1299] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[1300] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[1301] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1302] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[1303] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[1304] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[1305] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[1306] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[1307] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1308] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1309] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[1310] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[1311] This invention relates to a system that analyzes video data acquired from a surveillance device, detects individual people, performs facial recognition and attribute estimation, and analyzes customer behavior based on the results. The program and processing of this system are described in detail below.

[1312] Explain the program's processing in natural language.

[1313] 1. Automatic counting of customer visitors

[1314] Surveillance devices are installed at the store entrance and inside the store to capture video of visitors in real time. The video data is sent to a server, which uses an AI model to detect people in the video. It performs facial recognition on each person and estimates their age and gender. This information is temporarily stored in a database to determine if a customer is a repeat customer. Specifically, the server compares the facial features with an existing database, and if there is a match, the person is registered as a repeat customer. If there is no match, the person is registered as a new visitor in the database. Finally, visitor attribute data is compiled, and daily and monthly reports are generated. These reports are notified to the store operations manager.

[1315] Specific example

[1316] server

[1317] The server receives the video data and uses an AI model to detect people.

[1318] The server performs facial recognition and estimates age and gender. For example, it might save the information in the database as "male in his 30s."

[1319] The server matches the facial features against the database, counts the customer as a "repeat customer," and updates the database.

[1320] 2. Analysis of in-store movement and layout advice

[1321] Multiple monitoring devices within the store capture customer movement data in real time and transmit it to a server. The server analyzes this data to track customer movements. It correlates the movement data with product placement data to identify which products are attracting the most interest. Using an AI model, the server calculates the optimal product placement based on the movement data and notifies store managers with specific advice.

[1322] Specific example

[1323] server

[1324] The server analyzes customer movement patterns and determines that "Customer A has shown interest in product X."

[1325] Based on the customer flow data, the server advises and notifies the user to place product Y near the entrance.

[1326] 3. Analysis of interest in products that were not purchased.

[1327] Data on products that customers picked up but did not purchase is extracted from the video and sent to a server. The server analyzes this data to evaluate the customer's level of interest. Specifically, it correlates the customer's movement with the data on the target products and calculates an interest score. This data is then notified to marketing personnel and used for sales promotion activities.

[1328] Specific example

[1329] server

[1330] The server identifies product B, which the user picked up but did not purchase, and rates its level of interest as "high".

[1331] The server generates a marketing report based on the evaluation data and notifies the client of the marketing measures that should be adopted.

[1332] 4. Observation of customer behavior by time of day and operational advice

[1333] Monitoring devices capture video data of visitors at specific time intervals and transmit it to a server. The server organizes this data by time period and analyzes customer behavior patterns. Based on customer inflow and outflow, product attention levels, and sales data for specific time periods, an AI model calculates areas for improvement in store operations. This advice is communicated to store managers for their implementation.

[1334] Specific example

[1335] server

[1336] The server analyzes customer data by time of day and identifies that "customer traffic increases around 2 PM."

[1337] Based on the analysis results, the server will notify you of the advice to "deploy additional staff by 2 PM."

[1338] The above describes the operation and specific examples of the system in an embodiment for carrying out the present invention. This makes it possible to efficiently analyze customer behavior data and optimize store operations.

[1339] The following describes the processing flow.

[1340] Program processing flow

[1341] Automatic counting of customer visitors

[1342] Step 1:

[1343] The terminal acquires video data from the monitoring device and transmits it to the server in real time.

[1344] Step 2:

[1345] The server analyzes the received video data using an AI model to detect people in the video.

[1346] Step 3:

[1347] The server performs facial recognition on detected individuals and estimates their age and gender.

[1348] Step 4:

[1349] The server temporarily stores the estimated facial data and compares it with an existing database based on specific features.

[1350] Step 5:

[1351] The server identifies repeat customers and new visitors based on the results of matching them with the database, and updates and registers their respective data.

[1352] Step 6:

[1353] The server aggregates visitor attribute data, generates daily and monthly reports, and notifies store operations staff.

[1354] Analysis of in-store movement and layout advice

[1355] Step 1:

[1356] The terminal acquires video data from multiple monitoring devices within the store and transmits it to the server in real time.

[1357] Step 2:

[1358] The server analyzes the received video data using an AI model to track customer movement.

[1359] Step 3:

[1360] The server correlates customer movement data with product placement data to identify which products customers showed interest in.

[1361] Step 4:

[1362] The server calculates the optimal placement of products based on customer movement data.

[1363] Step 5:

[1364] Based on the calculated server placement, the server generates specific advice and notifies the store operations manager.

[1365] Analysis of interest in products that were not purchased.

[1366] Step 1:

[1367] The terminal acquires video data from the monitoring device and transmits it to the server in real time.

[1368] Step 2:

[1369] The server analyzes the video data to identify products that customers picked up but did not purchase.

[1370] Step 3:

[1371] The server tracks customer movement and evaluates their level of interest in products they did not purchase.

[1372] Step 4:

[1373] The server stores interest data in a database and organizes it as marketing data.

[1374] Step 5:

[1375] The server generates marketing reports and notifies marketing personnel.

[1376] Observation of customer behavior by time of day and operational advice

[1377] Step 1:

[1378] The terminal captures video data from the monitoring device, categorized by time period, and transmits it to the server in real time.

[1379] Step 2:

[1380] The server organizes the video data received by time period and compiles data on customer inflow and outflow.

[1381] Step 3:

[1382] The server analyzes data for each time period to identify customer behavior patterns.

[1383] Step 4:

[1384] Based on the data analyzed by the server, an AI model calculates areas for improvement in operations.

[1385] Step 5:

[1386] Based on the areas for improvement calculated by the server, it generates specific advice and notifies the store operations manager.

[1387] (Example 1)

[1388] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1389] Traditional systems made it difficult to effectively collect and analyze customer behavior data from store visitors and propose improvements to store operations. Furthermore, strategic marketing measures based on individual customer attributes and interests were not adequately implemented. In particular, identifying repeat customers, optimizing product placement, and evaluating interest in products that were not purchased were challenging, resulting in a lack of concrete, data-driven advice for efficient store operations and increased sales.

[1390] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[1391] In this invention, the server includes means for analyzing video data acquired from a monitoring device to detect individual people, means for recognizing the faces of detected people and estimating their age and gender, means for identifying repeat customers by comparing them with an existing database based on characteristic information, means for aggregating visitor attributes based on collected data and generating reports, means for providing improvement advice to each store, means for analyzing customer movement data to calculate the optimal placement of products and notifying each store, means for collecting data on products that customers picked up but did not purchase and evaluating their level of interest, means for acquiring video data from installed monitoring devices at different times of the day and analyzing customer behavior patterns, and means for calculating and notifying improvements to store operations by correlating the analysis results with sales data. This makes it possible to efficiently collect and analyze customer behavior data and optimize store operations.

[1392] A "surveillance device" is a device installed at the entrance or inside a store to acquire video data.

[1393] "Video data" refers to image and video information acquired by surveillance equipment.

[1394] "Analysis" refers to techniques that process information such as video data to detect individual people or analyze movement patterns.

[1395] "Person detection" is a method of identifying individual people from video data.

[1396] "Facial recognition" is a method of identifying a specific person by analyzing the face of a detected individual.

[1397] "Age and gender estimation" refers to a method of inferring a person's age group and gender based on person detection and facial recognition.

[1398] "Characteristic information" refers to information used to identify individual people, such as facial features and movement data.

[1399] A "database" is a system for organizing and storing collected data.

[1400] A "repeat customer" refers to a person who has visited the store before and is recorded in the existing database.

[1401] "Visitor attributes" refer to characteristics such as visitor age, gender, movement patterns, and purchase history.

[1402] "Report generation" refers to the process of compiling analysis results based on collected data and creating a report.

[1403] "Improvement advice" is a method of providing instructions on areas for improvement in store operations based on collected and analyzed data.

[1404] "Customer movement data" refers to data that shows the routes and behavioral patterns of customers as they move within a store.

[1405] "Product placement" refers to the location and method of displaying products within a store.

[1406] "Interest level" is a measure that evaluates the degree of interest a customer has in a product they picked up but did not purchase.

[1407] "Video data by time period" refers to video data acquired during a specific time period.

[1408] "Behavioral patterns" refer to data that shows the influx and outflow of customers and the trends of popular products during specific time periods.

[1409] This invention relates to a system that analyzes video data acquired from a surveillance device, detects individual people, performs facial recognition and attribute estimation, and analyzes customer behavior based on the results. The program and processing of this system are described below.

[1410] This system consists of monitoring devices, a server, and terminals. The monitoring devices are installed at the entrance and inside the store and acquire customer video data in real time. The acquired video data is transmitted to the server via the terminals.

[1411] The server uses an AI model to analyze the received video data. The AI ​​model utilizes generative AI technology to detect people in the video with high accuracy. At this stage, the server uses a facial recognition algorithm to identify the faces of detected individuals and estimate their age and gender. For example, it might store this information in a database as "a man in his 30s." The software used for this includes common facial recognition libraries and attribute estimation algorithms.

[1412] Furthermore, the server matches facial features against an existing database, counting repeat customers if a match is found. If there is no match, the customer is registered in the database as a new visitor. Finally, visitor attribute data is aggregated, and daily and monthly reports are generated. These reports are then communicated to store operations staff.

[1413] Next, the server analyzes customer movement data transmitted from multiple monitoring devices within the store to track customer behavior. It then correlates this movement data with product placement data to identify which products are attracting the most attention. Using an AI model, the server calculates the optimal product placement based on the movement data and provides specific advice to store managers.

[1414] The server also analyzes data on products that customers picked up but did not purchase, and evaluates the customer's level of interest. Specifically, it correlates customer movement with data on the target products and calculates an interest score. This data is then shared with marketing personnel and used for sales promotion activities.

[1415] Furthermore, monitoring devices capture video data of visitors at specific time intervals and transmit it to a server. The server organizes this data by time period and analyzes customer behavior patterns. Based on customer inflow and outflow, product attention levels, and sales data for specific time periods, an AI model calculates areas for improvement in store operations. This advice is then communicated to store managers for implementation.

[1416] For example, the server might analyze that "customer traffic increased during the 2 PM timeframe" and notify the store manager of the need to allocate additional staff.

[1417] Example of a prompt

[1418] "Analyze customer movement data recorded by security cameras to identify which products are attracting the most attention. Based on these results, propose the optimal product placement and notify the store operations manager."

[1419] The above outlines a specific implementation of this system. This makes it possible to efficiently analyze customer behavior data and optimize store operations.

[1420] The flow of the specific processing in Example 1 will be explained using Figure 11.

[1421] Step 1:

[1422] The terminal acquires video data in real time from surveillance devices installed at the entrance and inside the store.

[1423] Specific operation: The monitoring device captures video data using a camera sensor, and the terminal receives this data.

[1424] Input: Video data from a monitoring device.

[1425] Output: Video data received by the terminal.

[1426] Step 2:

[1427] The device sends the acquired video data to the server.

[1428] Specific operation: The device uploads the received video data to the server via the network.

[1429] Input: Video data received by the device.

[1430] Output: Video data received by the server.

[1431] Step 3:

[1432] The server inputs the received video data into an AI model to detect people in the video.

[1433] Specific operation: The server inputs video data into the AI ​​model, and the model detects people in each video frame.

[1434] Input: Video data received by the server.

[1435] Output: Person detection results by the AI ​​model.

[1436] Step 4:

[1437] The server performs facial recognition on detected individuals and estimates their age and gender.

[1438] Specific operation: The server uses a facial recognition algorithm to analyze the detected face and estimate its age and gender. For example, it might save it in the database as "male in his 30s."

[1439] Input: Person detection results from an AI model.

[1440] Output: Person data with estimated age and gender.

[1441] Step 5:

[1442] The server matches the facial features against an existing database. If there is a match, the person is counted as a repeat customer. If there is no match, they are registered in the database as a new visitor.

[1443] Specific operation: The server compares the identified facial feature data with the database to determine whether it is a new or repeat customer.

[1444] Input: Individual data with estimated age and gender, and existing databases.

[1445] Output: Repeat customer list, New visitor list.

[1446] Step 6:

[1447] The server aggregates visitor attribute data and generates daily and monthly reports.

[1448] Specific operation: The server aggregates data on new and repeat customers, and a report generation program creates daily and monthly reports.

[1449] Input: Repeat customer list, New visitor list.

[1450] Output: Daily and monthly reports.

[1451] Step 7:

[1452] The server notifies the store operations manager of the report.

[1453] Specific operation: The server sends the generated report to the responsible person via email or dashboard.

[1454] Input: Daily and monthly reports.

[1455] Output: Report sent to store operations staff.

[1456] Step 8:

[1457] The terminal acquires customer movement data in real time from multiple monitoring devices within the store.

[1458] Specific operation: The monitoring device tracks customer movements within the store and captures movement data in real time.

[1459] Input: Customer traffic data from monitoring devices within the store.

[1460] Output: Movement data received by the terminal.

[1461] Step 9:

[1462] The terminal sends the movement data it acquires to the server.

[1463] Specific operation: The terminal receives movement data and sends it to the server via the network.

[1464] Input: Movement data received by the terminal.

[1465] Output: Movement data received by the server.

[1466] Step 10:

[1467] The server analyzes the movement data it receives to track customer movements.

[1468] Specific operation: The server processes the movement data using an analysis algorithm to analyze customer movement patterns.

[1469] Input: Movement data received by the server.

[1470] Output: Customer movement analysis results.

[1471] Step 11:

[1472] The server correlates customer flow data with product placement data to identify which products are attracting the most attention.

[1473] Specific operation: The server matches customer flow data with product placement data to identify products of high interest.

[1474] Input: Customer movement analysis results, product placement data.

[1475] Output: A list of products the customer has shown interest in.

[1476] Step 12:

[1477] The server calculates the optimal placement of products and notifies store managers with specific advice.

[1478] Specific operation: The server uses an optimal placement algorithm to generate product placement advice and notifies the person in charge.

[1479] Input: A list of products that the customer has shown interest in.

[1480] Output: Placement advice report.

[1481] Step 13:

[1482] The terminal extracts data from in-store monitoring devices about items that customers picked up but did not purchase, and sends it to a server.

[1483] Specific operation: The monitoring device tracks customer behavior, collects data on items picked up but not purchased, and transmits it to the terminal.

[1484] Input: Data on products that were not purchased.

[1485] Output: Data on unpurchased items received by the server.

[1486] Step 14:

[1487] The server analyzes data on unpurchased items to assess the customer's level of interest.

[1488] Specific operation: The server analyzes data on unpurchased items and calculates an interest score.

[1489] Input: Data on unpurchased items received by the server.

[1490] Output: A list of products with interest scores.

[1491] Step 15:

[1492] The server notifies marketing personnel of interest levels, who then use the data for sales promotion activities.

[1493] Specific operation: The server creates a marketing report based on interest level data and sends it to the person in charge.

[1494] Input: A list of products with interest scores.

[1495] Output: Marketing report.

[1496] Step 16:

[1497] The terminal acquires video data of visitors from the monitoring device at specific time intervals and sends it to the server.

[1498] Specific operation: The device captures video data at specified time intervals and sends it to the server.

[1499] Input: Time-series video data from monitoring equipment.

[1500] Output: Video data received by the server for each time period.

[1501] Step 17:

[1502] The server organizes video data by time period and analyzes customer behavior patterns.

[1503] Specific operation: The server collects data for each time period and analyzes behavioral patterns.

[1504] Input: Video data received by the server for each time period.

[1505] Output: Customer behavior patterns by time of day.

[1506] Step 18:

[1507] The server correlates the analysis results with sales data to calculate areas for improvement in store operations.

[1508] Specific operation: The server analyzes the correlation between behavioral patterns and sales data to identify areas for improvement.

[1509] Input: Customer behavior patterns by time of day, sales data.

[1510] Output: A report identifying areas for improvement.

[1511] Step 19:

[1512] The server notifies the store operations manager with improvement advice.

[1513] Specific action: The server sends a report indicating areas for improvement to the responsible person and notifies them.

[1514] Input: A report identifying areas for improvement.

[1515] Output: Improvement advice notified to store operations staff.

[1516] (Application Example 1)

[1517] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1518] Traditional store operations have made it difficult to adequately analyze customer behavior data, particularly regarding customer movement patterns, product placement, and analysis of unpurchased items, without utilizing real-time data. Furthermore, there was a lack of means to observe customer behavior at different times of the day and to obtain concrete suggestions for operational improvements based on that observation. As a result, optimizing store operations was hindered, making it difficult to contribute to increased sales and improved customer satisfaction.

[1519] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[1520] In this invention, the server includes means for analyzing image data acquired from a monitoring device to detect individual people, means for recognizing the faces of detected people and estimating their age and gender, means for identifying repeat customers by comparing them with an existing information base based on specific features, means for aggregating visitor attributes based on collected data and generating reports, means for providing improvement suggestions to each facility, means for analyzing customer movement data and calculating the appropriate placement of products, means for notifying facility operators of the suggestions based on the calculated placement, means for acquiring data on products that customers showed interest in but did not purchase from collected image data, means for evaluating the level of interest and analyzing it as marketing data, means for organizing visitor data by time of day and analyzing customer behavior patterns, and means for calculating points for operational improvement in each time period and notifying operators. This makes it possible to efficiently collect and analyze customer behavior data within stores and to make operational improvement suggestions in real time.

[1521] A "surveillance device" refers to a device that acquires video data and performs tasks such as person detection and movement analysis.

[1522] "Image data" refers to video information acquired from surveillance devices and is used for person detection and movement analysis.

[1523] "Person detection" refers to a method of identifying individual people from image data.

[1524] "Facial recognition" refers to a method of identifying a person by analyzing the facial features of a detected individual.

[1525] "Age and gender estimation" refers to the act of inferring a person's age and gender from the results of facial recognition.

[1526] "Specific features" refer to facial characteristic information obtained through facial recognition, and are data used to compare this information with existing information bases.

[1527] "Existing information base" refers to a database containing customer data collected previously.

[1528] A "repeat customer" refers to a customer who has a history of visiting the store in the past and has been determined to be a returning customer based on matching specific characteristics.

[1529] "Collected data" refers to all data obtained from monitoring devices and analysis means.

[1530] "Visitor attributes" refer to characteristics such as the age, gender, and number of visits of customers who visit the store.

[1531] "Report generation" refers to the act of compiling collected data and creating a document that summarizes it in a specific format.

[1532] "Each facility" refers to the stores or business locations that are the subject of data analysis.

[1533] An "improvement suggestion" is a proposal that outlines specific measures to improve store operations based on collected data.

[1534] "Customer movement data" refers to information that shows the paths customers take within a store.

[1535] "Appropriate product placement" refers to advice on optimizing product placement based on customer traffic data.

[1536] A "facility operations manager" refers to a manager responsible for the operation of a store or business location.

[1537] "Data acquisition" refers to the act of extracting necessary information from image data.

[1538] "Interest level assessment" is the act of analyzing and evaluating the level of interest shown by customers.

[1539] "Marketing data" refers to data about customer interests and behavior extracted from collected data, and is used in marketing strategies.

[1540] "Visitor data by time slot" refers to customer visit information organized by specific time periods.

[1541] "Customer behavior patterns" refer to pattern information that shows how customers move around within a store and what actions they take.

[1542] "Areas for operational improvement" refers to the points necessary to improve store operations based on collected data.

[1543] This invention relates to a system that analyzes image data acquired from monitoring devices to track and analyze customer behavior in real time. This system collects customer attribute data, movement data, and purchasing behavior information, contributing to the optimization of facility operations. Specific embodiments of this system are described below.

[1544] The server uses multiple AI models to process image data. For example, it uses OpenCV to acquire image data and a FaceRecognitionModel to recognize and analyze people's faces. It also uses an ActionDetectionModel to analyze customer movement and behavior. Furthermore, it uses a SpatialAnalysisModel to analyze the collected movement data and calculate the appropriate placement of products. All of this data is then centrally managed and stored by DatabaseConnection.

[1545] Regarding the hardware and software to be used

[1546] Hardware: Surveillance equipment (cameras), servers, database servers, smartphones, head-mounted displays

[1547] Software: OpenCV, FaceRecognitionModel, ActionDetectionModel, SpatialAnalysisModel, DatabaseConnection

[1548] The server first acquires video data from the monitoring device. Next, it analyzes the video frames using OpenCV to detect faces. It extracts the facial features detected using FaceRecognitionModel and estimates age and gender based on that data. It also compares specific features with an existing information base to determine whether the person is a repeat customer or a new visitor. This information is stored in a database via DatabaseConnection and compiled into a report.

[1549] Next, the ActionDetectionModel is used to analyze customer movement data. For each frame, customer movement is analyzed and movement data is collected. This data is then analyzed by the SpatialAnalysisModel and used to optimize product placement. Based on these analysis results, the server provides improvement advice to facility managers.

[1550] Data on products that customers picked up but did not purchase is collected using the ActionDetectionModel, and their level of interest is evaluated. This evaluation is analyzed as marketing data, and specific marketing strategies are formulated based on the collected results.

[1551] Specific example

[1552] For example, if a customer shows interest in a particular product but doesn't purchase it, the server analyzes their behavior and evaluates whether other customers tend to be interested in similar products. Also, if there's a sudden surge in customer traffic during a specific time period, the server analyzes the cause and provides specific suggestions for operational improvements. This allows operations staff to make evidence-based decisions.

[1553] An example of a prompt for a generated AI model is: "Recognize customer faces from video footage obtained from surveillance cameras, estimate their age and gender, and determine if they are repeat customers. Also, analyze customer movement patterns and provide advice on the optimal placement of products."

[1554] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[1555] Step 1:

[1556] The server receives video data in real time from the monitoring device. This video data records the actions of customers from the moment they enter the store until they leave. Specifically, the monitoring device continuously captures frame images from cameras inside the store and sends this data to the server. The input is video data, and the output is video information stored on the server.

[1557] Step 2:

[1558] The server analyzes the received video data using OpenCV to detect the faces of individual customers. The input is the video data received in step 1, and the output is the coordinate information of the detected faces. Specifically, the server applies a face detection algorithm to each frame to identify the face region within the image.

[1559] Step 3:

[1560] The server extracts facial features detected using a FaceRecognitionModel and estimates age and sex. The input is the coordinate information of the face detected in step 2, and the output is the estimated age and sex information. Specifically, it applies a facial feature extraction algorithm and performs estimation using the age and sex model.

[1561] Step 4:

[1562] The server determines whether a customer is a repeat customer or a new visitor by comparing specific features with an existing information base. The input is the facial features extracted in step 3, and the output is the repeat customer determination result. Specifically, the server compares the features with existing data in the information base, calculates the degree of match, and makes the determination.

[1563] Step 5:

[1564] The server aggregates visitor attributes based on the collected data and generates a report. The input is all output data from steps 1 through 4, and the output is the aggregated report. Specifically, it applies a data analysis algorithm to create a report document in a specific format.

[1565] Step 6:

[1566] The server analyzes customer movement data using an ActionDetectionModel. The input is the video data received in step 1, and the output is the movement data. Specifically, it tracks the customer's position in each frame and stores the path information.

[1567] Step 7:

[1568] The server analyzes the movement data using SpatialAnalysisModel to calculate the appropriate placement of products. The input is the movement data analyzed in step 6, and the output is a specific proposal for the optimal product placement. Specifically, it applies an optimization algorithm based on the movement patterns and customer interests.

[1569] Step 8:

[1570] The server notifies the facility manager with advice based on the calculated layout. The input is the layout plan generated in step 7, and the output is the notified advice. Specifically, the server sends a recommendation for layout changes to the manager via the notification system.

[1571] Step 9:

[1572] The server retrieves data on products that customers showed interest in but did not purchase, from the collected image data. The input is the video data received in step 1, and the output is the data on products of interest. Specifically, it analyzes the customer's actions of picking up and putting back products and extracts that data.

[1573] Step 10:

[1574] The server evaluates the level of interest and analyzes it as marketing data. The input is the interest product data obtained in step 9, and the output is the level of interest evaluation data. Specifically, it calculates the level of interest score, analyzes the results, and organizes them as marketing data.

[1575] Step 11:

[1576] The server organizes visitor data by time period and analyzes customer behavior patterns. Input is the video data and all analysis results received in step 1, and output is the behavior pattern analysis results. Specifically, it divides the data by time period and analyzes customer inflow patterns.

[1577] Step 12:

[1578] The server calculates areas for operational improvement for each time period and notifies the operations staff. The input is the behavioral pattern data analyzed in step 11, and the output is the operational improvement suggestions that are notified. Specifically, it calculates a particular improvement action based on the analysis results and notifies the person in charge.

[1579] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[1580] This invention is a system that analyzes video data acquired from a monitoring device, detects individual people, performs facial recognition and attribute estimation, and analyzes customer behavior based on the results. Furthermore, by combining it with an emotion engine, it also analyzes visitor emotional data to evaluate customer satisfaction and emotional responses to products of interest. A specific embodiment of this system is described below.

[1581] Explain the program's processing in natural language.

[1582] 1. Automatic counting of customer visitors

[1583] Surveillance devices are installed at the store entrance and inside the store to capture video of visitors in real time. The video data is sent to a server, which uses an AI model to detect people in the video. It performs facial recognition on each person and estimates their age and gender. This information is temporarily stored in a database to determine if the person is a repeat customer. Specifically, the server compares the facial features with an existing database, and if there is a match, the person is registered as a repeat customer. If there is no match, the person is registered as a new visitor in the database. Finally, visitor attribute data is compiled, and daily and monthly reports are generated. These reports are notified to the store operations manager. In addition, the server uses an emotion engine to analyze the facial expressions of people in the video data and recognize the customer's emotions. The recognized emotion data is also stored as attribute data.

[1584] Specific example

[1585] server

[1586] The server receives the video data and uses an AI model to detect people.

[1587] The server performs facial recognition and estimates age and gender. For example, it might save the information in the database as "male in his 30s."

[1588] The server matches the facial features against the database, counts the customer as a "repeat customer," and updates the database.

[1589] The server uses an emotion engine to recognize emotions such as "smile" or "confusion" and store them in a database.

[1590] 2. Analysis of in-store movement and layout advice

[1591] Multiple monitoring devices within the store capture customer movement data in real time and transmit it to a server. The server analyzes this data to track customer movements. It correlates the movement data with product placement data to identify which products are attracting the most interest. Using an AI model, the server calculates the optimal product placement based on the movement data and notifies store managers with specific advice. It also integrates emotional data with the movement data to analyze customers' emotional responses to products and store layouts.

[1592] Specific example

[1593] server

[1594] The server analyzes customer movement patterns and determines that "Customer A has shown interest in product X."

[1595] Based on the customer flow data, the server advises and notifies the user to place product Y near the entrance.

[1596] The server integrates emotional data and evaluates, for example, "Customer A was pleased when they saw product X."

[1597] 3. Analysis of interest in products that were not purchased.

[1598] Data on products that customers picked up but did not purchase is extracted from the video and sent to a server. The server analyzes this data to evaluate the customer's level of interest. Specifically, it correlates the customer's movement with the data of the target products and calculates an interest score. This data is notified to marketing personnel and used for sales promotion activities. In addition, an emotion engine is used to evaluate the emotional response of customers to products they picked up but did not purchase.

[1599] Specific example

[1600] server

[1601] The server identifies product B, which the user picked up but did not purchase, and rates its level of interest as "high".

[1602] The server generates a marketing report based on the evaluation data and notifies the client of the marketing measures that should be adopted.

[1603] The server saves the emotional data to the database, indicating "interested but undecided."

[1604] 4. Observation of customer behavior by time of day and operational advice

[1605] Monitoring devices capture video data of visitors at specific time intervals and transmit it to a server. The server organizes this data by time period and analyzes customer behavior patterns. Based on customer inflow and outflow, product attention, and sales data for specific time periods, an AI model calculates areas for improvement in store operations. This advice is notified and implemented by store managers. Sentimental data for each time period is also analyzed to evaluate customer satisfaction for each time slot.

[1606] Specific example

[1607] server

[1608] The server analyzes customer data by time of day and identifies that "customer traffic increases around 2 PM."

[1609] Based on the analysis results, the server will notify you of the advice to "deploy additional staff by 2 PM."

[1610] The server analyzes sentiment data for each time period and evaluates that "customer satisfaction is high at 2 PM."

[1611] The above describes the operation and specific examples of the system in an embodiment for carrying out the present invention. This makes it possible to efficiently analyze customer behavior data and emotional data, thereby optimizing store operations and improving customer satisfaction.

[1612] The following describes the processing flow.

[1613] Program processing flow

[1614] Automatic customer visitor counting and emotion recognition

[1615] Step 1:

[1616] The terminal acquires video data from the monitoring device and transmits it to the server in real time.

[1617] Step 2:

[1618] The server analyzes the received video data using an AI model to detect people in the video.

[1619] Step 3:

[1620] The server performs facial recognition on detected individuals and estimates their age and gender.

[1621] Step 4:

[1622] The server temporarily stores the estimated facial data and compares it with an existing database based on specific features.

[1623] Step 5:

[1624] The server identifies repeat customers and new visitors based on the results of matching them with the database, and updates and registers their respective data.

[1625] Step 6:

[1626] The server uses an emotion engine to analyze the facial expressions of people in the video data and recognize their emotions.

[1627] Step 7:

[1628] The server saves the recognized emotion data as attribute data.

[1629] Step 8:

[1630] The server aggregates visitor attribute data, generates daily and monthly reports, and notifies store operations staff.

[1631] Analysis of in-store movement and layout advice

[1632] Step 1:

[1633] The terminal acquires video data from multiple monitoring devices within the store and transmits it to the server in real time.

[1634] Step 2:

[1635] The server analyzes the received video data using an AI model to track customer movement.

[1636] Step 3:

[1637] The server correlates customer movement data with product placement data to identify which products customers showed interest in.

[1638] Step 4:

[1639] The server calculates the optimal placement of products based on customer movement data.

[1640] Step 5:

[1641] Based on the calculated server placement, the server generates specific advice and notifies the store operations manager.

[1642] Step 6:

[1643] The server integrates emotional data with movement data to analyze customers' emotional responses to products of interest and store layouts.

[1644] Analysis of interest in products that were not purchased.

[1645] Step 1:

[1646] The terminal acquires video data from the monitoring device and transmits it to the server in real time.

[1647] Step 2:

[1648] The server analyzes the video data to identify products that customers picked up but did not purchase.

[1649] Step 3:

[1650] The server tracks customer movement and evaluates their level of interest in products they did not purchase.

[1651] Step 4:

[1652] The server stores interest data in a database and organizes it as marketing data.

[1653] Step 5:

[1654] The server uses an emotion engine to evaluate the emotional response of customers to products they picked up but did not purchase.

[1655] Step 6:

[1656] The server generates marketing reports and notifies marketing personnel.

[1657] Observation of customer behavior by time of day and operational advice

[1658] Step 1:

[1659] The terminal captures video data from the monitoring device, categorized by time period, and transmits it to the server in real time.

[1660] Step 2:

[1661] The server organizes the video data received by time period and compiles data on customer inflow and outflow.

[1662] Step 3:

[1663] The server analyzes data for each time period to identify customer behavior patterns.

[1664] Step 4:

[1665] Based on the data analyzed by the server, an AI model calculates areas for improvement in operations.

[1666] Step 5:

[1667] The server uses an emotion engine to analyze emotional data for each time period and evaluate customer satisfaction.

[1668] Step 6:

[1669] Based on the areas for improvement calculated by the server, it generates specific advice and notifies the store operations manager.

[1670] (Example 2)

[1671] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1672] Traditional store management systems had limitations in analyzing customer behavior and collecting attribute data, making it difficult to identify repeat customers or analyze their sentiment. Furthermore, data analysis for optimizing in-store product placement and providing appropriate marketing for customer interests was insufficient. This made optimizing store operations and improving customer satisfaction difficult.

[1673] The identification processing by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for analyzing video data acquired from a monitoring device to detect individual people, means for recognizing the faces of the detected people and estimating their age and gender, means for identifying repeat customers by comparing them with an existing database based on specific features, means for analyzing the facial expressions of people in the video data using an emotion engine and recognizing emotion data, means for aggregating visitor attributes and generating reports based on the collected data, means for providing improvement advice to each store, means for analyzing customer movement data within the store and calculating the appropriate placement of products, means for notifying store managers of advice based on the calculated placement, means for collecting data on products that customers showed interest in but did not purchase, and means for evaluating the level of interest and analyzing it as marketing data. This enables efficient analysis of customer behavior data and emotion data, optimizing store operations and improving customer satisfaction.

[1674] A "surveillance device" is a device installed at the entrance or inside a store to acquire video data in real time.

[1675] "Video data" refers to digital data acquired from surveillance devices that records the movements and actions of customers inside and outside the store.

[1676] "Person detection" is the process of identifying each individual within video data and confirming their presence.

[1677] "Facial recognition" is a technology that analyzes the faces of detected individuals and identifies the unique characteristics of each person.

[1678] "Age estimation" is the process of calculating a person's age based on data obtained through facial recognition.

[1679] "Gender estimation" is the process of determining a person's gender based on data obtained through facial recognition.

[1680] "Specific features" refer to unique facial feature data extracted through facial recognition to identify individual people.

[1681] "Existing database" refers to a database where past visitor data is stored.

[1682] A "repeat customer" is a customer who has visited the store before and is returning to the store.

[1683] An "emotion engine" is software that analyzes a person's facial expressions in video data and recognizes emotional data.

[1684] "Attribute data" is a general term for various types of customer-related data, such as age, gender, visit history, and emotions.

[1685] "Report generation" is the process of creating a report that includes statistics and analysis results based on collected data.

[1686] "Improvement advice" means suggesting areas for improvement in store operations based on the results of data analysis.

[1687] "Customer flow data" refers to data that records the routes and movements of customers as they move within a store.

[1688] "Appropriate product placement" refers to a placement method that optimizes the location of products based on customer movement data.

[1689] "Interest level assessment" is a process of measuring the degree of interest a customer has in a particular product.

[1690] "Marketing data" refers to sales promotion-related data analyzed based on customer behavior and interests.

[1691] "Analysis tools" refer to software and algorithms used to analyze collected data and extract useful information.

[1692] This invention is a system that analyzes customer behavior, performs facial recognition, estimates attributes, and analyzes emotions in store operations, and uses this information to support improvements in store operations. The following describes how this system is specifically implemented.

[1693] Automatic counting of customer visitors

[1694] terminal

[1695] The terminal uses surveillance equipment to capture real-time images of the store's entrance and interior, acquiring video data. This terminal includes a camera and video processing hardware.

[1696] server

[1697] The server inputs the acquired video data into the AI ​​model "PersonDetect_v2" to detect people in the video.

[1698] The server uses "FaceRecog_v3" to perform facial recognition on detected individuals and estimate their age and gender.

[1699] The server compares facial features with the database. If there is a match, the data is updated to indicate a repeat customer; otherwise, the person is registered in the database as a new visitor.

[1700] The server uses the emotion engine "EmotionEngine_v1" to analyze the facial expressions of people in the video, recognize emotion data, and save it to a database.

[1701] Based on the collected data, visitor attributes are compiled, and daily and monthly reports are generated. These reports are stored in a database and notifications are sent.

[1702] Analysis of in-store movement and layout advice

[1703] terminal

[1704] The terminal uses multiple monitoring devices to capture real-time data on customer movement within the store and transmits it to the server.

[1705] server

[1706] The server analyzes the received movement data using the AI ​​model "MovTrack_v2" to track customer movements.

[1707] The server correlates customer movement data with product placement data to identify products of high interest.

[1708] The server uses the AI ​​model "RecommendConfig_v1" to calculate the optimal placement of products and notifies store managers with specific placement advice.

[1709] Analysis of interest in products that were not purchased.

[1710] terminal

[1711] The terminal extracts data from the video of products that customers picked up but did not purchase, and sends it to the server.

[1712] server

[1713] The server analyzes the video data to identify products that were not purchased.

[1714] The server associates user flow data with data on the target product and calculates an interest score using the "InterestRate_v1" model.

[1715] The server generates a marketing report based on the evaluation data and notifies the marketing team.

[1716] Observation of customer behavior by time of day and operational advice

[1717] terminal

[1718] The terminal captures video data of visitors at set intervals and sends it to the server.

[1719] server

[1720] The server organizes video data by time period and uses "TimeSlotAnalyzer_v1" to analyze customer behavior patterns.

[1721] The server uses data on visitor inflow and outflow, product popularity, and sales figures for specific time periods to generate an AI model that calculates areas for improvement in store operations.

[1722] The server notifies the store operations manager of advice based on the analysis results.

[1723] Specific example

[1724] Example of a prompt

[1725] The following are specific examples of each process.

[1726] Automatic counting of customer visitors

[1727] "Please generate an AI model program that estimates a customer's age and gender and determines whether they are a repeat customer or a new customer."

[1728] Analysis of in-store movement and layout advice

[1729] "Please generate a program for a system that analyzes customer movement data and product placement data within a store to propose the optimal product layout."

[1730] Analysis of interest in products that were not purchased.

[1731] "Please create a program that analyzes data on products customers picked up but did not purchase, and evaluates their level of interest."

[1732] Observation of customer behavior by time of day and operational advice

[1733] "Please generate a program for a system that observes customer behavior at different times of the day and provides advice for store operations."

[1734] As described above, the system of the present invention efficiently analyzes customer behavior data and emotional data to optimize store operations and improve customer satisfaction.

[1735] The flow of the specific processing in Example 2 will be explained using Figure 13.

[1736] Automatic counting of customer visitors

[1737] Step 1: Acquire video data

[1738] terminal

[1739] Using surveillance equipment, the store entrance and interior are photographed in real time, and video data is acquired.

[1740] Input: Video of the store entrance and interior.

[1741] Output: Video data

[1742] Send the video data to the server.

[1743] Step 2: Receiving video data and detecting people

[1744] server

[1745] The video data received at the time of posting (10 minutes ago) is input into the AI ​​model "PersonDetect_v2" to detect people in the video.

[1746] Input: Video data

[1747] Output: Detected person data

[1748] The detected person data is temporarily stored in memory.

[1749] Step 3: Face Recognition and Attribute Estimation

[1750] server

[1751] Using the detected person data, the face recognition algorithm "FaceRecog_v3" is executed to identify each person's face.

[1752] Input: Detected person data

[1753] Output: Face recognition data

[1754] Age and gender are estimated based on the results of facial recognition.

[1755] Input: Face recognition data

[1756] Output: Age and gender data

[1757] Step 4: Matching with the database

[1758] server

[1759] The estimated facial features are compared with data from an existing database.

[1760] Input: Age and gender data, facial feature data

[1761] Output: Determination result for whether the visitor is a repeat customer or a new visitor.

[1762] If a match is found, the person is recognized as a repeat customer and the database is updated. If there is no match, the person is registered in the database as a new visitor.

[1763] Step 5: Sentiment analysis and data storage

[1764] server

[1765] The emotion engine "EmotionEngine_v1" is used to analyze the facial expressions of people in video data and recognize emotional data.

[1766] Input: Face recognition data

[1767] Output: Sentiment data

[1768] The recognized emotion data is stored in the database as attribute data.

[1769] Based on the collected data, visitor attributes are compiled, and daily and monthly reports are generated.

[1770] Input: Attribute data

[1771] Output: Daily and monthly reports

[1772] Analysis of in-store movement and layout advice

[1773] Step 1: Obtaining movement data

[1774] terminal

[1775] Multiple monitoring devices are used to capture real-time data on customer movement within the store and transmit it to a server.

[1776] Input: Video of customer movement within the store

[1777] Output: Movement data

[1778] Step 2: Receiving and analyzing movement data

[1779] server

[1780] The received movement data is analyzed using the AI ​​model "MovTrack_v2" to track customer movements.

[1781] Input: Movement data

[1782] Output: Tracking data

[1783] Step 3: Identifying the level of interest in the product

[1784] server

[1785] By linking customer traffic data with product placement data, we can identify products of high interest.

[1786] Input: Tracking data, product placement data

[1787] Output: List of products of high interest

[1788] Step 4: Calculation and notification of the optimal placement

[1789] server

[1790] The optimal placement of products is calculated using the AI ​​model "RecommendConfig_v1".

[1791] Input: List of products of high interest

[1792] Output: Optimal product placement plan

[1793] The calculated layout plan will be provided to the store operations manager as advice.

[1794] Analysis of interest in products that were not purchased.

[1795] Step 1: Data acquisition for the product picked up

[1796] terminal

[1797] The system captures video data of products that customers picked up but did not purchase, and sends it to a server.

[1798] Input: Customer behavior video

[1799] Output: Data of the product picked up

[1800] Step 2: Receiving and analyzing product data

[1801] server

[1802] The received video data is analyzed to identify products that were not purchased.

[1803] Input: Data of the product picked up

[1804] Output: List of items that were not purchased

[1805] Step 3: Evaluate and notify of interest level

[1806] server

[1807] The system correlates customer movement data with data on unpurchased items and calculates an interest score using "InterestRate_v1".

[1808] Input: List of unpurchased items, customer flow data

[1809] Output: Interest score

[1810] A marketing report is generated based on the interest score, and the marketing team is notified.

[1811] Observation of customer behavior by time of day and operational advice

[1812] Step 1: Obtain customer data by time of day

[1813] terminal

[1814] Video data of visitors is captured at each time slot and sent to the server.

[1815] Input: Video footage by time period

[1816] Output: Customer data by time period

[1817] Step 2: Data analysis by time period

[1818] server

[1819] The video data received for each time period is organized, and customer behavior patterns are analyzed using the "TimeSlotAnalyzer_v1" model.

[1820] Input: Customer data by time period

[1821] Output: Behavioral patterns by time of day

[1822] Step 3: Calculation and notification of areas for operational improvement

[1823] server

[1824] Based on visitor inflow and outflow during specific time periods, product popularity, and sales data, an AI model calculates areas for improvement in store operations.

[1825] Input: Behavioral patterns by time of day, attention level data, sales data

[1826] Output: Points for improvement in operations

[1827] Notify the store operations manager of the areas for improvement.

[1828] The above outlines the processing steps and specific operations of this system's program. This enables efficient analysis of customer behavior and emotional data, leading to optimized store operations and improved customer satisfaction.

[1829] (Application Example 2)

[1830] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1831] The problem that this invention aims to solve is to efficiently monitor customer behavior and emotions in physical stores and improve customer satisfaction. By combining real-time customer behavior data and emotional data, the aim is to provide more appropriate services, improve store operations, and ultimately increase sales.

[1832] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[1833] In this invention, the server includes means for analyzing video data acquired from a monitoring device to detect individual people, means for recognizing the faces of detected people and estimating their age and gender, means for identifying repeat customers by comparing them with an existing database based on specific characteristics, means for aggregating visitor attributes based on collected data and generating reports, means for providing improvement advice to each store, means for analyzing customer emotion data and evaluating customer satisfaction, means for analyzing customer movement data within the store and calculating the optimal placement of products, means for collecting data on products that customers showed interest in but did not purchase and evaluating their level of interest, and means for analyzing customer behavior data by time of day and suggesting areas for improvement in store operations. This makes it possible to analyze customer behavior and emotions in real time and immediately implement measures to improve store operations.

[1834] A "surveillance device" refers to devices such as cameras and sensors that are installed to acquire video data.

[1835] "Video data" refers to data containing visual information acquired by surveillance equipment.

[1836] "Person detection means" refers to algorithms and technologies for identifying individual people from video data.

[1837] "Facial recognition means" refers to technology for identifying and distinguishing the faces of individuals detected from video data.

[1838] "Age estimation methods" refer to technologies that estimate the age of a person whose face has been recognized.

[1839] "Gender estimation means" refers to technology that estimates the gender of a person whose face has been recognized.

[1840] "Specific characteristics" refer to unique attributes such as facial features or behavioral patterns.

[1841] A "database" is an information system that stores information such as that of repeat customers.

[1842] A "repeat customer identification method" is a technology that determines whether a customer is a repeat customer by comparing them with an existing database based on specific characteristics.

[1843] "Visitor attributes" refer to characteristics such as the age, gender, and emotions of visitors.

[1844] A "report generation method" is a technology that compiles analysis results in report format based on collected data.

[1845] A "method for providing improvement advice" is a technology that notifies store managers of areas for improvement based on data analysis results.

[1846] "Emotional data" refers to data that represents the emotions estimated from the facial expressions of customers.

[1847] A "customer satisfaction evaluation method" is a technology that evaluates customer satisfaction based on emotional data.

[1848] "Customer movement data" refers to data that represents the paths customers take within a store.

[1849] The "optimal placement calculation method" is a technology that calculates the optimal placement of products based on traffic flow data.

[1850] "Products of interest" are products that customers showed interest in but did not purchase.

[1851] An "interest level evaluation method" is a technology that evaluates how interested a customer is in a product they are interested in.

[1852] "Customer behavior data by time of day" refers to data that represents customer behavior patterns during specific time periods.

[1853] "Store operation improvement methods" refer to technologies that analyze customer behavior data for each time period and identify areas for improvement in store operations.

[1854] The embodiments for carrying out this invention are described in detail below. This system analyzes video data acquired from a monitoring device to detect individual people, perform facial recognition, estimate attributes, and analyze emotion data, thereby understanding customer behavior and emotions in physical stores in real time, and aiming to optimize store operations and improve customer satisfaction.

[1855] System Configuration

[1856] This system consists of monitoring devices, a cloud server, and smart glasses or smartphones. The monitoring devices include standard surveillance cameras and sensors, while the cloud server implements AI models and emotion engines. The smart glasses and smartphones are terminals for store employees to receive and respond to information in real time.

[1857] Operation flow

[1858] 1. Automatic counting of customer visitors

[1859] server:

[1860] The server receives video data transmitted from the monitoring device and uses an AI model to detect people in the video. It recognizes the faces of the detected people and estimates their age and gender. For example, it might store them in the database as "a man in his 30s." Based on the facial features, it compares them with existing databases to determine whether they are repeat customers or new visitors. An emotion engine is used to analyze facial expressions and add emotional data such as "smiling" or "confused."

[1861] 2. Analysis of in-store movement and layout advice

[1862] server:

[1863] The system receives video data transmitted from multiple monitoring devices within the store and analyzes customer movement data in real time. Based on this, it correlates it with product placement data to identify which products are attracting the most attention. The server uses an AI model to calculate the optimal product placement based on the movement data and notifies store managers with specific advice. In addition, emotional data is integrated with the movement data to analyze customers' emotional reactions to products of interest and to the sales floor.

[1864] 3. Analysis of interest in products that were not purchased.

[1865] server:

[1866] Data on products that customers picked up but did not purchase is extracted from the video and sent to a server. The server analyzes this data to evaluate the customer's level of interest. It correlates the movement data with the data on the target products and calculates an interest score. This data is notified to marketing personnel and used for sales promotion activities. In addition, an emotion engine is used to evaluate the emotional response of customers to products they picked up but did not purchase.

[1867] 4. Observation of customer behavior by time of day and operational advice

[1868] server:

[1869] Monitoring devices capture video data of visitors at specific time intervals and transmit it to a server. The server organizes this data by time period and analyzes customer behavior patterns. Based on customer inflow and outflow, product attention, and sales data for specific time periods, an AI model calculates areas for improvement in store operations. This advice is notified and implemented by store managers. Sentimental data for each time period is also analyzed to evaluate customer satisfaction for each time slot.

[1870] Examples and prompts for generative AI models

[1871] For example, if customers tend to gather in a particular product section around 2 PM, and many of them smile while looking at the products but do not make a purchase, the system will determine that "they are highly interested in this product but are hesitant," and will either place related products nearby or notify staff to follow up.

[1872] Example of a prompt

[1873] Prompt example:

[1874] "Around 2 PM, customers tend to pick up product X but don't end up purchasing it. Customer sentiment data indicates they are interested but hesitant. Please propose effective marketing strategies to address this situation."

[1875] This system enables store managers to respond appropriately in real time, leading to improved customer satisfaction and increased sales.

[1876] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[1877] Step 1:

[1878] The surveillance system captures real-time video data of the store's entrance and interior.

[1879] Input: Video data acquired from a monitoring device.

[1880] Output: Sends a stream of video data to the cloud server.

[1881] Operation: The surveillance device's camera captures video data and sends it to a cloud server via the internet.

[1882] Step 2:

[1883] The server receives video data and uses an AI model to detect people in the video.

[1884] Input: Video data sent in Step 1.

[1885] Output: Location information of people detected in the video.

[1886] Operation: The AI ​​model analyzes the video data and executes an algorithm to identify the contours and locations of people.

[1887] Step 3:

[1888] The server recognizes the face of the detected person and estimates their age and gender.

[1889] Input: Location information of the person detected in Step 2.

[1890] Output: Recognized facial feature data, estimated age and sex data.

[1891] Operation: A facial recognition algorithm identifies a person's face and performs age and gender estimation.

[1892] Step 4:

[1893] The server matches facial features against an existing database to determine whether the person is a repeat customer or a new visitor.

[1894] Input: Facial feature data obtained in Step 3.

[1895] Output: Determination result of whether the visitor is a repeat customer or a new visitor.

[1896] Operation: Executes an algorithm to match feature data against a database and check if any matching data exists.

[1897] Step 5:

[1898] The server uses an emotion engine to analyze facial expressions and extract emotional data.

[1899] Input: Video data of the face recognized in Step 3.

[1900] Output: Extracted emotion data (e.g., smile, confusion, etc.).

[1901] Operation: The emotion engine analyzes facial features and classifies the emotional state.

[1902] Step 6:

[1903] The server collects visitor attributes and generates a report.

[1904] Input: Visitor data and sentiment data obtained in Steps 4 and 5.

[1905] Output: Summary report of visitor attributes.

[1906] Operation: Aggregates individual data points stored in the database and automatically generates daily and monthly reports.

[1907] Step 7:

[1908] A server receives video data transmitted from multiple monitoring devices within the store and analyzes customer movement data in real time.

[1909] Input: Video data stream from the monitoring device.

[1910] Output: Customer movement data.

[1911] Operation: Video data obtained from multiple cameras is analyzed using a tracking algorithm to identify customer movement patterns.

[1912] Step 8:

[1913] The server calculates the optimal placement of products based on customer movement data.

[1914] Input: Movement data obtained in Step 7.

[1915] Output: Optimal placement plan for products.

[1916] Operation: Integrates customer flow data with product placement data and calculates the optimal product placement using an AI model.

[1917] Step 9:

[1918] The server collects data on products that were not purchased and evaluates the level of interest.

[1919] Input: Video data and movement data from monitoring devices.

[1920] Output: Interest score for products that were not purchased.

[1921] Operation: The system determines whether a user picked up a specific product from the video, and if they did not purchase it, it analyzes their behavior and emotional data to evaluate their level of interest.

[1922] Step 10:

[1923] The server analyzes customer behavior data for each time period and suggests areas for improvement in store operations.

[1924] Inputs: Video data by time of day, sales data, customer flow data, and sentiment data.

[1925] Output: Customer behavior report by time of day and operational improvement advice.

[1926] Function: It organizes and analyzes data for each time period, evaluates customer behavior patterns and emotions during specific time periods, and generates operational improvement advice based on that.

[1927] Through the above processing steps, it is possible to analyze customer behavior and emotions in physical stores in detail and achieve optimal store operations in real time.

[1928] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[1929] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include those described above. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions shown by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1930] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[1931] [Fourth Embodiment]

[1932] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[1933] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1934] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1935] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[1936] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[1937] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[1938] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[1939] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[1940] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[1941] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1942] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1943] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[1944] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1945] This invention relates to a system that analyzes video data acquired from a surveillance device, detects individual people, performs facial recognition and attribute estimation, and analyzes customer behavior based on the results. The program and processing of this system are described in detail below.

[1946] Explain the program's processing in natural language.

[1947] 1. Automatic counting of customer visitors

[1948] Surveillance devices are installed at the store entrance and inside the store to capture video of visitors in real time. The video data is sent to a server, which uses an AI model to detect people in the video. It performs facial recognition on each person and estimates their age and gender. This information is temporarily stored in a database to determine if a customer is a repeat customer. Specifically, the server compares the facial features with an existing database, and if there is a match, the person is registered as a repeat customer. If there is no match, the person is registered as a new visitor in the database. Finally, visitor attribute data is compiled, and daily and monthly reports are generated. These reports are notified to the store operations manager.

[1949] Specific example

[1950] server

[1951] The server receives the video data and uses an AI model to detect people.

[1952] The server performs facial recognition and estimates age and gender. For example, it might save the information in the database as "male in his 30s."

[1953] The server matches the facial features against the database, counts the customer as a "repeat customer," and updates the database.

[1954] 2. Analysis of in-store movement and layout advice

[1955] Multiple monitoring devices within the store capture customer movement data in real time and transmit it to a server. The server analyzes this data to track customer movements. It correlates the movement data with product placement data to identify which products are attracting the most interest. Using an AI model, the server calculates the optimal product placement based on the movement data and notifies store managers with specific advice.

[1956] Specific example

[1957] server

[1958] The server analyzes customer movement patterns and determines that "Customer A has shown interest in product X."

[1959] Based on the customer flow data, the server advises and notifies the user to place product Y near the entrance.

[1960] 3. Analysis of interest in products that were not purchased.

[1961] Data on products that customers picked up but did not purchase is extracted from the video and sent to a server. The server analyzes this data to evaluate the customer's level of interest. Specifically, it correlates the customer's movement with the data on the target products and calculates an interest score. This data is then notified to marketing personnel and used for sales promotion activities.

[1962] Specific example

[1963] server

[1964] The server identifies product B, which the user picked up but did not purchase, and rates its level of interest as "high".

[1965] The server generates a marketing report based on the evaluation data and notifies the client of the marketing measures that should be adopted.

[1966] 4. Observation of customer behavior by time of day and operational advice

[1967] Monitoring devices capture video data of visitors at specific time intervals and transmit it to a server. The server organizes this data by time period and analyzes customer behavior patterns. Based on customer inflow and outflow, product attention levels, and sales data for specific time periods, an AI model calculates areas for improvement in store operations. This advice is communicated to store managers for their implementation.

[1968] Specific example

[1969] server

[1970] The server analyzes customer data by time of day and identifies that "customer traffic increases around 2 PM."

[1971] Based on the analysis results, the server will notify you of the advice to "deploy additional staff by 2 PM."

[1972] The above describes the operation and specific examples of the system in an embodiment for carrying out the present invention. This makes it possible to efficiently analyze customer behavior data and optimize store operations.

[1973] The following describes the processing flow.

[1974] Program processing flow

[1975] Automatic counting of customer visitors

[1976] Step 1:

[1977] The terminal acquires video data from the monitoring device and transmits it to the server in real time.

[1978] Step 2:

[1979] The server analyzes the received video data using an AI model to detect people in the video.

[1980] Step 3:

[1981] The server performs facial recognition on detected individuals and estimates their age and gender.

[1982] Step 4:

[1983] The server temporarily stores the estimated facial data and compares it with an existing database based on specific features.

[1984] Step 5:

[1985] The server identifies repeat customers and new visitors based on the results of matching them with the database, and updates and registers their respective data.

[1986] Step 6:

[1987] The server aggregates visitor attribute data, generates daily and monthly reports, and notifies store operations staff.

[1988] Analysis of in-store movement and layout advice

[1989] Step 1:

[1990] The terminal acquires video data from multiple monitoring devices within the store and transmits it to the server in real time.

[1991] Step 2:

[1992] The server analyzes the received video data using an AI model to track customer movement.

[1993] Step 3:

[1994] The server correlates customer movement data with product placement data to identify which products customers showed interest in.

[1995] Step 4:

[1996] The server calculates the optimal placement of products based on customer movement data.

[1997] Step 5:

[1998] Based on the calculated server placement, the server generates specific advice and notifies the store operations manager.

[1999] Analysis of interest in products that were not purchased.

[2000] Step 1:

[2001] The terminal acquires video data from the monitoring device and transmits it to the server in real time.

[2002] Step 2:

[2003] The server analyzes the video data to identify products that customers picked up but did not purchase.

[2004] Step 3:

[2005] The server tracks customer movement and evaluates their level of interest in products they did not purchase.

[2006] Step 4:

[2007] The server stores interest data in a database and organizes it as marketing data.

[2008] Step 5:

[2009] The server generates marketing reports and notifies marketing personnel.

[2010] Observation of customer behavior by time of day and operational advice

[2011] Step 1:

[2012] The terminal captures video data from the monitoring device, categorized by time period, and transmits it to the server in real time.

[2013] Step 2:

[2014] The server organizes the video data received by time period and compiles data on customer inflow and outflow.

[2015] Step 3:

[2016] The server analyzes data for each time period to identify customer behavior patterns.

[2017] Step 4:

[2018] Based on the data analyzed by the server, an AI model calculates areas for improvement in operations.

[2019] Step 5:

[2020] Based on the areas for improvement calculated by the server, it generates specific advice and notifies the store operations manager.

[2021] (Example 1)

[2022] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[2023] Traditional systems made it difficult to effectively collect and analyze customer behavior data from store visitors and propose improvements to store operations. Furthermore, strategic marketing measures based on individual customer attributes and interests were not adequately implemented. In particular, identifying repeat customers, optimizing product placement, and evaluating interest in products that were not purchased were challenging, resulting in a lack of concrete, data-driven advice for efficient store operations and increased sales.

[2024] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[2025] In this invention, the server includes means for analyzing video data acquired from a monitoring device to detect individual people, means for recognizing the faces of detected people and estimating their age and gender, means for identifying repeat customers by comparing them with an existing database based on characteristic information, means for aggregating visitor attributes based on collected data and generating reports, means for providing improvement advice to each store, means for analyzing customer movement data to calculate the optimal placement of products and notifying each store, means for collecting data on products that customers picked up but did not purchase and evaluating their level of interest, means for acquiring video data from installed monitoring devices at different times of the day and analyzing customer behavior patterns, and means for calculating and notifying improvements to store operations by correlating the analysis results with sales data. This makes it possible to efficiently collect and analyze customer behavior data and optimize store operations.

[2026] A "surveillance device" is a device installed at the entrance or inside a store to acquire video data.

[2027] "Video data" refers to image and video information acquired by surveillance equipment.

[2028] "Analysis" refers to techniques that process information such as video data to detect individual people or analyze movement patterns.

[2029] "Person detection" is a method of identifying individual people from video data.

[2030] "Facial recognition" is a method of identifying a specific person by analyzing the face of a detected individual.

[2031] "Age and gender estimation" refers to a method of inferring a person's age group and gender based on person detection and facial recognition.

[2032] "Characteristic information" refers to information used to identify individual people, such as facial features and movement data.

[2033] A "database" is a system for organizing and storing collected data.

[2034] A "repeat customer" refers to a person who has visited the store before and is recorded in the existing database.

[2035] "Visitor attributes" refer to characteristics such as visitor age, gender, movement patterns, and purchase history.

[2036] "Report generation" refers to the process of compiling analysis results based on collected data and creating a report.

[2037] "Improvement advice" is a method of providing instructions on areas for improvement in store operations based on collected and analyzed data.

[2038] "Customer movement data" refers to data that shows the routes and behavioral patterns of customers as they move within a store.

[2039] "Product placement" refers to the location and method of displaying products within a store.

[2040] "Interest level" is a measure that evaluates the degree of interest a customer has in a product they picked up but did not purchase.

[2041] "Video data by time period" refers to video data acquired during a specific time period.

[2042] "Behavioral patterns" refer to data that shows the influx and outflow of customers and the trends of popular products during specific time periods.

[2043] This invention relates to a system that analyzes video data acquired from a surveillance device, detects individual people, performs facial recognition and attribute estimation, and analyzes customer behavior based on the results. The program and processing of this system are described below.

[2044] This system consists of monitoring devices, a server, and terminals. The monitoring devices are installed at the entrance and inside the store and acquire customer video data in real time. The acquired video data is transmitted to the server via the terminals.

[2045] The server uses an AI model to analyze the received video data. The AI ​​model utilizes generative AI technology to detect people in the video with high accuracy. At this stage, the server uses a facial recognition algorithm to identify the faces of detected individuals and estimate their age and gender. For example, it might store this information in a database as "a man in his 30s." The software used for this includes common facial recognition libraries and attribute estimation algorithms.

[2046] Furthermore, the server matches facial features against an existing database, counting repeat customers if a match is found. If there is no match, the customer is registered in the database as a new visitor. Finally, visitor attribute data is aggregated, and daily and monthly reports are generated. These reports are then communicated to store operations staff.

[2047] Next, the server analyzes customer movement data transmitted from multiple monitoring devices within the store to track customer behavior. It then correlates this movement data with product placement data to identify which products are attracting the most attention. Using an AI model, the server calculates the optimal product placement based on the movement data and provides specific advice to store managers.

[2048] The server also analyzes data on products that customers picked up but did not purchase, and evaluates the customer's level of interest. Specifically, it correlates customer movement with data on the target products and calculates an interest score. This data is then shared with marketing personnel and used for sales promotion activities.

[2049] Furthermore, monitoring devices capture video data of visitors at specific time intervals and transmit it to a server. The server organizes this data by time period and analyzes customer behavior patterns. Based on customer inflow and outflow, product attention levels, and sales data for specific time periods, an AI model calculates areas for improvement in store operations. This advice is then communicated to store managers for implementation.

[2050] For example, the server might analyze that "customer traffic increased during the 2 PM timeframe" and notify the store manager of the need to allocate additional staff.

[2051] Example of a prompt

[2052] "Analyze customer movement data recorded by security cameras to identify which products are attracting the most attention. Based on these results, propose the optimal product placement and notify the store operations manager."

[2053] The above outlines a specific implementation of this system. This makes it possible to efficiently analyze customer behavior data and optimize store operations.

[2054] The flow of the specific processing in Example 1 will be explained using Figure 11.

[2055] Step 1:

[2056] The terminal acquires video data in real time from surveillance devices installed at the entrance and inside the store.

[2057] Specific operation: The monitoring device captures video data using a camera sensor, and the terminal receives this data.

[2058] Input: Video data from a monitoring device.

[2059] Output: Video data received by the terminal.

[2060] Step 2:

[2061] The device sends the acquired video data to the server.

[2062] Specific operation: The device uploads the received video data to the server via the network.

[2063] Input: Video data received by the device.

[2064] Output: Video data received by the server.

[2065] Step 3:

[2066] The server inputs the received video data into an AI model to detect people in the video.

[2067] Specific operation: The server inputs video data into the AI ​​model, and the model detects people in each video frame.

[2068] Input: Video data received by the server.

[2069] Output: Person detection results by the AI ​​model.

[2070] Step 4:

[2071] The server performs facial recognition on detected individuals and estimates their age and gender.

[2072] Specific operation: The server uses a facial recognition algorithm to analyze the detected face and estimate its age and gender. For example, it might save it in the database as "male in his 30s."

[2073] Input: Person detection results from an AI model.

[2074] Output: Person data with estimated age and gender.

[2075] Step 5:

[2076] The server matches the facial features against an existing database. If there is a match, the person is counted as a repeat customer. If there is no match, they are registered in the database as a new visitor.

[2077] Specific operation: The server compares the identified facial feature data with the database to determine whether it is a new or repeat customer.

[2078] Input: Individual data with estimated age and gender, and existing databases.

[2079] Output: Repeat customer list, New visitor list.

[2080] Step 6:

[2081] The server aggregates visitor attribute data and generates daily and monthly reports.

[2082] Specific operation: The server aggregates data on new and repeat customers, and a report generation program creates daily and monthly reports.

[2083] Input: Repeat customer list, New visitor list.

[2084] Output: Daily and monthly reports.

[2085] Step 7:

[2086] The server notifies the store operations manager of the report.

[2087] Specific operation: The server sends the generated report to the responsible person via email or dashboard.

[2088] Input: Daily and monthly reports.

[2089] Output: Report sent to store operations staff.

[2090] Step 8:

[2091] The terminal acquires customer movement data in real time from multiple monitoring devices within the store.

[2092] Specific operation: The monitoring device tracks customer movements within the store and captures movement data in real time.

[2093] Input: Customer traffic data from monitoring devices within the store.

[2094] Output: Movement data received by the terminal.

[2095] Step 9:

[2096] The terminal sends the movement data it acquires to the server.

[2097] Specific operation: The terminal receives movement data and sends it to the server via the network.

[2098] Input: Movement data received by the terminal.

[2099] Output: Movement data received by the server.

[2100] Step 10:

[2101] The server analyzes the movement data it receives to track customer movements.

[2102] Specific operation: The server processes the movement data using an analysis algorithm to analyze customer movement patterns.

[2103] Input: Movement data received by the server.

[2104] Output: Customer movement analysis results.

[2105] Step 11:

[2106] The server correlates customer flow data with product placement data to identify which products are attracting the most attention.

[2107] Specific operation: The server matches customer flow data with product placement data to identify products of high interest.

[2108] Input: Customer movement analysis results, product placement data.

[2109] Output: A list of products the customer has shown interest in.

[2110] Step 12:

[2111] The server calculates the optimal placement of products and notifies store managers with specific advice.

[2112] Specific operation: The server uses an optimal placement algorithm to generate product placement advice and notifies the person in charge.

[2113] Input: A list of products that the customer has shown interest in.

[2114] Output: Placement advice report.

[2115] Step 13:

[2116] The terminal extracts data from in-store monitoring devices about items that customers picked up but did not purchase, and sends it to a server.

[2117] Specific operation: The monitoring device tracks customer behavior, collects data on items picked up but not purchased, and transmits it to the terminal.

[2118] Input: Data on products that were not purchased.

[2119] Output: Data on unpurchased items received by the server.

[2120] Step 14:

[2121] The server analyzes data on unpurchased items to assess the customer's level of interest.

[2122] Specific operation: The server analyzes data on unpurchased items and calculates an interest score.

[2123] Input: Data on unpurchased items received by the server.

[2124] Output: A list of products with interest scores.

[2125] Step 15:

[2126] The server notifies marketing personnel of interest levels, who then use the data for sales promotion activities.

[2127] Specific operation: The server creates a marketing report based on interest level data and sends it to the person in charge.

[2128] Input: A list of products with interest scores.

[2129] Output: Marketing report.

[2130] Step 16:

[2131] The terminal acquires video data of visitors from the monitoring device at specific time intervals and sends it to the server.

[2132] Specific operation: The device captures video data at specified time intervals and sends it to the server.

[2133] Input: Time-series video data from monitoring equipment.

[2134] Output: Video data received by the server for each time period.

[2135] Step 17:

[2136] The server organizes video data by time period and analyzes customer behavior patterns.

[2137] Specific operation: The server collects data for each time period and analyzes behavioral patterns.

[2138] Input: Video data received by the server for each time period.

[2139] Output: Customer behavior patterns by time of day.

[2140] Step 18:

[2141] The server correlates the analysis results with sales data to calculate areas for improvement in store operations.

[2142] Specific operation: The server analyzes the correlation between behavioral patterns and sales data to identify areas for improvement.

[2143] Input: Customer behavior patterns by time of day, sales data.

[2144] Output: A report identifying areas for improvement.

[2145] Step 19:

[2146] The server notifies the store operations manager with improvement advice.

[2147] Specific action: The server sends a report indicating areas for improvement to the responsible person and notifies them.

[2148] Input: A report identifying areas for improvement.

[2149] Output: Improvement advice notified to store operations staff.

[2150] (Application Example 1)

[2151] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[2152] Traditional store operations have made it difficult to adequately analyze customer behavior data, particularly regarding customer movement patterns, product placement, and analysis of unpurchased items, without utilizing real-time data. Furthermore, there was a lack of means to observe customer behavior at different times of the day and to obtain concrete suggestions for operational improvements based on that observation. As a result, optimizing store operations was hindered, making it difficult to contribute to increased sales and improved customer satisfaction.

[2153] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[2154] In this invention, the server includes means for analyzing image data acquired from a monitoring device to detect individual people, means for recognizing the faces of detected people and estimating their age and gender, means for identifying repeat customers by comparing them with an existing information base based on specific features, means for aggregating visitor attributes based on collected data and generating reports, means for providing improvement suggestions to each facility, means for analyzing customer movement data and calculating the appropriate placement of products, means for notifying facility operators of the suggestions based on the calculated placement, means for acquiring data on products that customers showed interest in but did not purchase from collected image data, means for evaluating the level of interest and analyzing it as marketing data, means for organizing visitor data by time of day and analyzing customer behavior patterns, and means for calculating points for operational improvement in each time period and notifying operators. This makes it possible to efficiently collect and analyze customer behavior data within stores and to make operational improvement suggestions in real time.

[2155] A "surveillance device" refers to a device that acquires video data and performs tasks such as person detection and movement analysis.

[2156] "Image data" refers to video information acquired from surveillance devices and is used for person detection and movement analysis.

[2157] "Person detection" refers to a method of identifying individual people from image data.

[2158] "Facial recognition" refers to a method of identifying a person by analyzing the facial features of a detected individual.

[2159] "Age and gender estimation" refers to the act of inferring a person's age and gender from the results of facial recognition.

[2160] "Specific features" refer to facial characteristic information obtained through facial recognition, and are data used to compare this information with existing information bases.

[2161] "Existing information base" refers to a database containing customer data collected previously.

[2162] A "repeat customer" refers to a customer who has a history of visiting the store in the past and has been determined to be a returning customer based on matching specific characteristics.

[2163] "Collected data" refers to all data obtained from monitoring devices and analysis means.

[2164] "Visitor attributes" refer to characteristics such as the age, gender, and number of visits of customers who visit the store.

[2165] "Report generation" refers to the act of compiling collected data and creating a document that summarizes it in a specific format.

[2166] "Each facility" refers to the stores or business locations that are the subject of data analysis.

[2167] An "improvement suggestion" is a proposal that outlines specific measures to improve store operations based on collected data.

[2168] "Customer movement data" refers to information that shows the paths customers take within a store.

[2169] "Appropriate product placement" refers to advice on optimizing product placement based on customer traffic data.

[2170] A "facility operations manager" refers to a manager responsible for the operation of a store or business location.

[2171] "Data acquisition" refers to the act of extracting necessary information from image data.

[2172] "Interest level assessment" is the act of analyzing and evaluating the level of interest shown by customers.

[2173] "Marketing data" refers to data about customer interests and behavior extracted from collected data, and is used in marketing strategies.

[2174] "Visitor data by time slot" refers to customer visit information organized by specific time periods.

[2175] "Customer behavior patterns" refer to pattern information that shows how customers move around within a store and what actions they take.

[2176] "Areas for operational improvement" refers to the points necessary to improve store operations based on collected data.

[2177] This invention relates to a system that analyzes image data acquired from monitoring devices to track and analyze customer behavior in real time. This system collects customer attribute data, movement data, and purchasing behavior information, contributing to the optimization of facility operations. Specific embodiments of this system are described below.

[2178] The server uses multiple AI models to process image data. For example, it uses OpenCV to acquire image data and a FaceRecognitionModel to recognize and analyze people's faces. It also uses an ActionDetectionModel to analyze customer movement and behavior. Furthermore, it uses a SpatialAnalysisModel to analyze the collected movement data and calculate the appropriate placement of products. All of this data is then centrally managed and stored by DatabaseConnection.

[2179] Regarding the hardware and software to be used

[2180] Hardware: Surveillance equipment (cameras), servers, database servers, smartphones, head-mounted displays

[2181] Software: OpenCV, FaceRecognitionModel, ActionDetectionModel, SpatialAnalysisModel, DatabaseConnection

[2182] The server first acquires video data from the monitoring device. Next, it analyzes the video frames using OpenCV to detect faces. It extracts the facial features detected using FaceRecognitionModel and estimates age and gender based on that data. It also compares specific features with an existing information base to determine whether the person is a repeat customer or a new visitor. This information is stored in a database via DatabaseConnection and compiled into a report.

[2183] Next, the ActionDetectionModel is used to analyze customer movement data. For each frame, customer movement is analyzed and movement data is collected. This data is then analyzed by the SpatialAnalysisModel and used to optimize product placement. Based on these analysis results, the server provides improvement advice to facility managers.

[2184] Data on products that customers picked up but did not purchase is collected using the ActionDetectionModel, and their level of interest is evaluated. This evaluation is analyzed as marketing data, and specific marketing strategies are formulated based on the collected results.

[2185] Specific example

[2186] For example, if a customer shows interest in a particular product but doesn't purchase it, the server analyzes their behavior and evaluates whether other customers tend to be interested in similar products. Also, if there's a sudden surge in customer traffic during a specific time period, the server analyzes the cause and provides specific suggestions for operational improvements. This allows operations staff to make evidence-based decisions.

[2187] An example of a prompt for a generated AI model is: "Recognize customer faces from video footage obtained from surveillance cameras, estimate their age and gender, and determine if they are repeat customers. Also, analyze customer movement patterns and provide advice on the optimal placement of products."

[2188] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[2189] Step 1:

[2190] The server receives video data in real time from the monitoring device. This video data records the actions of customers from the moment they enter the store until they leave. Specifically, the monitoring device continuously captures frame images from cameras inside the store and sends this data to the server. The input is video data, and the output is video information stored on the server.

[2191] Step 2:

[2192] The server analyzes the received video data using OpenCV to detect the faces of individual customers. The input is the video data received in step 1, and the output is the coordinate information of the detected faces. Specifically, the server applies a face detection algorithm to each frame to identify the face region within the image.

[2193] Step 3:

[2194] The server extracts facial features detected using a FaceRecognitionModel and estimates age and sex. The input is the coordinate information of the face detected in step 2, and the output is the estimated age and sex information. Specifically, it applies a facial feature extraction algorithm and performs estimation using the age and sex model.

[2195] Step 4:

[2196] The server determines whether a customer is a repeat customer or a new visitor by comparing specific features with an existing information base. The input is the facial features extracted in step 3, and the output is the repeat customer determination result. Specifically, the server compares the features with existing data in the information base, calculates the degree of match, and makes the determination.

[2197] Step 5:

[2198] The server aggregates visitor attributes based on the collected data and generates a report. The input is all output data from steps 1 through 4, and the output is the aggregated report. Specifically, it applies a data analysis algorithm to create a report document in a specific format.

[2199] Step 6:

[2200] The server analyzes customer movement data using an ActionDetectionModel. The input is the video data received in step 1, and the output is the movement data. Specifically, it tracks the customer's position in each frame and stores the path information.

[2201] Step 7:

[2202] The server analyzes the movement data using SpatialAnalysisModel to calculate the appropriate placement of products. The input is the movement data analyzed in step 6, and the output is a specific proposal for the optimal product placement. Specifically, it applies an optimization algorithm based on the movement patterns and customer interests.

[2203] Step 8:

[2204] The server notifies the facility manager with advice based on the calculated layout. The input is the layout plan generated in step 7, and the output is the notified advice. Specifically, the server sends a recommendation for layout changes to the manager via the notification system.

[2205] Step 9:

[2206] The server retrieves data on products that customers showed interest in but did not purchase, from the collected image data. The input is the video data received in step 1, and the output is the data on products of interest. Specifically, it analyzes the customer's actions of picking up and putting back products and extracts that data.

[2207] Step 10:

[2208] The server evaluates the level of interest and analyzes it as marketing data. The input is the interest product data obtained in step 9, and the output is the level of interest evaluation data. Specifically, it calculates the level of interest score, analyzes the results, and organizes them as marketing data.

[2209] Step 11:

[2210] The server organizes visitor data by time period and analyzes customer behavior patterns. Input is the video data and all analysis results received in step 1, and output is the behavior pattern analysis results. Specifically, it divides the data by time period and analyzes customer inflow patterns.

[2211] Step 12:

[2212] The server calculates areas for operational improvement for each time period and notifies the operations staff. The input is the behavioral pattern data analyzed in step 11, and the output is the operational improvement suggestions that are notified. Specifically, it calculates a particular improvement action based on the analysis results and notifies the person in charge.

[2213] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[2214] This invention is a system that analyzes video data acquired from a monitoring device, detects individual people, performs facial recognition and attribute estimation, and analyzes customer behavior based on the results. Furthermore, by combining it with an emotion engine, it also analyzes visitor emotional data to evaluate customer satisfaction and emotional responses to products of interest. A specific embodiment of this system is described below.

[2215] Explain the program's processing in natural language.

[2216] 1. Automatic counting of customer visitors

[2217] Surveillance devices are installed at the store entrance and inside the store to capture video of visitors in real time. The video data is sent to a server, which uses an AI model to detect people in the video. It performs facial recognition on each person and estimates their age and gender. This information is temporarily stored in a database to determine if the person is a repeat customer. Specifically, the server compares the facial features with an existing database, and if there is a match, the person is registered as a repeat customer. If there is no match, the person is registered as a new visitor in the database. Finally, visitor attribute data is compiled, and daily and monthly reports are generated. These reports are notified to the store operations manager. In addition, the server uses an emotion engine to analyze the facial expressions of people in the video data and recognize the customer's emotions. The recognized emotion data is also stored as attribute data.

[2218] Specific example

[2219] server

[2220] The server receives the video data and uses an AI model to detect people.

[2221] The server performs facial recognition and estimates age and gender. For example, it might save the information in the database as "male in his 30s."

[2222] The server matches the facial features against the database, counts the customer as a "repeat customer," and updates the database.

[2223] The server uses an emotion engine to recognize emotions such as "smile" or "confusion" and store them in a database.

[2224] 2. Analysis of in-store movement and layout advice

[2225] Multiple monitoring devices within the store capture customer movement data in real time and transmit it to a server. The server analyzes this data to track customer movements. It correlates the movement data with product placement data to identify which products are attracting the most interest. Using an AI model, the server calculates the optimal product placement based on the movement data and notifies store managers with specific advice. It also integrates emotional data with the movement data to analyze customers' emotional responses to products and store layouts.

[2226] Specific example

[2227] server

[2228] The server analyzes customer movement patterns and determines that "Customer A has shown interest in product X."

[2229] Based on the customer flow data, the server advises and notifies the user to place product Y near the entrance.

[2230] The server integrates emotional data and evaluates, for example, "Customer A was pleased when they saw product X."

[2231] 3. Analysis of interest in products that were not purchased.

[2232] Data on products that customers picked up but did not purchase is extracted from the video and sent to a server. The server analyzes this data to evaluate the customer's level of interest. Specifically, it correlates the customer's movement with the data of the target products and calculates an interest score. This data is notified to marketing personnel and used for sales promotion activities. In addition, an emotion engine is used to evaluate the emotional response of customers to products they picked up but did not purchase.

[2233] Specific example

[2234] server

[2235] The server identifies product B, which the user picked up but did not purchase, and rates its level of interest as "high".

[2236] The server generates a marketing report based on the evaluation data and notifies the client of the marketing measures that should be adopted.

[2237] The server saves the emotional data to the database, indicating "interested but undecided."

[2238] 4. Observation of customer behavior by time of day and operational advice

[2239] Monitoring devices capture video data of visitors at specific time intervals and transmit it to a server. The server organizes this data by time period and analyzes customer behavior patterns. Based on customer inflow and outflow, product attention, and sales data for specific time periods, an AI model calculates areas for improvement in store operations. This advice is notified and implemented by store managers. Sentimental data for each time period is also analyzed to evaluate customer satisfaction for each time slot.

[2240] Specific example

[2241] server

[2242] The server analyzes customer data by time of day and identifies that "customer traffic increases around 2 PM."

[2243] Based on the analysis results, the server will notify you of the advice to "deploy additional staff by 2 PM."

[2244] The server analyzes sentiment data for each time period and evaluates that "customer satisfaction is high at 2 PM."

[2245] The above describes the operation and specific examples...

Claims

1. A means of analyzing video data acquired from a surveillance device to detect individual people, A means for recognizing the face of a detected person and estimating their age and gender, A method for identifying repeat customers by matching them with an existing database based on specific characteristics, A means to aggregate visitor attributes and generate reports based on the collected data, A system that includes means of providing improvement advice to each store.

2. In the system described in claim 1, A method for analyzing customer movement data within a store and calculating the appropriate placement of products, The system further includes a means of notifying store managers of advice based on the calculated layout.

3. In the system described in claim 1, A means of collecting data on products that customers showed interest in but did not purchase, A system that further includes means to evaluate the level of interest and analyze it as marketing data.

Images