System

Abstract

A system is provided.SOLUTION: A system including a means for acquiring video data from a camera disposed in a store, a means for analyzing the acquired video data to detect a user's action and facial expression, a means for tracking a stay time, a position, and a line of sight of the user to learn a behavior pattern, a means for determining a user exhibiting a specific behavior pattern as a purchase contemplation user, and a means for transmitting a notification to a salesperson when the purchase contemplation user is specified.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In modern retail stores, it is important to understand the trends customers experience before purchasing a product and provide support at the appropriate time. However, conventional systems have difficulty analyzing customer behavior in the store in real time and identifying customers with purchasing intent. This prevents sales staff from providing support at the appropriate time, resulting in lost sales opportunities and reduced customer satisfaction. Furthermore, optimization of store layout based on customer behavior data is insufficient, hindering efficient store operations. [Means for solving the problem]

[0005] To solve the above problems, the present invention provides a system including: means for acquiring video data from cameras installed in a store; means for analyzing the acquired video data to detect user movements and facial expressions; means for tracking user stay time, location, and flow to learn behavioral patterns; means for determining users exhibiting specific behavioral patterns as potential purchase users; and means for sending notifications to sales staff when potential purchase users are identified. This system enables sales staff to provide support to customers in a timely manner and maximize sales opportunities. Furthermore, efficient store management is enabled by generating proposals for optimizing store layout and display based on accumulated customer data.

[0006] A "camera" is a device for acquiring video data in real time and for recording and transmitting images of a particular object or area.

[0007] "Video data" refers to the digital data format of images or videos captured by a camera.

[0008] "Analysis" refers to the process of processing acquired video data to extract or identify specific information.

[0009] "User movements and expressions" refers to the physical movements and facial expressions of customers that the system recognizes, and are particularly indicators of whether or not they are interested in purchasing.

[0010] "Dwell time" refers to the length of time a user stays in a particular location.

[0011] "Location" is information indicating that the user is at a specific location within the store.

[0012] "Path" refers to the route or pattern that a user follows within a store.

[0013] A "behavioral pattern" is a pattern derived by analyzing the series of behavioral trends and specific actions that a user exhibits in a store.

[0014] A "user considering purchase" is a user who, based on the analysis results, is determined to have a strong interest in a particular product and is considering purchasing it.

[0015] A "notification" is an action in which a system sends pre-defined information to a specific recipient when a specific state or event is triggered.

[0016] A "sales associate" is a person who sells products in a store and provides support and advice to customers.

[0017] "Layout" refers to the physical structure of a store, such as the arrangement of products and aisles.

[0018] "Display" refers to the way products are arranged in a store so that they are visually visible to customers.

[0019] "Suggestions" refer to improvement and optimization suggestions that the system provides to store managers based on the analysis results. [Brief explanation of the drawings]

[0020] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7]FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0021] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0022] First, the terms used in the following description will be explained.

[0023] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0024] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0025] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0026] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0027] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0028] [First embodiment]

[0029] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0030] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0031] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0032] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0033] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0034] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0035] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0036] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0037] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0038] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0039] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0040] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0041] The present invention provides a system that analyzes the behavior of users (customers) in a store in real time and sends a notification to sales staff at an appropriate time when it is determined that the user has a purchase intention.The system also makes suggestions for optimizing store layout and product displays based on the accumulated customer behavior data.Detailed embodiments of the present invention are described below.

[0042] Camera image acquisition and video data analysis

[0043] The device collects video data in real time from multiple cameras in the store. This video data is sent from cameras positioned to cover specific areas and shelves in the store. The server receives this video data and analyzes it in real time.

[0044] Detecting user movements and facial expressions

[0045] The server analyzes the user's movements and facial expressions based on the received video data. Specifically, it uses facial recognition technology and object detection algorithms to determine where the user is and what products they are interested in. This analysis can be performed using open-source computer vision libraries (e.g., OpenCV) and machine learning frameworks (e.g., TENSORFLOW®).

[0046] Tracking time spent, location, and path, and learning behavioral patterns

[0047] The server tracks users' time spent, location, and route in real time. This allows it to determine how long a user stays at a particular shelf and the path they take. This data is used to learn user behavior patterns. For example, if a user spends a long time in a certain area, it can be determined that they are interested in that area.

[0048] Identifying and notifying potential purchasers

[0049] Based on the user's behavioral patterns, the server identifies users who are deemed to have a high intent to purchase as "purchase-consideration users." This identification is performed using behavioral analysis algorithms and machine learning models (e.g., random forests and support vector machines). When a user is identified as a purchase-consideration user, the server immediately sends a notification to the salesperson via the terminal. This notification includes information about the user's current location and the products they are interested in.

[0050] Salesperson's response

[0051] The salesperson, who is the user, receives a notification on their device. Upon receiving the notification, the salesperson will go to the user's designated location and provide appropriate support, such as providing detailed explanations about a specific product or suggesting additional products.

[0052] Accumulating user behavior data and optimizing layout

[0053] The server stores user behavior data and the results of sales staff responses in a database. Based on this data, it analyzes user patterns throughout the store, understanding the popularity of specific areas and products, and user movement patterns. This makes it possible to optimize layouts and product displays to suit the characteristics of each store. For example, if a specific product shelf is frequently visited, it can suggest adding products to the surrounding area or changing the layout.

[0054] Layout optimization example

[0055] For example, suppose there is a specific product area where many users gather on weekends. Based on this data, the server can suggest placing related products around that area or recommend expanding aisles to make user flow smoother. This can improve the user's shopping experience and increase store sales.

[0056] In this way, the present invention provides a system and an operating method for analyzing user behavior in a store in real time and maximizing sales opportunities.

[0057] The processing flow will be explained below.

[0058] Step 1:

[0059] The device captures real-time video data from cameras installed in the store, covering different areas and shelves within the store, and stores multiple video streams in an internal buffer.

[0060] Step 2:

[0061] The video data acquired by the terminal is compressed and sent to the server via the network. The data is compressed using a video codec to improve communication efficiency.

[0062] Step 3:

[0063] The server analyzes the received video data, uses computer vision algorithms to detect the user's movements and facial expressions, uses facial recognition technology to identify the user's face, and uses object recognition technology to identify the product the user is holding.

[0064] Step 4:

[0065] The server tracks the user's dwell time, location, and path, collecting data on how long the user spends in a particular area or in front of a shelf, and recording the user's path.

[0066] Step 5:

[0067] The server learns user behavior patterns based on the collected data, and uses machine learning algorithms to analyze user behavior trends and patterns to identify users with strong purchasing intent.

[0068] Step 6:

[0069] The server determines that users who exhibit certain behavioral patterns are considering a purchase. For example, if a user repeatedly spends a long time in front of a particular product shelf, picking up a product and checking its details, it is determined that the user has a strong intention to purchase.

[0070] Step 7:

[0071] If the server identifies a potential buyer, it immediately sends a notification to a salesperson, including information about the user's current location and the product they are interested in.

[0072] Step 8:

[0073] The salesperson, who is the user, receives the notification, reviews the content, and then contacts the specific user based on the content of the notification to provide appropriate support, such as providing the user with detailed information about the product or suggesting other related products.

[0074] Step 9:

[0075] The server records the results of salespeople's responses and data on interactions with users, which is used to evaluate the effectiveness of sales activities and build a database that can be used for future responses.

[0076] Step 10:

[0077] Based on the accumulated user behavior data, the server generates proposals to optimize the layout and display of each store. It analyzes whether specific areas or products are frequently visited and uses an optimization algorithm to propose layout changes.

[0078] Step 11:

[0079] The store manager, who is the user, reviews the proposals and takes steps to optimize the store layout and displays, thereby achieving efficient store operations and improving customer satisfaction.

[0080] Example 1

[0081] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0082] In conventional stores, there are limited means of understanding customer behavior patterns and purchasing intentions in real time, making it difficult to provide efficient sales support and effectively optimize store layouts. Therefore, there is a need for a method that can accurately grasp customer purchasing intentions to avoid missing sales opportunities, as well as to create effective store layout plans based on customer behavior data and provide a better shopping experience.

[0083] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0084] In this invention, the server includes a means for acquiring video data from a camera installed in the store, a means for analyzing the acquired video data to detect customer movements and facial expressions, a means for tracking customer stay time, location, and flow to learn behavioral patterns, a means for determining whether a customer exhibiting a specific behavioral pattern is a potential purchaser, and a means for sending a notification to a salesperson when a potential purchaser is identified. This allows for efficient sales support by grasping customer purchasing intentions in real time and notifying salespersons at the appropriate time. Furthermore, the server can propose optimization of store layout and product display based on customer behavior data, which is expected to improve customer satisfaction and increase store sales.

[0085] A "camera" is a device installed in a store to monitor and record customer behavior.

[0086] "Video data" refers to images and video information captured by a photographing device.

[0087] "Customer" means a person browsing or purchasing products in a store.

[0088] "Movement" refers to the physical movements and actions that customers perform in the store.

[0089] "Facial expressions" refer to the customer's facial expressions and gestures that show emotion.

[0090] "Dwell time" refers to the amount of time a customer spends in a particular area or in front of a shelf.

[0091] "Location" is information indicating where the customer is located within the store.

[0092] "Flow" refers to the route or pattern that customers take within a store.

[0093] "Behavioral patterns" refer to the trends and characteristics of customer behavior analyzed based on data such as customer length of stay, location, and route.

[0094] "Considering purchase customers" are customers whose behavioral patterns indicate a high level of purchasing intent.

[0095] A "salesperson" is an employee who guides and explains products to customers in a store.

[0096] A "notification" is a message or alert sent from the server to a salesperson.

[0097] The "server" is a central processing unit that analyzes video data and tracks and records customer behavior.

[0098] "Layout" refers to the layout plan within the store and the placement and arrangement of products.

[0099] "Product display" refers to the display and arrangement of products within a store.

[0100] The present invention provides a system that analyzes customer behavior in a store in real time and sends a notification to a salesperson at an appropriate time when it is determined that the customer has a purchase intention. The system also proposes optimization of store layout and product display based on the accumulated customer behavior data. Detailed embodiments of the present invention are described below.

[0101] Acquiring camera footage

[0102] The device acquires video data in real time from multiple cameras placed in the store. These cameras are positioned to cover specific areas or product shelves within the store. For example, cameras can be placed near key product shelves or store entrances to monitor customer behavior from all directions.

[0103] Video data analysis

[0104] The server receives and analyzes video data sent from the device in real time. The video data is processed frame by frame, and pre-processing is performed to identify customer movements and positions. Specifically, the analysis is performed using open-source computer vision libraries (e.g., OpenCV) and machine learning frameworks (e.g., TensorFlow).

[0105] User behavior and facial expression detection

[0106] The server detects the customer's movements and facial expressions from the video data. It uses facial recognition technology to identify the customer's face, and an object detection algorithm to identify the customer's movements and products of interest. For example, by combining facial recognition using OpenCV and object detection using TensorFlow, it can analyze which product the customer is in front of and how they are moving.

[0107] Tracking time spent, location, and traffic

[0108] The server tracks the customer's time spent, location, and movement path (pathway). This makes it possible to determine how long a customer stayed at a particular shelf and which path they took. For example, if a customer stays in front of a particular shelf for five minutes, the server can use this data to analyze the customer's behavioral patterns.

[0109] Determining users considering purchasing

[0110] Based on the customer's behavioral patterns, the server identifies customers who are judged to have a high intent to purchase as "considering purchase." This identification is performed using behavioral analysis algorithms and machine learning models (e.g., random forests and support vector machines). When a considering purchase customer is identified, the server immediately sends a notification to the salesperson via the terminal.

[0111] Notification to sales staff

[0112] The server notifies the salesperson via the terminal of information about the identified potential purchaser, including details about the customer's current location and the product they are interested in. For example, the terminal may be notified that "Customer A is interested in the product on shelf B."

[0113] Salesperson's response

[0114] The salesperson, who is the user, receives a notification from the terminal. Upon receiving the notification, the salesperson quickly goes to the specified customer's location and provides appropriate support, such as providing detailed explanations about the product or suggesting additional items.

[0115] Accumulation and analysis of user behavior data

[0116] The server stores all customer behavior data and the results of sales staff responses in a database, making it possible to analyze customer behavior patterns across the entire store.

[0117] Layout optimization proposals

[0118] The server generates suggestions for improving store layout and product displays based on the accumulated behavioral data, such as "Since a particular area is crowded on weekends, place related products around that area" or "Expand aisles to streamline customer flow."

[0119] Examples of prompt statements

[0120] The following is a specific example of a prompt sentence to be input to the generative AI model:

[0121] "Write a program that analyzes video data obtained from a camera and uses an algorithm to identify behavioral patterns based on the customer's movements and facial expressions to detect customers who intend to make a purchase in real time. Include a function to send a notification to a salesperson if a customer shows interest in a product on the shelf. Use the OpenCV and TensorFlow libraries."

[0122] As described above, the present invention provides a system and an operating method thereof for analyzing customer behavior in a store in real time and maximizing sales opportunities.

[0123] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0124] System program processing flow

[0125] Step 1: Acquire camera footage

[0126] The terminal acquires video data in real time from multiple cameras placed in the store. These cameras are positioned to cover specific areas and product shelves, monitoring customer behavior from all directions. The input is video data from the cameras, and the output is raw video data stored in the terminal.

[0127] Step 2: Sending video data

[0128] The video data acquired by the device is sent to the server in real time. Here, the input is the raw video data stored on the device, and the output is the video data to be sent to the server. For example, the setting is to send the data to the server periodically, frame by frame.

[0129] Step 3: Analyzing the video data

[0130] The server processes the received video data frame by frame and performs preprocessing to identify the customer's movements and position. Specifically, it uses OpenCV to preprocess the images, remove noise, and resize the images. The input is the video data sent to the server, and the output is the preprocessed image data.

[0131] Step 4: Detect customer behavior and facial expressions

[0132] The server detects the customer's movements and facial expressions from the preprocessed image data. It uses facial recognition technology to identify the customer's face and TensorFlow to perform object detection and movement recognition. The input is the preprocessed image data, and the output is analytical data on the customer's movements and facial expressions. For example, it detects whether the customer is smiling or looking at a specific product.

[0133] Step 5: Tracking dwell time, location, and path

[0134] The server records customer location information in real time and tracks their time spent and their path (pathway). The input is the detected customer location data, and the output is behavioral data including the customer's time spent, location, and path. For example, it tracks how long a customer stays in front of a particular product.

[0135] Step 6: Identify potential buyers

[0136] Based on customer behavior data, the server identifies customers who are determined to have a high intent to purchase as "considering purchase customers." It uses behavioral analysis algorithms such as random forests and support vector machines. The input is customer behavior data, and the output is the identification data of customers considering purchase. For example, it may identify "Customer A is interested in Product B."

[0137] Step 7: Notify salespeople

[0138] The server notifies the salesperson via the terminal of the information about the identified customer who is considering purchasing. The input is the specific data of the customer who is considering purchasing, and the output is the notification data sent to the salesperson's terminal. For example, the notification may say, "Customer A is interested in the product on shelf B."

[0139] Step 8: Salesperson response

[0140] The salesperson, who is the user, receives a notification from the terminal and quickly goes to the specified customer's location to provide appropriate support. The input is the notification data, and the output is the action of providing support to the customer, such as providing detailed explanations about the product or suggesting additional products.

[0141] Step 9: Collect and analyze user behavior data

[0142] The server stores all customer behavior data and salesperson response results in a database. The input is customer behavior data and salesperson response data, and the output is the stored database.

[0143] Step 10: Layout optimization proposal

[0144] The server generates suggestions for improving store layout and product displays based on the accumulated behavioral data. The input is the accumulated behavioral data, and the output is a layout optimization proposal. For example, it may suggest, "Since a particular area is crowded on weekends, place related products around that area."

[0145] As a result, the system of the present invention provides a system and an operating method for analyzing customer behavior in a store in real time and maximizing sales opportunities.

[0146] (Application example 1)

[0147] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0148] In recent years, there has been a demand for optimizing customer service and improving the customer experience in physical stores. However, the current situation lacks a system that can analyze customer behavior and purchasing intent in real time and send notifications to sales staff at the appropriate time. This can lead to missed sales opportunities and dissatisfaction among customers who do not receive the support they need. Furthermore, store layout and product display are not sufficiently optimized, making it difficult to increase sales and operate efficiently.

[0149] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0150] In this invention, the server includes: means for acquiring video data from cameras installed in the store; means for analyzing the acquired video data to detect user movements and facial expressions; means for tracking user time spent, location, and flow to learn behavioral patterns; means for determining whether a user exhibiting a specific behavioral pattern is a potential purchaser; means for sending a notification to a salesperson when a potential purchaser is identified; and means for sending a notification to the salesperson in real time using a smart device. This allows salespersons to provide support to customers in a timely manner. Furthermore, store layout and product display can be optimized based on user behavior data, improving store operational efficiency and leading to increased sales.

[0151] "Video data" refers to visual information acquired by a camera or other imaging device, and is data in the form of images or videos.

[0152] "Analysis" refers to processing acquired data to extract and understand specific information.

[0153] A "user" is a customer who is considering products in a store or thinking about making a purchase.

[0154] "Movement" refers to the physical movements and actions that users perform within the store.

[0155] "Facial expressions" refer to changes in the user's face that indicate emotions or intentions.

[0156] "Dwell time" refers to how long a user stays in front of a particular area or product.

[0157] "Location" is information indicating where the user is located within the store.

[0158] A "path" is the route a user follows when moving around a store.

[0159] "Behavioral patterns" are specific trends that are identified from a user's series of actions and location information.

[0160] "Purchase intent" refers to the possibility or willingness of a user to purchase a product.

[0161] A "user considering purchasing" is a user whose behavioral patterns indicate a high level of purchasing intent.

[0162] "Notification" refers to a message or alert that the system sends to a salesperson to inform them of specific information.

[0163] "Salesperson" refers to a staff member who explains products and provides support to customers in a store.

[0164] "Smart devices" refers to mobile information terminals, eyeglass-type devices, head-mounted displays, etc. that have communication functions and can connect to the Internet.

[0165] "Layout optimization" means arranging products and customer flow in a store in the most efficient way possible based on user behavior data.

[0166] "Product display optimization" refers to improving the way products are arranged based on product sales and customer interest.

[0167] The present invention provides a system that analyzes customer behavior in a physical store in real time and notifies sales staff of customers with high purchasing intent, thereby improving store management efficiency and customer satisfaction.

[0168] Acquiring camera footage and analyzing video data

[0169] The server collects video data in real time from multiple cameras placed in the store. These cameras are positioned to cover specific areas and shelves, and the video data from each camera is sent to the server. This video data is then analyzed using a computer vision library (e.g., OpenCV) or a machine learning framework (e.g., TensorFlow).

[0170] Detecting user movements and facial expressions

[0171] The server analyzes the user's movements and facial expressions based on the received video data. It uses facial recognition technology and object detection algorithms to determine where the user is and what products they are interested in. For example, if the user is paying attention to a particular product shelf, it can retrieve information about that product from the database.

[0172] Tracking user time spent, location, and path, and learning behavioral patterns

[0173] The server tracks users' time spent and locations in real time. Based on this information, it learns their behavioral patterns and determines whether a particular pattern indicates a purchasing intent. For example, if a user spends a long time on a particular shelf, it determines that the user is interested in that product.

[0174] Identifying and notifying potential purchasers

[0175] The server identifies users who are deemed to have a high intent to purchase based on their behavioral patterns as "purchase-consideration users." This determination is made using behavioral analysis algorithms and machine learning models (e.g., random forests and support vector machines). When a purchase-consideration user is identified, the server immediately sends a notification to a salesperson via a smart device (e.g., smart glasses or smartphone).

[0176] Salesperson's response

[0177] The salesperson receives a notification on the smart device, which includes the user's current location and product information of interest. Upon receiving the notification, the salesperson will go to the user's specified location and provide appropriate assistance. For example, the salesperson may provide detailed explanations about a specific product or suggest additional products.

[0178] Accumulating user behavior data and optimizing layout

[0179] The server stores user behavior data and the results of sales staff responses in a database. Based on this data, it analyzes user patterns throughout the store, understanding the popularity of specific areas and products, and user movement patterns. This makes it possible to optimize layouts and product displays to suit the characteristics of each store. For example, if a specific product shelf is frequently visited, it can suggest adding products to the surrounding area or changing the layout.

[0180] Practical examples and prompts

[0181] For example, if a customer spends a long time considering a particular product, "Product XYZ," in "Aisle 3" of a store, a notification will be sent to a sales associate wearing smart glasses who will provide appropriate assistance.

[0182] Example prompt sentence:

[0183] "Create a system that analyzes customer behavior in real time and sends notifications about customers who are deemed to have high purchasing intent."

[0184] In this way, the present invention provides a system and an operating method for analyzing user behavior in a physical store in real time and maximizing sales opportunities.

[0185] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0186] Step 1:

[0187] The server acquires video data in real time from multiple cameras installed in the store. The video data sent from the cameras is temporarily stored in the server's storage. This processing uses the camera devices, network connections, and storage functions. The input is video data from the cameras, and the output is video data stored on the server.

[0188] Step 2:

[0189] The server analyzes the captured video data to detect the user's movements and facial expressions. This uses a computer vision library (e.g., OpenCV) or a machine learning framework (e.g., TensorFlow). The input is the stored video data, and the output is analytical data on the user's position, movements, and facial expressions. Based on the analysis, the server determines which area the user is in.

[0190] Step 3:

[0191] The server performs processing to track users' time spent, location, and path in real time. Based on this information, it learns user behavior patterns. The inputs are location data, time spent, and path data, and the output is user behavior pattern data. The server uses storage data and real-time analysis data to track user paths.

[0192] Step 4:

[0193] The server analyzes behavioral patterns and runs an algorithm to determine whether users who exhibit certain behavioral patterns are "considerers." This process uses behavioral analysis algorithms and machine learning models (e.g., random forests and support vector machines). The input is behavioral pattern data, and the output is identification data for users considering purchasing.

[0194] Step 5:

[0195] When a "purchase-consideration user" is identified, the server immediately sends a notification to the smart device (e.g., smart glasses, smartphone). The input is the user's identification data, and the output is the notification data sent to the smart device. The server uses a network connection to send the notification.

[0196] Step 6:

[0197] The terminal receives the notification and displays an alert to the salesperson. The salesperson checks the notification on their smart device, goes to the specified user's location, and takes appropriate action to provide support. The input is the notification data sent from the server, and the output is the alert display to the salesperson.

[0198] Step 7:

[0199] The server stores user behavior data and the results of sales staff responses in a database. The stored data is used to optimize store layout and product displays. The input is user behavior data and response result data, and the output is optimization proposal data stored in the database. The server analyzes the stored data and generates optimization proposals for layout and display.

[0200] In this way, specific operations are clarified at each step, and the input and output data are defined for each step. Following this flow allows the entire system to operate effectively.

[0201] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0202] The present invention combines an emotion engine with a system that analyzes the behavior of users (customers) in a store in real time and sends a notification to a salesperson at the appropriate time if it is determined that the user has an intention to purchase. This system recognizes the user's emotions and can provide more accurate support based on that information. Detailed embodiments of the present invention are described below.

[0203] Camera image acquisition and video data analysis

[0204] The device collects video data in real time from multiple cameras in the store. This video data is sent from cameras positioned to cover specific areas and shelves in the store. The server receives this video data and analyzes it in real time.

[0205] Detecting user movements and facial expressions

[0206] The server analyzes the user's movements and facial expressions based on the received video data. Specifically, it uses facial recognition technology and object detection algorithms to determine where the user is and what products they are interested in. This analysis can be performed using open-source computer vision libraries (e.g., OpenCV) and machine learning frameworks (e.g., TensorFlow).

[0207] Introducing the Emotion Engine

[0208] Based on the analyzed facial recognition data, the server uses an emotion engine to recognize the user's emotions. The emotion engine analyzes the user's facial muscle movements and other facial features to determine emotions such as happiness, surprise, anger, sadness, etc. This information, combined with the analysis of movements and facial expressions, is used to more precisely understand the user's overall behavioral patterns.

[0209] Tracking time spent, location, and path, and learning behavioral patterns

[0210] The server tracks users' time spent, location, and route in real time. This allows it to determine how long a user stays at a particular shelf and the path they take. This data is used to learn user behavior patterns. For example, if a user spends a long time in a certain area, it can be determined that they are interested in that area.

[0211] Identifying and notifying potential purchasers

[0212] Based on the user's behavioral patterns and emotional data, the server identifies users who are deemed to have a high intent to purchase as "considering purchase users." For example, if a user spends a long time in front of a particular product shelf, repeatedly picks up a product to check its details, or shows positive emotions (e.g., joy), the server determines that the user has a high intent to purchase.

[0213] When a potential buyer is identified, the server immediately sends a notification to a salesperson, containing information about the user's current location, the product they are interested in, and their current emotional state.

[0214] Salesperson's response

[0215] The salesperson, who is the user, receives the notification on their device and checks the content. Based on the content of the notification, they can contact the specific user and provide appropriate support. For example, they can provide the user with detailed information about the product or suggest other related products. In addition, they can suggest additional services or benefits to users who show particularly positive emotions.

[0216] Accumulating user behavior data and optimizing layout

[0217] The server stores user behavior data and the results of sales staff responses in a database. Based on this data, it analyzes user patterns throughout the store, understanding the popularity of specific areas and products, and user movement patterns. This makes it possible to optimize layouts and product displays to suit the characteristics of each store. For example, if a specific product shelf is frequently visited, it can suggest adding products to the surrounding area or changing the layout.

[0218] Layout optimization example

[0219] For example, suppose there is a specific product area where many users gather on weekends. Based on this data, the server can suggest placing related products around that area or recommend expanding aisles to make user flow smoother. This can improve the user's shopping experience and increase store sales.

[0220] In this way, the present invention provides a system and an operating method for analyzing user behavior and emotions in a store in real time to maximize sales opportunities.

[0221] The processing flow will be explained below.

[0222] Step 1:

[0223] The device captures real-time video data from cameras installed in the store, covering different areas and shelves within the store, and stores multiple video streams in an internal buffer.

[0224] Step 2:

[0225] The video data acquired by the terminal is compressed and sent to the server via the network. The data is compressed using a video codec to improve communication efficiency.

[0226] Step 3:

[0227] The server analyzes the received video data, uses computer vision algorithms to detect the user's movements and facial expressions, uses facial recognition technology to identify the user's face, and uses object recognition technology to identify the product the user is holding.

[0228] Step 4:

[0229] The server uses an emotion engine to recognize the user's emotions from the analyzed facial recognition data. The emotion engine analyzes the user's facial muscle movements and facial features to determine emotions such as joy, surprise, anger, and sadness.

[0230] Step 5:

[0231] The server tracks the user's dwell time, location, and path, collecting data on how long the user spends in a particular area or in front of a shelf, and recording the user's path.

[0232] Step 6:

[0233] The server learns user behavior patterns based on the collected data, and uses machine learning algorithms to analyze user behavior trends and patterns to identify users with strong purchasing intent.

[0234] Step 7:

[0235] The server combines users who exhibit specific behavioral patterns with positive emotions generated by the emotion engine to determine whether they are considering a purchase. For example, if a user spends a long time in front of a particular product shelf, checking product details, or expresses happy emotions, it determines that the user has a strong intention to purchase.

[0236] Step 8:

[0237] When the server identifies a potential buyer, it immediately sends a notification to the salesperson, including the user's current location, the products they are interested in, and their current emotional state.

[0238] Step 9:

[0239] The salesperson, who is the user, receives the notification, reviews its contents, and then contacts the specific user based on the content of the notification to provide appropriate support. For example, the salesperson can provide the user with more information about the product or suggest other related products. Additionally, the salesperson can offer additional services or special offers to users who show particularly positive emotions.

[0240] Step 10:

[0241] The server records the results of salespeople's responses and data on interactions with users, which is used to evaluate the effectiveness of sales activities and build a database that can be used for future responses.

[0242] Step 11:

[0243] The server generates proposals to optimize the store's overall layout and display based on accumulated user behavior and emotion data. It analyzes whether specific areas or products are frequently visited and uses an optimization algorithm to propose layout changes.

[0244] Step 12:

[0245] The store manager, who is the user, reviews the proposals and takes steps to optimize the store layout and displays, thereby achieving efficient store operations and improving customer satisfaction.

[0246] Example 2

[0247] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0248] In modern retail stores, it is difficult for sales staff to accurately grasp users' purchasing intentions and provide timely support. Furthermore, there is a lack of methods for analyzing and utilizing user emotion and behavior data in real time to implement effective sales strategies and optimize layouts. These problems need to be solved.

[0249] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for acquiring video data from a monitoring device installed in the store, a means for analyzing the acquired video data to detect the user's movements and facial expressions, a means for recognizing emotions based on the user's facial expression data, a means for tracking the user's stay time, location, and flow to learn behavioral patterns, a means for determining that a user exhibiting a specific behavioral pattern or emotion is a user considering purchasing, and a means for sending a notification to a notification target when a user considering purchasing is identified. This allows the user's purchasing intention to be properly understood and sales staff to provide timely support. Furthermore, analyzing user behavioral data can optimize store layout and improve sales strategies.

[0250] A "monitoring device" is a device that is placed in a store and captures users' movements and facial expressions in real time.

[0251] "Video data" refers to visual information including the user's movements and facial expressions acquired by a monitoring device.

[0252] "Analysis" is the process of processing video data, recognizing the user's movements and facial expressions, and extracting this as data.

[0253] "Facial expression data" is information that indicates emotions extracted from the user's facial movements and features.

[0254] "Emotion recognition" is the process of analyzing facial expression data to identify the emotion a user is expressing (e.g., joy, surprise, anger, sadness, etc.).

[0255] "Behavioral patterns" are the tendencies and characteristics of user behavior learned based on data such as the user's length of stay, location, and path.

[0256] A "purchase consideration user" is a user who is determined to have a high desire to purchase because they exhibit specific behavioral patterns and emotions based on analysis results and emotion recognition data.

[0257] The "notification recipient" is a person (e.g., a salesperson) who should receive a notification when a user considering purchase is identified.

[0258] A "notification" is a message containing information about a user considering purchasing that is sent from the server to the notification target.

[0259] "Layout optimization" is the process of optimizing the placement and display within a location based on tracked user behavior data.

[0260] This invention is a system that analyzes the behavior of users (customers) in a store in real time, identifies the user's purchase intentions, and notifies the salesperson. By combining this system with an emotion engine that recognizes the user's emotions, it is possible to provide more accurate support.

[0261] Camera image acquisition and video data analysis

[0262] The terminal acquires video data in real time from multiple monitoring devices (e.g., network cameras) placed in the store. The monitoring devices are placed to cover specific areas or product shelves. The server receives this video data and performs analysis. The analysis is performed using Python's OpenCV library and TensorFlow.

[0263] Detecting user movements and facial expressions

[0264] The server analyzes the user's movements and facial expressions based on the received video data. Specifically, it uses facial recognition technology and object detection algorithms to identify the user's location and products they are interested in. This analysis is performed using open-source computer vision libraries (e.g., OpenCV) and machine learning frameworks (e.g., TensorFlow).

[0265] Introducing the Emotion Engine

[0266] Based on the analyzed facial recognition data, the server uses an emotion engine to recognize the user's emotions. The emotion engine analyzes the user's facial muscle movements and other facial features to determine emotions such as happiness, surprise, anger, sadness, etc. This information, combined with the analysis of movements and facial expressions, is used to more precisely understand the user's overall behavioral patterns.

[0267] Tracking user behavior and learning patterns

[0268] The server tracks the user's time spent, location, and path in real time, allowing it to determine how long the user spent at a particular shelf and the path they took. For example, if a user spends 10 minutes in front of a particular shelf, it can determine that they are interested in that product.

[0269] Determining and notifying purchase intent

[0270] Based on the user's behavioral patterns and emotional data, the server identifies users who are deemed to have a strong intent to purchase as "considering purchase users." For example, if a user lingers in front of a particular product shelf for a long time, repeatedly picks up a product to check its details, or shows positive emotions (e.g., joy). When a considering purchase user is identified, the server immediately sends a notification to the salesperson.

[0271] Salesperson's response

[0272] The salesperson, who is the user, receives the notification sent to their device and checks its contents. The notification includes the user's current location, products of interest, and the user's current emotional state. Based on this, the salesperson contacts the specific user and provides detailed product information or suggests related products.

[0273] Accumulating user behavior data and optimizing layout

[0274] The server stores user behavior data and the results of sales staff responses in a database. This data is then analyzed to determine user patterns throughout the store. By understanding the popularity of specific areas and products, as well as user movement patterns, it becomes possible to optimize layouts and product displays to suit the characteristics of each store. For example, based on data showing that a specific product area is frequently visited on weekends, it can propose changes to the layout of that area and the placement of related products.

[0275] Examples of concrete examples and prompts

[0276] Examples:

[0277] For example, a user enters a store and heads to the food section. The device receives video data via a camera installed in the device, which is then analyzed by a server. The server recognizes that the user spends a long time in front of the shelves in the food section, picking up and looking at several products. It combines this information with emotional data to determine that the user is interested in those products. The server then sends a notification to a salesperson, who then checks the user's location, products of interest, and emotional state on the device. The salesperson then contacts the user, explains the details of the products, and suggests related products.

[0278] Example prompt for a generative AI model:

[0279] "What is the appropriate way to respond when a customer lingers in front of a particular shelf in a store for a long time?"

[0280] "Please explain the specific method for recognizing user emotions and determining purchasing intent from video data."

[0281] As described above, the present invention provides a system and an operating method for analyzing user behavior and emotions in real time to maximize sales opportunities.

[0282] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0283] Step 1: Acquire camera footage

[0284] The terminal acquires video data from surveillance equipment installed in the store. The surveillance equipment uses network cameras, each positioned to cover a specific area or shelf. For example, a camera captures a food section, and the video data is streamed to the terminal. The input is real-time video data, and the output is video data sent to a server via the terminal.

[0285] Step 2: Analyzing the video data

[0286] The server receives video data sent from the device. The server uses Python's OpenCV library to extract frames of the video data and perform preprocessing such as noise removal. Next, it uses TensorFlow to analyze the user's movements and facial expressions. For example, the server analyzes frames every second to detect the movements of the user's face and hands. The input is the captured video data, and the output is the analyzed user's movements and facial expression data.

[0287] Step 3: Detecting user movements and facial expressions

[0288] The server uses OpenCV's Haar Cascade and TensorFlow to identify the position of the user's face and hands and detect changes in facial expressions and movements. For example, it recognizes the moment when the user smiles or picks up a specific product. The input is the video frame extracted in the previous step, and the output is specific data on the user's movements and facial expressions.

[0289] Step 4: Emotion Recognition with the Emotion Engine

[0290] The server uses Affectiva's API to send the facial expression data obtained in the previous step and recognize the user's emotions in real time. The emotion engine determines emotions (happiness, surprise, anger, sadness, etc.) from the user's facial muscle movements and facial features. The input is the user's facial expression data, and the output is the recognized emotion data.

[0291] Step 5: Track user behavior and learn patterns

[0292] The server tracks the user's time spent, location, and path in real time and records this data. For example, the database stores information such as how long the user stayed in front of a particular shelf and the path they took. The input is the user's location and path data, and the output is behavioral patterns and learned user data.

[0293] Step 6: Determine and communicate purchase intent

[0294] The server analyzes the user's behavioral patterns and emotional data, and identifies users who are determined to have a strong intention to purchase as "considering purchase users." For example, this applies when a user lingers in front of a particular product shelf for a long time, picks up the product to check its details, or shows positive emotions (joy). The input is behavioral pattern data and emotional data, and the output is a notification to the salesperson.

[0295] Step 7: Salesperson response

[0296] The salesperson, who is the user, receives the notification sent to the terminal and checks the content. The notification includes the user's current location, products of interest, and emotional state. Based on this, the salesperson contacts the user to provide detailed product information and suggest related products. The input is the notification data from the server, and the output is specific support for the user.

[0297] Step 8: Accumulating user behavior data and optimizing layout

[0298] The server stores user behavior data and the results of sales staff responses in a database. This data is analyzed to understand the popularity of specific areas and products, as well as user movement patterns. For example, if a specific product area is frequently visited on weekends, the server will suggest changes to the layout of that area and the placement of related products. The input is the stored behavioral data, and the output is specific proposals for layout optimization.

[0299] Through these steps, the system can analyze user behavior and emotions in real time, enabling effective sales strategies and store layout optimization.

[0300] (Application example 2)

[0301] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0302] In traditional brick-and-mortar stores, it was difficult to analyze customer behavior and emotions in real time and provide appropriate support at the right time. This made it difficult to effectively stimulate customer purchasing desire, limiting store sales. Furthermore, data analysis to improve customer experience was insufficient, and store layout and product display were not optimized.

[0303] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0304] In this invention, the server includes: means for acquiring video data from cameras installed in the store; means for analyzing the acquired video data to detect user movements and facial expressions; means for tracking user stay time, location, and flow to learn behavioral patterns; means for recognizing user emotions based on the analyzed facial recognition data; means for determining that a user exhibiting a specific behavioral pattern and emotional state is a user considering a purchase; and means for sending a notification to a salesperson when a user considering a purchase is identified. This allows for timely notification to a salesperson based on the customer's behavior and emotions, enabling effective customer service. Furthermore, it is possible to optimize the store layout and product display based on user behavior data and emotional data, improving the customer experience and increasing sales.

[0305] "In-store" refers to the physical location where customers can actually touch and purchase products.

[0306] A "camera" is a visual sensor device for acquiring video data.

[0307] "Video data" refers to the digital information of images or video captured by a camera.

[0308] A "user" is a customer who performs actions such as viewing products in a store or making a purchase.

[0309] "Movement" refers to the movement or change in position of the user's body.

[0310] "Facial expressions" are the emotional expressions and muscle movements that appear on the user's face.

[0311] "Dwell time" refers to the amount of time a user stays in a particular location or on a shelf.

[0312] "Location" is information that indicates which area of ​​the store the user is in.

[0313] "Pathways" refer to the routes and patterns that users take within a store.

[0314] "Behavioral patterns" refer to habitual behavior that can be inferred from a user's series of actions, changes in location, length of stay, etc.

[0315] A "purchase consideration user" is a user who is determined to have a strong intention to purchase based on their behavioral patterns and emotional state.

[0316] A "notification" is a message sent to convey specific information.

[0317] A "salesperson" is a staff member who explains products and provides support to customers in a store.

[0318] "Facial recognition data" is digital information used to identify and analyze a user's face.

[0319] "Emotion" refers to a user's psychological state as judged from their facial expressions and behavior.

[0320] A "server" is a computer system that analyzes video data, stores various data, and sends notifications.

[0321] "Optimization" refers to the efficient rearrangement of layout and product displays based on data analysis.

[0322] The present invention relates to a system that analyzes customer behavior and emotions in real time in a store and provides appropriate support at the appropriate time. This system is realized by capturing video data using cameras placed in the store and analyzing that video data.

[0323] Acquiring camera footage

[0324] The server collects real-time video data from multiple cameras placed throughout the store, which are positioned to cover specific areas and shelves, allowing for comprehensive monitoring of customer behavior.

[0325] Video data analysis

[0326] The server analyzes the captured video data to detect the customer's movements and facial expressions. This analysis uses open-source computer vision libraries and machine learning frameworks such as OpenCV and TensorFlow. Specifically, it uses a facial recognition model to detect the customer's facial expressions and recognizes the customer's emotions based on the facial expression data.

[0327] Introducing the Emotion Engine

[0328] The server recognizes the customer's emotions using an emotion engine based on the analyzed facial recognition data. The emotion engine analyzes the customer's facial muscle movements and other facial features to determine emotions such as happiness, surprise, anger, sadness, etc. This information, combined with the analysis of movements and facial expressions, is used to more precisely understand the customer's overall behavioral patterns.

[0329] Tracking user time, location, and path

[0330] The server tracks customer dwell time, location, and path in real time, allowing it to determine how long a customer spends at a particular shelf and the path they take. This data is used to learn user behavior patterns.

[0331] Identifying and notifying potential purchasers

[0332] Based on the user's behavioral patterns and emotional data, the server identifies users who are determined to have a high intent to purchase as "purchase-consideration users." For example, if a user spends a long time in front of a particular product shelf, repeatedly picks up products to check their details, or shows positive emotions (e.g., joy), the server determines that the user has a high intent to purchase. When a user is identified as a purchase-consideration user, the server immediately sends a notification to the salesperson. The notification includes information about the user's current location, the products they are interested in, and their current emotional state.

[0333] Salesperson's response

[0334] Salespeople receive notifications on their devices, review the content, and contact specific users based on the content of the notifications to provide appropriate support. For example, they can provide detailed product information or suggest other related products. Additionally, they can offer additional services or special offers to users who show particularly positive emotions.

[0335] Accumulating user behavior data and optimizing layout

[0336] The server stores user behavior data and sales staff response results in a database. Based on this data, it analyzes user patterns throughout the store, understanding the popularity of specific areas and products, and user movement patterns. This makes it possible to optimize layouts and product displays to suit the characteristics of each store.

[0337] Examples of concrete examples and prompts

[0338] As a specific example, if customer A is seen smiling in the in-store camera footage and spends a long time in front of a particular product, the application will notify the salesperson of this information so that the salesperson can provide appropriate assistance.

[0339] Example prompt for a generative AI model:

[0340] "We will develop a system that analyzes video footage from multiple cameras in real time, recognizes faces and emotions, determines the purchasing intent of customers in a specific area, and notifies sales staff at the appropriate time."

[0341] Thus, the present invention provides a system and an operating method thereof for analyzing customer behavior and emotions in real time and maximizing sales opportunities.

[0342] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0343] Step 1:

[0344] Acquiring video data from the camera

[0345] The server receives real-time video data from cameras placed around the store, capturing images of specific areas of the store and product shelves. It receives the video data from the cameras as input and converts it into a processable format as output.

[0346] Step 2:

[0347] Video data analysis

[0348] The server detects the user's movements and facial expressions based on the acquired video data. Specifically, it uses OpenCV to perform face detection and identify the user's location. The input is video data from the camera, and the output is the location information of the detected face.

[0349] Step 3:

[0350] Emotion recognition based on facial recognition data

[0351] The server uses facial recognition data to recognize the user's emotions. Using a TensorFlow model, it analyzes the user's facial muscle movements and other features to determine emotions such as joy, surprise, and sadness. The input is the facial position information and video data obtained in step 2, and the output is the user's emotional state.

[0352] Step 4:

[0353] Tracking time spent, location, and traffic

[0354] The server tracks users' time spent, location, and route in real time, allowing it to determine how long they stay in a particular area and the route they take. The input is video data from the camera and facial recognition data, and the output is user behavior pattern data.

[0355] Step 5:

[0356] Identifying and notifying potential purchasers

[0357] The server identifies users who are deemed to have a high intent to purchase based on their behavioral patterns and emotional data as "consideration users." If identified, it sends a notification to the salesperson containing information about the user's current location, products they are interested in, and their emotional state. The input is the data obtained in Step 4 and Step 3, and the output is a notification to the salesperson.

[0358] Step 6:

[0359] Salesperson's response

[0360] The terminal allows the salesperson to receive the notification and provide support to the user based on its content. Specifically, it provides detailed product information and suggests other related products to the user. In particular, it suggests additional services and benefits to users who show positive emotions. The input is the notification sent to the salesperson's terminal, and the output is the result of the support provided.

[0361] Step 7:

[0362] Accumulating user behavior data and optimizing layout

[0363] The server accumulates user behavior data and salesperson response results in a database and analyzes user patterns throughout the store. This allows the popularity of specific areas within the store and user movement patterns to be understood, and layout optimization proposals are generated. The input is the data obtained at each step, and the output is layout and display optimization proposals.

[0364] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0365] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0366] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0367] [Second embodiment]

[0368] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0369] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0370] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0371] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0372] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0373] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0374] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0375] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0376] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0377] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0378] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0379] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0380] The present invention provides a system that analyzes the behavior of users (customers) in a store in real time and sends a notification to sales staff at an appropriate time when it is determined that the user has a purchase intention.The system also makes suggestions for optimizing store layout and product displays based on the accumulated customer behavior data.Detailed embodiments of the present invention are described below.

[0381] Camera image acquisition and video data analysis

[0382] The device collects video data in real time from multiple cameras in the store. This video data is sent from cameras positioned to cover specific areas and shelves in the store. The server receives this video data and analyzes it in real time.

[0383] Detecting user movements and facial expressions

[0384] The server analyzes the user's movements and facial expressions based on the received video data. Specifically, it uses facial recognition technology and object detection algorithms to determine where the user is and what products they are interested in. This analysis can be performed using open-source computer vision libraries (e.g., OpenCV) and machine learning frameworks (e.g., TensorFlow).

[0385] Tracking time spent, location, and path, and learning behavioral patterns

[0386] The server tracks users' time spent, location, and route in real time. This allows it to determine how long a user stays at a particular shelf and the path they take. This data is used to learn user behavior patterns. For example, if a user spends a long time in a certain area, it can be determined that they are interested in that area.

[0387] Identifying and notifying potential purchasers

[0388] Based on the user's behavioral patterns, the server identifies users who are deemed to have a high intent to purchase as "purchase-consideration users." This identification is performed using behavioral analysis algorithms and machine learning models (e.g., random forests and support vector machines). When a user is identified as a purchase-consideration user, the server immediately sends a notification to the salesperson via the terminal. This notification includes information about the user's current location and the products they are interested in.

[0389] Salesperson's response

[0390] The salesperson, who is the user, receives a notification on their device. Upon receiving the notification, the salesperson will go to the user's designated location and provide appropriate support, such as providing detailed explanations about a specific product or suggesting additional products.

[0391] Accumulating user behavior data and optimizing layout

[0392] The server stores user behavior data and the results of sales staff responses in a database. Based on this data, it analyzes user patterns throughout the store, understanding the popularity of specific areas and products, and user movement patterns. This makes it possible to optimize layouts and product displays to suit the characteristics of each store. For example, if a specific product shelf is frequently visited, it can suggest adding products to the surrounding area or changing the layout.

[0393] Layout optimization example

[0394] For example, suppose there is a specific product area where many users gather on weekends. Based on this data, the server can suggest placing related products around that area or recommend expanding aisles to make user flow smoother. This can improve the user's shopping experience and increase store sales.

[0395] In this way, the present invention provides a system and an operating method for analyzing user behavior in a store in real time and maximizing sales opportunities.

[0396] The processing flow will be explained below.

[0397] Step 1:

[0398] The device captures real-time video data from cameras installed in the store, covering different areas and shelves within the store, and stores multiple video streams in an internal buffer.

[0399] Step 2:

[0400] The video data acquired by the terminal is compressed and sent to the server via the network. The data is compressed using a video codec to improve communication efficiency.

[0401] Step 3:

[0402] The server analyzes the received video data, uses computer vision algorithms to detect the user's movements and facial expressions, uses facial recognition technology to identify the user's face, and uses object recognition technology to identify the product the user is holding.

[0403] Step 4:

[0404] The server tracks the user's dwell time, location, and path, collecting data on how long the user spends in a particular area or in front of a shelf, and recording the user's path.

[0405] Step 5:

[0406] The server learns user behavior patterns based on the collected data, and uses machine learning algorithms to analyze user behavior trends and patterns to identify users with strong purchasing intent.

[0407] Step 6:

[0408] The server determines that users who exhibit certain behavioral patterns are considering a purchase. For example, if a user repeatedly spends a long time in front of a particular product shelf, picking up a product and checking its details, it is determined that the user has a strong intention to purchase.

[0409] Step 7:

[0410] If the server identifies a potential buyer, it immediately sends a notification to a salesperson, including information about the user's current location and the product they are interested in.

[0411] Step 8:

[0412] The salesperson, who is the user, receives the notification, reviews the content, and then contacts the specific user based on the content of the notification to provide appropriate support, such as providing the user with detailed information about the product or suggesting other related products.

[0413] Step 9:

[0414] The server records the results of salespeople's responses and data on interactions with users, which is used to evaluate the effectiveness of sales activities and build a database that can be used for future responses.

[0415] Step 10:

[0416] Based on the accumulated user behavior data, the server generates proposals to optimize the layout and display of each store. It analyzes whether specific areas or products are frequently visited and uses an optimization algorithm to propose layout changes.

[0417] Step 11:

[0418] The store manager, who is the user, reviews the proposals and takes steps to optimize the store layout and displays, thereby achieving efficient store operations and improving customer satisfaction.

[0419] Example 1

[0420] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0421] In conventional stores, there are limited means of understanding customer behavior patterns and purchasing intentions in real time, making it difficult to provide efficient sales support and effectively optimize store layouts. Therefore, there is a need for a method that can accurately grasp customer purchasing intentions to avoid missing sales opportunities, as well as to create effective store layout plans based on customer behavior data and provide a better shopping experience.

[0422] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0423] In this invention, the server includes a means for acquiring video data from a camera installed in the store, a means for analyzing the acquired video data to detect customer movements and facial expressions, a means for tracking customer stay time, location, and flow to learn behavioral patterns, a means for determining whether a customer exhibiting a specific behavioral pattern is a potential purchaser, and a means for sending a notification to a salesperson when a potential purchaser is identified. This allows for efficient sales support by grasping customer purchasing intentions in real time and notifying salespersons at the appropriate time. Furthermore, the server can propose optimization of store layout and product display based on customer behavior data, which is expected to improve customer satisfaction and increase store sales.

[0424] A "camera" is a device installed in a store to monitor and record customer behavior.

[0425] "Video data" refers to images and video information captured by a photographing device.

[0426] "Customer" means a person browsing or purchasing products in a store.

[0427] "Movement" refers to the physical movements and actions that customers perform in the store.

[0428] "Facial expressions" refer to the customer's facial expressions and gestures that show emotion.

[0429] "Dwell time" refers to the amount of time a customer spends in a particular area or in front of a shelf.

[0430] "Location" is information indicating where the customer is located within the store.

[0431] "Flow" refers to the route or pattern that customers take within a store.

[0432] "Behavioral patterns" refer to the trends and characteristics of customer behavior analyzed based on data such as customer length of stay, location, and route.

[0433] "Considering purchase customers" are customers whose behavioral patterns indicate a high level of purchasing intent.

[0434] A "salesperson" is an employee who guides and explains products to customers in a store.

[0435] A "notification" is a message or alert sent from the server to a salesperson.

[0436] The "server" is a central processing unit that analyzes video data and tracks and records customer behavior.

[0437] "Layout" refers to the layout plan within the store and the placement and arrangement of products.

[0438] "Product display" refers to the display and arrangement of products within a store.

[0439] The present invention provides a system that analyzes customer behavior in a store in real time and sends a notification to a salesperson at an appropriate time when it is determined that the customer has a purchase intention. The system also proposes optimization of store layout and product display based on the accumulated customer behavior data. Detailed embodiments of the present invention are described below.

[0440] Acquiring camera footage

[0441] The device acquires video data in real time from multiple cameras placed in the store. These cameras are positioned to cover specific areas or product shelves within the store. For example, cameras can be placed near key product shelves or store entrances to monitor customer behavior from all directions.

[0442] Video data analysis

[0443] The server receives and analyzes video data sent from the device in real time. The video data is processed frame by frame, and pre-processing is performed to identify customer movements and positions. Specifically, the analysis is performed using open-source computer vision libraries (e.g., OpenCV) and machine learning frameworks (e.g., TensorFlow).

[0444] User behavior and facial expression detection

[0445] The server detects the customer's movements and facial expressions from the video data. It uses facial recognition technology to identify the customer's face, and an object detection algorithm to identify the customer's movements and products of interest. For example, by combining facial recognition using OpenCV and object detection using TensorFlow, it can analyze which product the customer is in front of and how they are moving.

[0446] Tracking time spent, location, and traffic

[0447] The server tracks the customer's time spent, location, and movement path (pathway). This makes it possible to determine how long a customer stayed at a particular shelf and which path they took. For example, if a customer stays in front of a particular shelf for five minutes, the server can use this data to analyze the customer's behavioral patterns.

[0448] Determining users considering purchasing

[0449] Based on the customer's behavioral patterns, the server identifies customers who are judged to have a high intent to purchase as "considering purchase." This identification is performed using behavioral analysis algorithms and machine learning models (e.g., random forests and support vector machines). When a considering purchase customer is identified, the server immediately sends a notification to the salesperson via the terminal.

[0450] Notification to sales staff

[0451] The server notifies the salesperson via the terminal of information about the identified potential purchaser, including details about the customer's current location and the product they are interested in. For example, the terminal may be notified that "Customer A is interested in the product on shelf B."

[0452] Salesperson's response

[0453] The salesperson, who is the user, receives a notification from the terminal. Upon receiving the notification, the salesperson quickly goes to the specified customer's location and provides appropriate support, such as providing detailed explanations about the product or suggesting additional items.

[0454] Accumulation and analysis of user behavior data

[0455] The server stores all customer behavior data and the results of sales staff responses in a database, making it possible to analyze customer behavior patterns across the entire store.

[0456] Layout optimization proposals

[0457] The server generates suggestions for improving store layout and product displays based on the accumulated behavioral data, such as "Since a particular area is crowded on weekends, place related products around that area" or "Expand aisles to streamline customer flow."

[0458] Examples of prompt statements

[0459] The following is a specific example of a prompt sentence to be input to the generative AI model:

[0460] "Write a program that analyzes video data obtained from a camera and uses an algorithm to identify behavioral patterns based on the customer's movements and facial expressions to detect customers who intend to make a purchase in real time. Include a function to send a notification to a salesperson if a customer shows interest in a product on the shelf. Use the OpenCV and TensorFlow libraries."

[0461] As described above, the present invention provides a system and an operating method thereof for analyzing customer behavior in a store in real time and maximizing sales opportunities.

[0462] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0463] System program processing flow

[0464] Step 1: Acquire camera footage

[0465] The terminal acquires video data in real time from multiple cameras placed in the store. These cameras are positioned to cover specific areas and product shelves, monitoring customer behavior from all directions. The input is video data from the cameras, and the output is raw video data stored in the terminal.

[0466] Step 2: Sending video data

[0467] The video data acquired by the device is sent to the server in real time. Here, the input is the raw video data stored on the device, and the output is the video data to be sent to the server. For example, the setting is to send the data to the server periodically, frame by frame.

[0468] Step 3: Analyzing the video data

[0469] The server processes the received video data frame by frame and performs preprocessing to identify the customer's movements and position. Specifically, it uses OpenCV to preprocess the images, remove noise, and resize the images. The input is the video data sent to the server, and the output is the preprocessed image data.

[0470] Step 4: Detect customer behavior and facial expressions

[0471] The server detects the customer's movements and facial expressions from the preprocessed image data. It uses facial recognition technology to identify the customer's face and TensorFlow to perform object detection and movement recognition. The input is the preprocessed image data, and the output is analytical data on the customer's movements and facial expressions. For example, it detects whether the customer is smiling or looking at a specific product.

[0472] Step 5: Tracking dwell time, location, and path

[0473] The server records customer location information in real time and tracks their time spent and their path (pathway). The input is the detected customer location data, and the output is behavioral data including the customer's time spent, location, and path. For example, it tracks how long a customer stays in front of a particular product.

[0474] Step 6: Identify potential buyers

[0475] Based on customer behavior data, the server identifies customers who are determined to have a high intent to purchase as "considering purchase customers." It uses behavioral analysis algorithms such as random forests and support vector machines. The input is customer behavior data, and the output is the identification data of customers considering purchase. For example, it may identify "Customer A is interested in Product B."

[0476] Step 7: Notify salespeople

[0477] The server notifies the salesperson via the terminal of the information about the identified customer who is considering purchasing. The input is the specific data of the customer who is considering purchasing, and the output is the notification data sent to the salesperson's terminal. For example, the notification may say, "Customer A is interested in the product on shelf B."

[0478] Step 8: Salesperson response

[0479] The salesperson, who is the user, receives a notification from the terminal and quickly goes to the specified customer's location to provide appropriate support. The input is the notification data, and the output is the action of providing support to the customer, such as providing detailed explanations about the product or suggesting additional products.

[0480] Step 9: Collect and analyze user behavior data

[0481] The server stores all customer behavior data and salesperson response results in a database. The input is customer behavior data and salesperson response data, and the output is the stored database.

[0482] Step 10: Layout optimization proposal

[0483] The server generates suggestions for improving store layout and product displays based on the accumulated behavioral data. The input is the accumulated behavioral data, and the output is a layout optimization proposal. For example, it may suggest, "Since a particular area is crowded on weekends, place related products around that area."

[0484] As a result, the system of the present invention provides a system and an operating method for analyzing customer behavior in a store in real time and maximizing sales opportunities.

[0485] (Application example 1)

[0486] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0487] In recent years, there has been a demand for optimizing customer service and improving the customer experience in physical stores. However, the current situation lacks a system that can analyze customer behavior and purchasing intent in real time and send notifications to sales staff at the appropriate time. This can lead to missed sales opportunities and dissatisfaction among customers who do not receive the support they need. Furthermore, store layout and product display are not sufficiently optimized, making it difficult to increase sales and operate efficiently.

[0488] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0489] In this invention, the server includes: means for acquiring video data from cameras installed in the store; means for analyzing the acquired video data to detect user movements and facial expressions; means for tracking user time spent, location, and flow to learn behavioral patterns; means for determining whether a user exhibiting a specific behavioral pattern is a potential purchaser; means for sending a notification to a salesperson when a potential purchaser is identified; and means for sending a notification to the salesperson in real time using a smart device. This allows salespersons to provide support to customers in a timely manner. Furthermore, store layout and product display can be optimized based on user behavior data, improving store operational efficiency and leading to increased sales.

[0490] "Video data" refers to visual information acquired by a camera or other imaging device, and is data in the form of images or videos.

[0491] "Analysis" refers to processing acquired data to extract and understand specific information.

[0492] A "user" is a customer who is considering products in a store or thinking about making a purchase.

[0493] "Movement" refers to the physical movements and actions that users perform within the store.

[0494] "Facial expressions" refer to changes in the user's face that indicate emotions or intentions.

[0495] "Dwell time" refers to how long a user stays in front of a particular area or product.

[0496] "Location" is information indicating where the user is located within the store.

[0497] A "path" is the route a user follows when moving around a store.

[0498] "Behavioral patterns" are specific trends that are identified from a user's series of actions and location information.

[0499] "Purchase intent" refers to the possibility or willingness of a user to purchase a product.

[0500] A "user considering purchasing" is a user whose behavioral patterns indicate a high level of purchasing intent.

[0501] "Notification" refers to a message or alert that the system sends to a salesperson to inform them of specific information.

[0502] "Salesperson" refers to a staff member who explains products and provides support to customers in a store.

[0503] "Smart devices" refers to mobile information terminals, eyeglass-type devices, head-mounted displays, etc. that have communication functions and can connect to the Internet.

[0504] "Layout optimization" means arranging products and customer flow in a store in the most efficient way possible based on user behavior data.

[0505] "Product display optimization" refers to improving the way products are arranged based on product sales and customer interest.

[0506] The present invention provides a system that analyzes customer behavior in a physical store in real time and notifies sales staff of customers with high purchasing intent, thereby improving store management efficiency and customer satisfaction.

[0507] Acquiring camera footage and analyzing video data

[0508] The server collects video data in real time from multiple cameras placed in the store. These cameras are positioned to cover specific areas and shelves, and the video data from each camera is sent to the server. This video data is then analyzed using a computer vision library (e.g., OpenCV) or a machine learning framework (e.g., TensorFlow).

[0509] Detecting user movements and facial expressions

[0510] The server analyzes the user's movements and facial expressions based on the received video data. It uses facial recognition technology and object detection algorithms to determine where the user is and what products they are interested in. For example, if the user is paying attention to a particular product shelf, it can retrieve information about that product from the database.

[0511] Tracking user time spent, location, and path, and learning behavioral patterns

[0512] The server tracks users' time spent and locations in real time. Based on this information, it learns their behavioral patterns and determines whether a particular pattern indicates a purchasing intent. For example, if a user spends a long time on a particular shelf, it determines that the user is interested in that product.

[0513] Identifying and notifying potential purchasers

[0514] The server identifies users who are deemed to have a high intent to purchase based on their behavioral patterns as "purchase-consideration users." This determination is made using behavioral analysis algorithms and machine learning models (e.g., random forests and support vector machines). When a purchase-consideration user is identified, the server immediately sends a notification to a salesperson via a smart device (e.g., smart glasses or smartphone).

[0515] Salesperson's response

[0516] The salesperson receives a notification on the smart device, which includes the user's current location and product information of interest. Upon receiving the notification, the salesperson will go to the user's specified location and provide appropriate assistance. For example, the salesperson may provide detailed explanations about a specific product or suggest additional products.

[0517] Accumulating user behavior data and optimizing layout

[0518] The server stores user behavior data and the results of sales staff responses in a database. Based on this data, it analyzes user patterns throughout the store, understanding the popularity of specific areas and products, and user movement patterns. This makes it possible to optimize layouts and product displays to suit the characteristics of each store. For example, if a specific product shelf is frequently visited, it can suggest adding products to the surrounding area or changing the layout.

[0519] Practical examples and prompts

[0520] For example, if a customer spends a long time considering a particular product, "Product XYZ," in "Aisle 3" of a store, a notification will be sent to a sales associate wearing smart glasses who will provide appropriate assistance.

[0521] Example prompt sentence:

[0522] "Create a system that analyzes customer behavior in real time and sends notifications about customers who are deemed to have high purchasing intent."

[0523] In this way, the present invention provides a system and an operating method for analyzing user behavior in a physical store in real time and maximizing sales opportunities.

[0524] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0525] Step 1:

[0526] The server acquires video data in real time from multiple cameras installed in the store. The video data sent from the cameras is temporarily stored in the server's storage. This processing uses the camera devices, network connections, and storage functions. The input is video data from the cameras, and the output is video data stored on the server.

[0527] Step 2:

[0528] The server analyzes the captured video data to detect the user's movements and facial expressions. This uses a computer vision library (e.g., OpenCV) or a machine learning framework (e.g., TensorFlow). The input is the stored video data, and the output is analytical data on the user's position, movements, and facial expressions. Based on the analysis, the server determines which area the user is in.

[0529] Step 3:

[0530] The server performs processing to track users' time spent, location, and path in real time. Based on this information, it learns user behavior patterns. The inputs are location data, time spent, and path data, and the output is user behavior pattern data. The server uses storage data and real-time analysis data to track user paths.

[0531] Step 4:

[0532] The server analyzes behavioral patterns and runs an algorithm to determine whether users who exhibit certain behavioral patterns are "considerers." This process uses behavioral analysis algorithms and machine learning models (e.g., random forests and support vector machines). The input is behavioral pattern data, and the output is identification data for users considering purchasing.

[0533] Step 5:

[0534] When a "purchase-consideration user" is identified, the server immediately sends a notification to the smart device (e.g., smart glasses, smartphone). The input is the user's identification data, and the output is the notification data sent to the smart device. The server uses a network connection to send the notification.

[0535] Step 6:

[0536] The terminal receives the notification and displays an alert to the salesperson. The salesperson checks the notification on their smart device, goes to the specified user's location, and takes appropriate action to provide support. The input is the notification data sent from the server, and the output is the alert display to the salesperson.

[0537] Step 7:

[0538] The server stores user behavior data and the results of sales staff responses in a database. The stored data is used to optimize store layout and product displays. The input is user behavior data and response result data, and the output is optimization proposal data stored in the database. The server analyzes the stored data and generates optimization proposals for layout and display.

[0539] In this way, specific operations are clarified at each step, and the input and output data are defined for each step. Following this flow allows the entire system to operate effectively.

[0540] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0541] The present invention combines an emotion engine with a system that analyzes the behavior of users (customers) in a store in real time and sends a notification to a salesperson at the appropriate time if it is determined that the user has an intention to purchase. This system recognizes the user's emotions and can provide more accurate support based on that information. Detailed embodiments of the present invention are described below.

[0542] Camera image acquisition and video data analysis

[0543] The device collects video data in real time from multiple cameras in the store. This video data is sent from cameras positioned to cover specific areas and shelves in the store. The server receives this video data and analyzes it in real time.

[0544] Detecting user movements and facial expressions

[0545] The server analyzes the user's movements and facial expressions based on the received video data. Specifically, it uses facial recognition technology and object detection algorithms to determine where the user is and what products they are interested in. This analysis can be performed using open-source computer vision libraries (e.g., OpenCV) and machine learning frameworks (e.g., TensorFlow).

[0546] Introducing the Emotion Engine

[0547] Based on the analyzed facial recognition data, the server uses an emotion engine to recognize the user's emotions. The emotion engine analyzes the user's facial muscle movements and other facial features to determine emotions such as happiness, surprise, anger, sadness, etc. This information, combined with the analysis of movements and facial expressions, is used to more precisely understand the user's overall behavioral patterns.

[0548] Tracking time spent, location, and path, and learning behavioral patterns

[0549] The server tracks users' time spent, location, and route in real time. This allows it to determine how long a user stays at a particular shelf and the path they take. This data is used to learn user behavior patterns. For example, if a user spends a long time in a certain area, it can be determined that they are interested in that area.

[0550] Identifying and notifying potential purchasers

[0551] Based on the user's behavioral patterns and emotional data, the server identifies users who are deemed to have a high intent to purchase as "considering purchase users." For example, if a user spends a long time in front of a particular product shelf, repeatedly picks up a product to check its details, or shows positive emotions (e.g., joy), the server determines that the user has a high intent to purchase.

[0552] When a potential buyer is identified, the server immediately sends a notification to a salesperson, containing information about the user's current location, the product they are interested in, and their current emotional state.

[0553] Salesperson's response

[0554] The salesperson, who is the user, receives the notification on their device and checks the content. Based on the content of the notification, they can contact the specific user and provide appropriate support. For example, they can provide the user with detailed information about the product or suggest other related products. In addition, they can suggest additional services or benefits to users who show particularly positive emotions.

[0555] Accumulating user behavior data and optimizing layout

[0556] The server stores user behavior data and the results of sales staff responses in a database. Based on this data, it analyzes user patterns throughout the store, understanding the popularity of specific areas and products, and user movement patterns. This makes it possible to optimize layouts and product displays to suit the characteristics of each store. For example, if a specific product shelf is frequently visited, it can suggest adding products to the surrounding area or changing the layout.

[0557] Layout optimization example

[0558] For example, suppose there is a specific product area where many users gather on weekends. Based on this data, the server can suggest placing related products around that area or recommend expanding aisles to make user flow smoother. This can improve the user's shopping experience and increase store sales.

[0559] In this way, the present invention provides a system and an operating method for analyzing user behavior and emotions in a store in real time to maximize sales opportunities.

[0560] The processing flow will be explained below.

[0561] Step 1:

[0562] The device captures real-time video data from cameras installed in the store, covering different areas and shelves within the store, and stores multiple video streams in an internal buffer.

[0563] Step 2:

[0564] The video data acquired by the terminal is compressed and sent to the server via the network. The data is compressed using a video codec to improve communication efficiency.

[0565] Step 3:

[0566] The server analyzes the received video data, uses computer vision algorithms to detect the user's movements and facial expressions, uses facial recognition technology to identify the user's face, and uses object recognition technology to identify the product the user is holding.

[0567] Step 4:

[0568] The server uses an emotion engine to recognize the user's emotions from the analyzed facial recognition data. The emotion engine analyzes the user's facial muscle movements and facial features to determine emotions such as joy, surprise, anger, and sadness.

[0569] Step 5:

[0570] The server tracks the user's dwell time, location, and path, collecting data on how long the user spends in a particular area or in front of a shelf, and recording the user's path.

[0571] Step 6:

[0572] The server learns user behavior patterns based on the collected data, and uses machine learning algorithms to analyze user behavior trends and patterns to identify users with strong purchasing intent.

[0573] Step 7:

[0574] The server combines users who exhibit specific behavioral patterns with positive emotions generated by the emotion engine to determine whether they are considering a purchase. For example, if a user spends a long time in front of a particular product shelf, checking product details, or expresses happy emotions, it determines that the user has a strong intention to purchase.

[0575] Step 8:

[0576] When the server identifies a potential buyer, it immediately sends a notification to the salesperson, including the user's current location, the products they are interested in, and their current emotional state.

[0577] Step 9:

[0578] The salesperson, who is the user, receives the notification, reviews its contents, and then contacts the specific user based on the content of the notification to provide appropriate support. For example, the salesperson can provide the user with more information about the product or suggest other related products. Additionally, the salesperson can offer additional services or special offers to users who show particularly positive emotions.

[0579] Step 10:

[0580] The server records the results of salespeople's responses and data on interactions with users, which is used to evaluate the effectiveness of sales activities and build a database that can be used for future responses.

[0581] Step 11:

[0582] The server generates proposals to optimize the store's overall layout and display based on accumulated user behavior and emotion data. It analyzes whether specific areas or products are frequently visited and uses an optimization algorithm to propose layout changes.

[0583] Step 12:

[0584] The store manager, who is the user, reviews the proposals and takes steps to optimize the store layout and displays, thereby achieving efficient store operations and improving customer satisfaction.

[0585] Example 2

[0586] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0587] In modern retail stores, it is difficult for sales staff to accurately grasp users' purchasing intentions and provide timely support. Furthermore, there is a lack of methods for analyzing and utilizing user emotion and behavior data in real time to implement effective sales strategies and optimize layouts. These problems need to be solved.

[0588] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for acquiring video data from a monitoring device installed in the store, a means for analyzing the acquired video data to detect the user's movements and facial expressions, a means for recognizing emotions based on the user's facial expression data, a means for tracking the user's stay time, location, and flow to learn behavioral patterns, a means for determining that a user exhibiting a specific behavioral pattern or emotion is a user considering purchasing, and a means for sending a notification to a notification target when a user considering purchasing is identified. This allows the user's purchasing intention to be properly understood and sales staff to provide timely support. Furthermore, analyzing user behavioral data can optimize store layout and improve sales strategies.

[0589] A "monitoring device" is a device that is placed in a store and captures users' movements and facial expressions in real time.

[0590] "Video data" refers to visual information including the user's movements and facial expressions acquired by a monitoring device.

[0591] "Analysis" is the process of processing video data, recognizing the user's movements and facial expressions, and extracting this as data.

[0592] "Facial expression data" is information that indicates emotions extracted from the user's facial movements and features.

[0593] "Emotion recognition" is the process of analyzing facial expression data to identify the emotion a user is expressing (e.g., joy, surprise, anger, sadness, etc.).

[0594] "Behavioral patterns" are the tendencies and characteristics of user behavior learned based on data such as the user's length of stay, location, and path.

[0595] A "purchase consideration user" is a user who is determined to have a high desire to purchase because they exhibit specific behavioral patterns and emotions based on analysis results and emotion recognition data.

[0596] The "notification recipient" is a person (e.g., a salesperson) who should receive a notification when a user considering purchase is identified.

[0597] A "notification" is a message containing information about a user considering purchasing that is sent from the server to the notification target.

[0598] "Layout optimization" is the process of optimizing the placement and display within a location based on tracked user behavior data.

[0599] This invention is a system that analyzes the behavior of users (customers) in a store in real time, identifies the user's purchase intentions, and notifies the salesperson. By combining this system with an emotion engine that recognizes the user's emotions, it is possible to provide more accurate support.

[0600] Camera image acquisition and video data analysis

[0601] The terminal acquires video data in real time from multiple monitoring devices (e.g., network cameras) placed in the store. The monitoring devices are placed to cover specific areas or product shelves. The server receives this video data and performs analysis. The analysis is performed using Python's OpenCV library and TensorFlow.

[0602] Detecting user movements and facial expressions

[0603] The server analyzes the user's movements and facial expressions based on the received video data. Specifically, it uses facial recognition technology and object detection algorithms to identify the user's location and products they are interested in. This analysis is performed using open-source computer vision libraries (e.g., OpenCV) and machine learning frameworks (e.g., TensorFlow).

[0604] Introducing the Emotion Engine

[0605] Based on the analyzed facial recognition data, the server uses an emotion engine to recognize the user's emotions. The emotion engine analyzes the user's facial muscle movements and other facial features to determine emotions such as happiness, surprise, anger, sadness, etc. This information, combined with the analysis of movements and facial expressions, is used to more precisely understand the user's overall behavioral patterns.

[0606] Tracking user behavior and learning patterns

[0607] The server tracks the user's time spent, location, and path in real time, allowing it to determine how long the user spent at a particular shelf and the path they took. For example, if a user spends 10 minutes in front of a particular shelf, it can determine that they are interested in that product.

[0608] Determining and notifying purchase intent

[0609] Based on the user's behavioral patterns and emotional data, the server identifies users who are deemed to have a strong intent to purchase as "considering purchase users." For example, if a user lingers in front of a particular product shelf for a long time, repeatedly picks up a product to check its details, or shows positive emotions (e.g., joy). When a considering purchase user is identified, the server immediately sends a notification to the salesperson.

[0610] Salesperson's response

[0611] The salesperson, who is the user, receives the notification sent to their device and checks its contents. The notification includes the user's current location, products of interest, and the user's current emotional state. Based on this, the salesperson contacts the specific user and provides detailed product information or suggests related products.

[0612] Accumulating user behavior data and optimizing layout

[0613] The server stores user behavior data and the results of sales staff responses in a database. This data is then analyzed to determine user patterns throughout the store. By understanding the popularity of specific areas and products, as well as user movement patterns, it becomes possible to optimize layouts and product displays to suit the characteristics of each store. For example, based on data showing that a specific product area is frequently visited on weekends, it can propose changes to the layout of that area and the placement of related products.

[0614] Examples of concrete examples and prompts

[0615] Examples:

[0616] For example, a user enters a store and heads to the food section. The device receives video data via a camera installed in the device, which is then analyzed by a server. The server recognizes that the user spends a long time in front of the shelves in the food section, picking up and looking at several products. It combines this information with emotional data to determine that the user is interested in those products. The server then sends a notification to a salesperson, who then checks the user's location, products of interest, and emotional state on the device. The salesperson then contacts the user, explains the details of the products, and suggests related products.

[0617] Example prompt for a generative AI model:

[0618] "What is the appropriate way to respond when a customer lingers in front of a particular shelf in a store for a long time?"

[0619] "Please explain the specific method for recognizing user emotions and determining purchasing intent from video data."

[0620] As described above, the present invention provides a system and an operating method for analyzing user behavior and emotions in real time to maximize sales opportunities.

[0621] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0622] Step 1: Acquire camera footage

[0623] The terminal acquires video data from surveillance equipment installed in the store. The surveillance equipment uses network cameras, each positioned to cover a specific area or shelf. For example, a camera captures a food section, and the video data is streamed to the terminal. The input is real-time video data, and the output is video data sent to a server via the terminal.

[0624] Step 2: Analyzing the video data

[0625] The server receives video data sent from the device. The server uses Python's OpenCV library to extract frames of the video data and perform preprocessing such as noise removal. Next, it uses TensorFlow to analyze the user's movements and facial expressions. For example, the server analyzes frames every second to detect the movements of the user's face and hands. The input is the captured video data, and the output is the analyzed user's movements and facial expression data.

[0626] Step 3: Detecting user movements and facial expressions

[0627] The server uses OpenCV's Haar Cascade and TensorFlow to identify the position of the user's face and hands and detect changes in facial expressions and movements. For example, it recognizes the moment when the user smiles or picks up a specific product. The input is the video frame extracted in the previous step, and the output is specific data on the user's movements and facial expressions.

[0628] Step 4: Emotion Recognition with the Emotion Engine

[0629] The server uses Affectiva's API to send the facial expression data obtained in the previous step and recognize the user's emotions in real time. The emotion engine determines emotions (happiness, surprise, anger, sadness, etc.) from the user's facial muscle movements and facial features. The input is the user's facial expression data, and the output is the recognized emotion data.

[0630] Step 5: Track user behavior and learn patterns

[0631] The server tracks the user's time spent, location, and path in real time and records this data. For example, the database stores information such as how long the user stayed in front of a particular shelf and the path they took. The input is the user's location and path data, and the output is behavioral patterns and learned user data.

[0632] Step 6: Determine and communicate purchase intent

[0633] The server analyzes the user's behavioral patterns and emotional data, and identifies users who are determined to have a strong intention to purchase as "considering purchase users." For example, this applies when a user lingers in front of a particular product shelf for a long time, picks up the product to check its details, or shows positive emotions (joy). The input is behavioral pattern data and emotional data, and the output is a notification to the salesperson.

[0634] Step 7: Salesperson response

[0635] The salesperson, who is the user, receives the notification sent to the terminal and checks the content. The notification includes the user's current location, products of interest, and emotional state. Based on this, the salesperson contacts the user to provide detailed product information and suggest related products. The input is the notification data from the server, and the output is specific support for the user.

[0636] Step 8: Accumulating user behavior data and optimizing layout

[0637] The server stores user behavior data and the results of sales staff responses in a database. This data is analyzed to understand the popularity of specific areas and products, as well as user movement patterns. For example, if a specific product area is frequently visited on weekends, the server will suggest changes to the layout of that area and the placement of related products. The input is the stored behavioral data, and the output is specific proposals for layout optimization.

[0638] Through these steps, the system can analyze user behavior and emotions in real time, enabling effective sales strategies and store layout optimization.

[0639] (Application example 2)

[0640] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0641] In traditional brick-and-mortar stores, it was difficult to analyze customer behavior and emotions in real time and provide appropriate support at the right time. This made it difficult to effectively stimulate customer purchasing desire, limiting store sales. Furthermore, data analysis to improve customer experience was insufficient, and store layout and product display were not optimized.

[0642] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0643] In this invention, the server includes: means for acquiring video data from cameras installed in the store; means for analyzing the acquired video data to detect user movements and facial expressions; means for tracking user stay time, location, and flow to learn behavioral patterns; means for recognizing user emotions based on the analyzed facial recognition data; means for determining that a user exhibiting a specific behavioral pattern and emotional state is a user considering a purchase; and means for sending a notification to a salesperson when a user considering a purchase is identified. This allows for timely notification to a salesperson based on the customer's behavior and emotions, enabling effective customer service. Furthermore, it is possible to optimize the store layout and product display based on user behavior data and emotional data, improving the customer experience and increasing sales.

[0644] "In-store" refers to the physical location where customers can actually touch and purchase products.

[0645] A "camera" is a visual sensor device for acquiring video data.

[0646] "Video data" refers to the digital information of images or video captured by a camera.

[0647] A "user" is a customer who performs actions such as viewing products in a store or making a purchase.

[0648] "Movement" refers to the movement or change in position of the user's body.

[0649] "Facial expressions" are the emotional expressions and muscle movements that appear on the user's face.

[0650] "Dwell time" refers to the amount of time a user stays in a particular location or on a shelf.

[0651] "Location" is information that indicates which area of ​​the store the user is in.

[0652] "Pathways" refer to the routes and patterns that users take within a store.

[0653] "Behavioral patterns" refer to habitual behavior that can be inferred from a user's series of actions, changes in location, length of stay, etc.

[0654] A "purchase consideration user" is a user who is determined to have a strong intention to purchase based on their behavioral patterns and emotional state.

[0655] A "notification" is a message sent to convey specific information.

[0656] A "salesperson" is a staff member who explains products and provides support to customers in a store.

[0657] "Facial recognition data" is digital information used to identify and analyze a user's face.

[0658] "Emotion" refers to a user's psychological state as judged from their facial expressions and behavior.

[0659] A "server" is a computer system that analyzes video data, stores various data, and sends notifications.

[0660] "Optimization" refers to the efficient rearrangement of layout and product displays based on data analysis.

[0661] The present invention relates to a system that analyzes customer behavior and emotions in real time in a store and provides appropriate support at the appropriate time. This system is realized by capturing video data using cameras placed in the store and analyzing that video data.

[0662] Acquiring camera footage

[0663] The server collects real-time video data from multiple cameras placed throughout the store, which are positioned to cover specific areas and shelves, allowing for comprehensive monitoring of customer behavior.

[0664] Video data analysis

[0665] The server analyzes the captured video data to detect the customer's movements and facial expressions. This analysis uses open-source computer vision libraries and machine learning frameworks such as OpenCV and TensorFlow. Specifically, it uses a facial recognition model to detect the customer's facial expressions and recognizes the customer's emotions based on the facial expression data.

[0666] Introducing the Emotion Engine

[0667] The server recognizes the customer's emotions using an emotion engine based on the analyzed facial recognition data. The emotion engine analyzes the customer's facial muscle movements and other facial features to determine emotions such as happiness, surprise, anger, sadness, etc. This information, combined with the analysis of movements and facial expressions, is used to more precisely understand the customer's overall behavioral patterns.

[0668] Tracking user time, location, and path

[0669] The server tracks customer dwell time, location, and path in real time, allowing it to determine how long a customer spends at a particular shelf and the path they take. This data is used to learn user behavior patterns.

[0670] Identifying and notifying potential purchasers

[0671] Based on the user's behavioral patterns and emotional data, the server identifies users who are determined to have a high intent to purchase as "purchase-consideration users." For example, if a user spends a long time in front of a particular product shelf, repeatedly picks up products to check their details, or shows positive emotions (e.g., joy), the server determines that the user has a high intent to purchase. When a user is identified as a purchase-consideration user, the server immediately sends a notification to the salesperson. The notification includes information about the user's current location, the products they are interested in, and their current emotional state.

[0672] Salesperson's response

[0673] Salespeople receive notifications on their devices, review the content, and contact specific users based on the content of the notifications to provide appropriate support. For example, they can provide detailed product information or suggest other related products. Additionally, they can offer additional services or special offers to users who show particularly positive emotions.

[0674] Accumulating user behavior data and optimizing layout

[0675] The server stores user behavior data and sales staff response results in a database. Based on this data, it analyzes user patterns throughout the store, understanding the popularity of specific areas and products, and user movement patterns. This makes it possible to optimize layouts and product displays to suit the characteristics of each store.

[0676] Examples of concrete examples and prompts

[0677] As a specific example, if customer A is seen smiling in the in-store camera footage and spends a long time in front of a particular product, the application will notify the salesperson of this information so that the salesperson can provide appropriate assistance.

[0678] Example prompt for a generative AI model:

[0679] "We will develop a system that analyzes video footage from multiple cameras in real time, recognizes faces and emotions, determines the purchasing intent of customers in a specific area, and notifies sales staff at the appropriate time."

[0680] Thus, the present invention provides a system and an operating method thereof for analyzing customer behavior and emotions in real time and maximizing sales opportunities.

[0681] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0682] Step 1:

[0683] Acquiring video data from the camera

[0684] The server receives real-time video data from cameras placed around the store, capturing images of specific areas of the store and product shelves. It receives the video data from the cameras as input and converts it into a processable format as output.

[0685] Step 2:

[0686] Video data analysis

[0687] The server detects the user's movements and facial expressions based on the acquired video data. Specifically, it uses OpenCV to perform face detection and identify the user's location. The input is video data from the camera, and the output is the location information of the detected face.

[0688] Step 3:

[0689] Emotion recognition based on facial recognition data

[0690] The server uses facial recognition data to recognize the user's emotions. Using a TensorFlow model, it analyzes the user's facial muscle movements and other features to determine emotions such as joy, surprise, and sadness. The input is the facial position information and video data obtained in step 2, and the output is the user's emotional state.

[0691] Step 4:

[0692] Tracking time spent, location, and traffic

[0693] The server tracks users' time spent, location, and route in real time, allowing it to determine how long they stay in a particular area and the route they take. The input is video data from the camera and facial recognition data, and the output is user behavior pattern data.

[0694] Step 5:

[0695] Identifying and notifying potential purchasers

[0696] The server identifies users who are deemed to have a high intent to purchase based on their behavioral patterns and emotional data as "consideration users." If identified, it sends a notification to the salesperson containing information about the user's current location, products they are interested in, and their emotional state. The input is the data obtained in Step 4 and Step 3, and the output is a notification to the salesperson.

[0697] Step 6:

[0698] Salesperson's response

[0699] The terminal allows the salesperson to receive the notification and provide support to the user based on its content. Specifically, it provides detailed product information and suggests other related products to the user. In particular, it suggests additional services and benefits to users who show positive emotions. The input is the notification sent to the salesperson's terminal, and the output is the result of the support provided.

[0700] Step 7:

[0701] Accumulating user behavior data and optimizing layout

[0702] The server accumulates user behavior data and salesperson response results in a database and analyzes user patterns throughout the store. This allows the popularity of specific areas within the store and user movement patterns to be understood, and layout optimization proposals are generated. The input is the data obtained at each step, and the output is layout and display optimization proposals.

[0703] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0704] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0705] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0706] [Third embodiment]

[0707] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0708] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0709] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0710] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0711] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0712] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0713] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0714] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0715] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0716] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0717] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0718] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0719] The present invention provides a system that analyzes the behavior of users (customers) in a store in real time and sends a notification to sales staff at an appropriate time when it is determined that the user has a purchase intention.The system also makes suggestions for optimizing store layout and product displays based on the accumulated customer behavior data.Detailed embodiments of the present invention are described below.

[0720] Camera image acquisition and video data analysis

[0721] The device collects video data in real time from multiple cameras in the store. This video data is sent from cameras positioned to cover specific areas and shelves in the store. The server receives this video data and analyzes it in real time.

[0722] Detecting user movements and facial expressions

[0723] The server analyzes the user's movements and facial expressions based on the received video data. Specifically, it uses facial recognition technology and object detection algorithms to determine where the user is and what products they are interested in. This analysis can be performed using open-source computer vision libraries (e.g., OpenCV) and machine learning frameworks (e.g., TensorFlow).

[0724] Tracking time spent, location, and path, and learning behavioral patterns

[0725] The server tracks users' time spent, location, and route in real time. This allows it to determine how long a user stays at a particular shelf and the path they take. This data is used to learn user behavior patterns. For example, if a user spends a long time in a certain area, it can be determined that they are interested in that area.

[0726] Identifying and notifying potential purchasers

[0727] Based on the user's behavioral patterns, the server identifies users who are deemed to have a high intent to purchase as "purchase-consideration users." This identification is performed using behavioral analysis algorithms and machine learning models (e.g., random forests and support vector machines). When a user is identified as a purchase-consideration user, the server immediately sends a notification to the salesperson via the terminal. This notification includes information about the user's current location and the products they are interested in.

[0728] Salesperson's response

[0729] The salesperson, who is the user, receives a notification on their device. Upon receiving the notification, the salesperson will go to the user's designated location and provide appropriate support, such as providing detailed explanations about a specific product or suggesting additional products.

[0730] Accumulating user behavior data and optimizing layout

[0731] The server stores user behavior data and the results of sales staff responses in a database. Based on this data, it analyzes user patterns throughout the store, understanding the popularity of specific areas and products, and user movement patterns. This makes it possible to optimize layouts and product displays to suit the characteristics of each store. For example, if a specific product shelf is frequently visited, it can suggest adding products to the surrounding area or changing the layout.

[0732] Layout optimization example

[0733] For example, suppose there is a specific product area where many users gather on weekends. Based on this data, the server can suggest placing related products around that area or recommend expanding aisles to make user flow smoother. This can improve the user's shopping experience and increase store sales.

[0734] In this way, the present invention provides a system and an operating method for analyzing user behavior in a store in real time and maximizing sales opportunities.

[0735] The processing flow will be explained below.

[0736] Step 1:

[0737] The device captures real-time video data from cameras installed in the store, covering different areas and shelves within the store, and stores multiple video streams in an internal buffer.

[0738] Step 2:

[0739] The video data acquired by the terminal is compressed and sent to the server via the network. The data is compressed using a video codec to improve communication efficiency.

[0740] Step 3:

[0741] The server analyzes the received video data, uses computer vision algorithms to detect the user's movements and facial expressions, uses facial recognition technology to identify the user's face, and uses object recognition technology to identify the product the user is holding.

[0742] Step 4:

[0743] The server tracks the user's dwell time, location, and path, collecting data on how long the user spends in a particular area or in front of a shelf, and recording the user's path.

[0744] Step 5:

[0745] The server learns user behavior patterns based on the collected data, and uses machine learning algorithms to analyze user behavior trends and patterns to identify users with strong purchasing intent.

[0746] Step 6:

[0747] The server determines that users who exhibit certain behavioral patterns are considering a purchase. For example, if a user repeatedly spends a long time in front of a particular product shelf, picking up a product and checking its details, it is determined that the user has a strong intention to purchase.

[0748] Step 7:

[0749] If the server identifies a potential buyer, it immediately sends a notification to a salesperson, including information about the user's current location and the product they are interested in.

[0750] Step 8:

[0751] The salesperson, who is the user, receives the notification, reviews the content, and then contacts the specific user based on the content of the notification to provide appropriate support, such as providing the user with detailed information about the product or suggesting other related products.

[0752] Step 9:

[0753] The server records the results of salespeople's responses and data on interactions with users, which allows the effectiveness of sales activities to be evaluated and a database to be built for future use.

[0754] Step 10:

[0755] Based on the accumulated user behavior data, the server generates proposals to optimize the layout and display of each store. It analyzes whether specific areas or products are frequently visited and uses an optimization algorithm to propose layout changes.

[0756] Step 11:

[0757] The store manager, who is the user, reviews the proposals and takes steps to optimize the store layout and displays, thereby achieving efficient store operations and improving customer satisfaction.

[0758] Example 1

[0759] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0760] In conventional stores, there are limited means of understanding customer behavior patterns and purchasing intentions in real time, making it difficult to provide efficient sales support and effectively optimize store layouts. Therefore, there is a need for a method that can accurately grasp customer purchasing intentions to avoid missing sales opportunities, as well as to create effective store layout plans based on customer behavior data and provide a better shopping experience.

[0761] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0762] In this invention, the server includes a means for acquiring video data from a camera installed in the store, a means for analyzing the acquired video data to detect customer movements and facial expressions, a means for tracking customer stay time, location, and flow to learn behavioral patterns, a means for determining whether a customer exhibiting a specific behavioral pattern is a potential purchaser, and a means for sending a notification to a salesperson when a potential purchaser is identified. This allows for efficient sales support by grasping customer purchasing intentions in real time and notifying salespersons at the appropriate time. Furthermore, the server can propose optimization of store layout and product display based on customer behavior data, which is expected to improve customer satisfaction and increase store sales.

[0763] A "camera" is a device installed in a store to monitor and record customer behavior.

[0764] "Video data" refers to images and video information captured by a photographing device.

[0765] "Customer" means a person browsing or purchasing products in a store.

[0766] "Movement" refers to the physical movements and actions that customers perform in the store.

[0767] "Facial expressions" refer to the customer's facial expressions and gestures that show emotion.

[0768] "Dwell time" refers to the amount of time a customer spends in a particular area or in front of a shelf.

[0769] "Location" is information indicating where the customer is located within the store.

[0770] "Flow" refers to the route or pattern that customers take within a store.

[0771] "Behavioral patterns" refer to the trends and characteristics of customer behavior analyzed based on data such as customer length of stay, location, and route.

[0772] "Considering purchase customers" are customers whose behavioral patterns indicate a high level of purchasing intent.

[0773] A "salesperson" is an employee who guides and explains products to customers in a store.

[0774] A "notification" is a message or alert sent from the server to a salesperson.

[0775] The "server" is a central processing unit that analyzes video data and tracks and records customer behavior.

[0776] "Layout" refers to the layout plan within the store and the placement and arrangement of products.

[0777] "Product display" refers to the display and arrangement of products within a store.

[0778] The present invention provides a system that analyzes customer behavior in a store in real time and sends a notification to a salesperson at an appropriate time when it is determined that the customer has a purchase intention. The system also proposes optimization of store layout and product display based on the accumulated customer behavior data. Detailed embodiments of the present invention are described below.

[0779] Acquiring camera footage

[0780] The device acquires video data in real time from multiple cameras placed in the store. These cameras are positioned to cover specific areas or product shelves within the store. For example, cameras can be placed near key product shelves or store entrances to monitor customer behavior from all directions.

[0781] Video data analysis

[0782] The server receives and analyzes video data sent from the device in real time. The video data is processed frame by frame, and pre-processing is performed to identify customer movements and positions. Specifically, the analysis is performed using open-source computer vision libraries (e.g., OpenCV) and machine learning frameworks (e.g., TensorFlow).

[0783] User behavior and facial expression detection

[0784] The server detects the customer's movements and facial expressions from the video data. It uses facial recognition technology to identify the customer's face, and an object detection algorithm to identify the customer's movements and products of interest. For example, by combining facial recognition using OpenCV and object detection using TensorFlow, it can analyze which product the customer is in front of and how they are moving.

[0785] Tracking time spent, location, and traffic

[0786] The server tracks the customer's time spent, location, and movement path (pathway). This makes it possible to determine how long a customer stayed at a particular shelf and which path they took. For example, if a customer stays in front of a particular shelf for five minutes, the server can use this data to analyze the customer's behavioral patterns.

[0787] Determining users considering purchasing

[0788] Based on the customer's behavioral patterns, the server identifies customers who are judged to have a high intent to purchase as "considering purchase." This identification is performed using behavioral analysis algorithms and machine learning models (e.g., random forests and support vector machines). When a considering purchase customer is identified, the server immediately sends a notification to the salesperson via the terminal.

[0789] Notification to sales staff

[0790] The server notifies the salesperson via the terminal of information about the identified potential purchaser, including details about the customer's current location and the product they are interested in. For example, the terminal may be notified that "Customer A is interested in the product on shelf B."

[0791] Salesperson's response

[0792] The salesperson, who is the user, receives a notification from the terminal. Upon receiving the notification, the salesperson quickly goes to the specified customer's location and provides appropriate support, such as providing detailed explanations about the product or suggesting additional items.

[0793] Accumulation and analysis of user behavior data

[0794] The server stores all customer behavior data and the results of sales staff responses in a database, making it possible to analyze customer behavior patterns across the entire store.

[0795] Layout optimization proposals

[0796] The server generates suggestions for improving store layout and product displays based on the accumulated behavioral data, such as "Since a particular area is crowded on weekends, place related products around that area" or "Expand aisles to streamline customer flow."

[0797] Examples of prompt statements

[0798] The following is a specific example of a prompt sentence to be input to the generative AI model:

[0799] "Write a program that analyzes video data obtained from a camera and uses an algorithm to identify behavioral patterns based on the customer's movements and facial expressions to detect customers who intend to make a purchase in real time. Include a function to send a notification to a salesperson if a customer shows interest in a product on the shelf. Use the OpenCV and TensorFlow libraries."

[0800] As described above, the present invention provides a system and an operating method thereof for analyzing customer behavior in a store in real time and maximizing sales opportunities.

[0801] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0802] System program processing flow

[0803] Step 1: Acquire camera footage

[0804] The terminal acquires video data in real time from multiple cameras placed in the store. These cameras are positioned to cover specific areas and product shelves, monitoring customer behavior from all directions. The input is video data from the cameras, and the output is raw video data stored in the terminal.

[0805] Step 2: Sending video data

[0806] The video data acquired by the device is sent to the server in real time. Here, the input is the raw video data stored on the device, and the output is the video data to be sent to the server. For example, the setting is to send the data to the server periodically, frame by frame.

[0807] Step 3: Analyzing the video data

[0808] The server processes the received video data frame by frame and performs preprocessing to identify the customer's movements and position. Specifically, it uses OpenCV to preprocess the images, remove noise, and resize the images. The input is the video data sent to the server, and the output is the preprocessed image data.

[0809] Step 4: Detect customer behavior and facial expressions

[0810] The server detects the customer's movements and facial expressions from the preprocessed image data. It uses facial recognition technology to identify the customer's face and TensorFlow to perform object detection and movement recognition. The input is the preprocessed image data, and the output is analytical data on the customer's movements and facial expressions. For example, it detects whether the customer is smiling or looking at a specific product.

[0811] Step 5: Tracking dwell time, location, and path

[0812] The server records customer location information in real time and tracks their time spent and their path (pathway). The input is the detected customer location data, and the output is behavioral data including the customer's time spent, location, and path. For example, it tracks how long a customer stays in front of a particular product.

[0813] Step 6: Identify potential buyers

[0814] Based on customer behavior data, the server identifies customers who are determined to have a high intent to purchase as "considering purchase customers." It uses behavioral analysis algorithms such as random forests and support vector machines. The input is customer behavior data, and the output is the identification data of customers considering purchase. For example, it may identify "Customer A is interested in Product B."

[0815] Step 7: Notify salespeople

[0816] The server notifies the salesperson via the terminal of the information about the identified customer who is considering purchasing. The input is the specific data of the customer who is considering purchasing, and the output is the notification data sent to the salesperson's terminal. For example, the notification may say, "Customer A is interested in the product on shelf B."

[0817] Step 8: Salesperson response

[0818] The salesperson, who is the user, receives a notification from the terminal and quickly goes to the specified customer's location to provide appropriate support. The input is the notification data, and the output is the action of providing support to the customer, such as providing detailed explanations about the product or suggesting additional products.

[0819] Step 9: Collect and analyze user behavior data

[0820] The server stores all customer behavior data and salesperson response results in a database. The input is customer behavior data and salesperson response data, and the output is the stored database.

[0821] Step 10: Layout optimization proposal

[0822] The server generates suggestions for improving store layout and product displays based on the accumulated behavioral data. The input is the accumulated behavioral data, and the output is a layout optimization proposal. For example, it may suggest, "Since a particular area is crowded on weekends, place related products around that area."

[0823] As a result, the system of the present invention provides a system and an operating method for analyzing customer behavior in a store in real time and maximizing sales opportunities.

[0824] (Application example 1)

[0825] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0826] In recent years, there has been a demand for optimizing customer service and improving the customer experience in physical stores. However, the current situation lacks a system that can analyze customer behavior and purchasing intent in real time and send notifications to sales staff at the appropriate time. This can lead to missed sales opportunities and dissatisfaction among customers who do not receive the support they need. Furthermore, store layout and product display are not sufficiently optimized, making it difficult to increase sales and operate efficiently.

[0827] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0828] In this invention, the server includes: means for acquiring video data from cameras installed in the store; means for analyzing the acquired video data to detect user movements and facial expressions; means for tracking user time spent, location, and flow to learn behavioral patterns; means for determining whether a user exhibiting a specific behavioral pattern is a potential purchaser; means for sending a notification to a salesperson when a potential purchaser is identified; and means for sending a notification to the salesperson in real time using a smart device. This allows salespersons to provide support to customers in a timely manner. Furthermore, store layout and product display can be optimized based on user behavior data, improving store operational efficiency and leading to increased sales.

[0829] "Video data" refers to visual information acquired by a camera or other imaging device, and is data in the form of images or videos.

[0830] "Analysis" refers to processing acquired data to extract and understand specific information.

[0831] A "user" is a customer who is considering products in a store or thinking about making a purchase.

[0832] "Movement" refers to the physical movements and actions that users perform within the store.

[0833] "Facial expressions" refer to changes in the user's face that indicate emotions or intentions.

[0834] "Dwell time" refers to how long a user stays in front of a particular area or product.

[0835] "Location" is information indicating where the user is located within the store.

[0836] A "path" is the route a user follows when moving around a store.

[0837] "Behavioral patterns" are specific trends that are identified from a user's series of actions and location information.

[0838] "Purchase intent" refers to the possibility or willingness of a user to purchase a product.

[0839] A "user considering purchasing" is a user whose behavioral patterns indicate a high level of purchasing intent.

[0840] "Notification" refers to a message or alert that the system sends to a salesperson to inform them of specific information.

[0841] "Salesperson" refers to a staff member who explains products and provides support to customers in a store.

[0842] "Smart devices" refers to mobile information terminals, eyeglass-type devices, head-mounted displays, etc. that have communication functions and can connect to the Internet.

[0843] "Layout optimization" means arranging products and customer flow in a store in the most efficient way possible based on user behavior data.

[0844] "Product display optimization" refers to improving the way products are arranged based on product sales and customer interest.

[0845] The present invention provides a system that analyzes customer behavior in a physical store in real time and notifies sales staff of customers with high purchasing intent, thereby improving store management efficiency and customer satisfaction.

[0846] Acquiring camera footage and analyzing video data

[0847] The server collects video data in real time from multiple cameras placed in the store. These cameras are positioned to cover specific areas and shelves, and the video data from each camera is sent to the server. This video data is then analyzed using a computer vision library (e.g., OpenCV) or a machine learning framework (e.g., TensorFlow).

[0848] Detecting user movements and facial expressions

[0849] The server analyzes the user's movements and facial expressions based on the received video data. It uses facial recognition technology and object detection algorithms to determine where the user is and what products they are interested in. For example, if the user is paying attention to a particular product shelf, it can retrieve information about that product from the database.

[0850] Tracking user time spent, location, and path, and learning behavioral patterns

[0851] The server tracks users' time spent and locations in real time. Based on this information, it learns their behavioral patterns and determines whether a particular pattern indicates a purchasing intent. For example, if a user spends a long time on a particular shelf, it determines that the user is interested in that product.

[0852] Identifying and notifying potential purchasers

[0853] The server identifies users who are deemed to have a high intent to purchase based on their behavioral patterns as "purchase-consideration users." This determination is made using behavioral analysis algorithms and machine learning models (e.g., random forests and support vector machines). When a purchase-consideration user is identified, the server immediately sends a notification to a salesperson via a smart device (e.g., smart glasses or smartphone).

[0854] Salesperson's response

[0855] The salesperson receives a notification on the smart device, which includes the user's current location and product information of interest. Upon receiving the notification, the salesperson will go to the user's specified location and provide appropriate assistance. For example, the salesperson may provide detailed explanations about a specific product or suggest additional products.

[0856] Accumulating user behavior data and optimizing layout

[0857] The server stores user behavior data and the results of sales staff responses in a database. Based on this data, it analyzes user patterns throughout the store, understanding the popularity of specific areas and products, and user movement patterns. This makes it possible to optimize layouts and product displays to suit the characteristics of each store. For example, if a specific product shelf is frequently visited, it can suggest adding products to the surrounding area or changing the layout.

[0858] Practical examples and prompts

[0859] For example, if a customer spends a long time considering a particular product, "Product XYZ," in "Aisle 3" of a store, a notification will be sent to a sales associate wearing smart glasses who will provide appropriate assistance.

[0860] Example prompt sentence:

[0861] "Create a system that analyzes customer behavior in real time and sends notifications about customers who are deemed to have high purchasing intent."

[0862] In this way, the present invention provides a system and an operating method for analyzing user behavior in a physical store in real time and maximizing sales opportunities.

[0863] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0864] Step 1:

[0865] The server acquires video data in real time from multiple cameras installed in the store. The video data sent from the cameras is temporarily stored in the server's storage. This processing uses the camera devices, network connections, and storage functions. The input is video data from the cameras, and the output is video data stored on the server.

[0866] Step 2:

[0867] The server analyzes the captured video data to detect the user's movements and facial expressions. This uses a computer vision library (e.g., OpenCV) or a machine learning framework (e.g., TensorFlow). The input is the stored video data, and the output is analytical data on the user's position, movements, and facial expressions. Based on the analysis, the server determines which area the user is in.

[0868] Step 3:

[0869] The server performs processing to track users' time spent, location, and path in real time. Based on this information, it learns user behavior patterns. The inputs are location data, time spent, and path data, and the output is user behavior pattern data. The server uses storage data and real-time analysis data to track user paths.

[0870] Step 4:

[0871] The server analyzes behavioral patterns and runs an algorithm to determine whether users who exhibit certain behavioral patterns are "considerers." This process uses behavioral analysis algorithms and machine learning models (e.g., random forests and support vector machines). The input is behavioral pattern data, and the output is identification data for users considering purchasing.

[0872] Step 5:

[0873] When a "purchase-consideration user" is identified, the server immediately sends a notification to the smart device (e.g., smart glasses, smartphone). The input is the user's identification data, and the output is the notification data sent to the smart device. The server uses a network connection to send the notification.

[0874] Step 6:

[0875] The terminal receives the notification and displays an alert to the salesperson. The salesperson checks the notification on their smart device, goes to the specified user's location, and takes appropriate action to provide support. The input is the notification data sent from the server, and the output is the alert display to the salesperson.

[0876] Step 7:

[0877] The server stores user behavior data and the results of sales staff responses in a database. The stored data is used to optimize store layout and product displays. The input is user behavior data and response result data, and the output is optimization proposal data stored in the database. The server analyzes the stored data and generates optimization proposals for layout and display.

[0878] In this way, specific operations are clarified at each step, and the input and output data are defined for each step. Following this flow allows the entire system to operate effectively.

[0879] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0880] The present invention combines an emotion engine with a system that analyzes the behavior of users (customers) in a store in real time and sends a notification to a salesperson at the appropriate time if it is determined that the user has an intention to purchase. This system recognizes the user's emotions and can provide more accurate support based on that information. Detailed embodiments of the present invention are described below.

[0881] Camera image acquisition and video data analysis

[0882] The device collects video data in real time from multiple cameras in the store. This video data is sent from cameras positioned to cover specific areas and shelves in the store. The server receives this video data and analyzes it in real time.

[0883] Detecting user movements and facial expressions

[0884] The server analyzes the user's movements and facial expressions based on the received video data. Specifically, it uses facial recognition technology and object detection algorithms to determine where the user is and what products they are interested in. This analysis can be performed using open-source computer vision libraries (e.g., OpenCV) and machine learning frameworks (e.g., TensorFlow).

[0885] Introducing the Emotion Engine

[0886] Based on the analyzed facial recognition data, the server uses an emotion engine to recognize the user's emotions. The emotion engine analyzes the user's facial muscle movements and other facial features to determine emotions such as happiness, surprise, anger, sadness, etc. This information, combined with the analysis of movements and facial expressions, is used to more precisely understand the user's overall behavioral patterns.

[0887] Tracking time spent, location, and path, and learning behavioral patterns

[0888] The server tracks users' time spent, location, and route in real time. This allows it to determine how long a user stays at a particular shelf and the path they take. This data is used to learn user behavior patterns. For example, if a user spends a long time in a certain area, it can be determined that they are interested in that area.

[0889] Identifying and notifying potential purchasers

[0890] Based on the user's behavioral patterns and emotional data, the server identifies users who are deemed to have a high intent to purchase as "considering purchase users." For example, if a user spends a long time in front of a particular product shelf, repeatedly picks up a product to check its details, or shows positive emotions (e.g., joy), the server determines that the user has a high intent to purchase.

[0891] When a potential buyer is identified, the server immediately sends a notification to a salesperson, containing information about the user's current location, the product they are interested in, and their current emotional state.

[0892] Salesperson's response

[0893] The salesperson, who is the user, receives the notification on their device and checks the content. Based on the content of the notification, they can contact the specific user and provide appropriate support. For example, they can provide the user with detailed information about the product or suggest other related products. In addition, they can suggest additional services or benefits to users who show particularly positive emotions.

[0894] Accumulating user behavior data and optimizing layout

[0895] The server stores user behavior data and the results of sales staff responses in a database. Based on this data, it analyzes user patterns throughout the store, understanding the popularity of specific areas and products, and user movement patterns. This makes it possible to optimize layouts and product displays to suit the characteristics of each store. For example, if a specific product shelf is frequently visited, it can suggest adding products to the surrounding area or changing the layout.

[0896] Layout optimization example

[0897] For example, suppose there is a specific product area where many users gather on weekends. Based on this data, the server can suggest placing related products around that area or recommend expanding aisles to make user flow smoother. This can improve the user's shopping experience and increase store sales.

[0898] In this way, the present invention provides a system and an operating method for analyzing user behavior and emotions in a store in real time to maximize sales opportunities.

[0899] The processing flow will be explained below.

[0900] Step 1:

[0901] The device captures real-time video data from cameras installed in the store, covering different areas and shelves within the store, and stores multiple video streams in an internal buffer.

[0902] Step 2:

[0903] The video data acquired by the terminal is compressed and sent to the server via the network. The data is compressed using a video codec to improve communication efficiency.

[0904] Step 3:

[0905] The server analyzes the received video data, uses computer vision algorithms to detect the user's movements and facial expressions, uses facial recognition technology to identify the user's face, and uses object recognition technology to identify the product the user is holding.

[0906] Step 4:

[0907] The server uses an emotion engine to recognize the user's emotions from the analyzed facial recognition data. The emotion engine analyzes the user's facial muscle movements and facial features to determine emotions such as joy, surprise, anger, and sadness.

[0908] Step 5:

[0909] The server tracks the user's dwell time, location, and path, collecting data on how long the user spends in a particular area or in front of a shelf, and recording the user's path.

[0910] Step 6:

[0911] The server learns user behavior patterns based on the collected data, and uses machine learning algorithms to analyze user behavior trends and patterns to identify users with strong purchasing intent.

[0912] Step 7:

[0913] The server combines users who exhibit specific behavioral patterns with positive emotions generated by the emotion engine to determine whether they are considering a purchase. For example, if a user spends a long time in front of a particular product shelf, checking product details, or expresses happy emotions, it determines that the user has a strong intention to purchase.

[0914] Step 8:

[0915] When the server identifies a potential buyer, it immediately sends a notification to the salesperson, including the user's current location, the products they are interested in, and their current emotional state.

[0916] Step 9:

[0917] The salesperson, who is the user, receives the notification, reviews its contents, and then contacts the specific user based on the content of the notification to provide appropriate support. For example, the salesperson can provide the user with more information about the product or suggest other related products. Additionally, the salesperson can offer additional services or special offers to users who show particularly positive emotions.

[0918] Step 10:

[0919] The server records the results of salespeople's responses and data on interactions with users, which allows the effectiveness of sales activities to be evaluated and a database to be built for future use.

[0920] Step 11:

[0921] The server generates proposals to optimize the store's overall layout and display based on accumulated user behavior and emotion data. It analyzes whether specific areas or products are frequently visited and uses an optimization algorithm to propose layout changes.

[0922] Step 12:

[0923] The store manager, who is the user, reviews the proposals and takes steps to optimize the store layout and displays, thereby achieving efficient store operations and improving customer satisfaction.

[0924] Example 2

[0925] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0926] In modern retail stores, it is difficult for sales staff to accurately grasp users' purchasing intentions and provide timely support. Furthermore, there is a lack of methods to effectively optimize sales strategies and layouts by analyzing and utilizing user emotion and behavior data in real time. These problems need to be solved.

[0927] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for acquiring video data from a monitoring device installed in the store, a means for analyzing the acquired video data to detect the user's movements and facial expressions, a means for recognizing emotions based on the user's facial expression data, a means for tracking the user's stay time, location, and flow to learn behavioral patterns, a means for determining that a user exhibiting a specific behavioral pattern or emotion is a user considering purchasing, and a means for sending a notification to a notification target when a user considering purchasing is identified. This allows the user's purchasing intention to be properly understood and sales staff to provide timely support. Furthermore, analyzing user behavioral data can optimize store layout and improve sales strategies.

[0928] A "monitoring device" is a device that is placed in a store and captures users' movements and facial expressions in real time.

[0929] "Video data" refers to visual information including the user's movements and facial expressions acquired by a monitoring device.

[0930] "Analysis" is the process of processing video data, recognizing the user's movements and facial expressions, and extracting this as data.

[0931] "Facial expression data" is information that indicates emotions extracted from the user's facial movements and features.

[0932] "Emotion recognition" is the process of analyzing facial expression data to identify the emotion a user is expressing (e.g., joy, surprise, anger, sadness, etc.).

[0933] "Behavioral patterns" are the tendencies and characteristics of user behavior learned based on data such as the user's length of stay, location, and path.

[0934] A "purchase consideration user" is a user who is determined to have a high desire to purchase because they exhibit specific behavioral patterns and emotions based on analysis results and emotion recognition data.

[0935] The "notification recipient" is a person (e.g., a salesperson) who should receive a notification when a user considering purchase is identified.

[0936] A "notification" is a message containing information about a user considering purchasing that is sent from the server to the notification target.

[0937] "Layout optimization" is the process of optimizing the placement and display within a location based on tracked user behavior data.

[0938] This invention is a system that analyzes the behavior of users (customers) in a store in real time, identifies the user's purchase intentions, and notifies the salesperson. By combining this system with an emotion engine that recognizes the user's emotions, it is possible to provide more accurate support.

[0939] Camera image acquisition and video data analysis

[0940] The terminal acquires video data in real time from multiple monitoring devices (e.g., network cameras) placed in the store. The monitoring devices are placed to cover specific areas or product shelves. The server receives this video data and performs analysis. The analysis is performed using Python's OpenCV library and TensorFlow.

[0941] Detecting user movements and facial expressions

[0942] The server analyzes the user's movements and facial expressions based on the received video data. Specifically, it uses facial recognition technology and object detection algorithms to identify the user's location and products they are interested in. This analysis is performed using open-source computer vision libraries (e.g., OpenCV) and machine learning frameworks (e.g., TensorFlow).

[0943] Introducing the Emotion Engine

[0944] Based on the analyzed facial recognition data, the server uses an emotion engine to recognize the user's emotions. The emotion engine analyzes the user's facial muscle movements and other facial features to determine emotions such as happiness, surprise, anger, sadness, etc. This information, combined with the analysis of movements and facial expressions, is used to more precisely understand the user's overall behavioral patterns.

[0945] Tracking user behavior and learning patterns

[0946] The server tracks the user's time spent, location, and path in real time, allowing it to determine how long the user spent at a particular shelf and the path they took. For example, if a user spends 10 minutes in front of a particular shelf, it can determine that they are interested in that product.

[0947] Determining and notifying purchase intent

[0948] Based on the user's behavioral patterns and emotional data, the server identifies users who are deemed to have a strong intent to purchase as "considering purchase users." For example, if a user lingers in front of a particular product shelf for a long time, repeatedly picks up a product to check its details, or shows positive emotions (e.g., joy). When a considering purchase user is identified, the server immediately sends a notification to the salesperson.

[0949] Salesperson's response

[0950] The salesperson, who is the user, receives the notification sent to their device and checks its contents. The notification includes the user's current location, products of interest, and the user's current emotional state. Based on this, the salesperson contacts the specific user and provides detailed product information or suggests related products.

[0951] Accumulating user behavior data and optimizing layout

[0952] The server stores user behavior data and the results of sales staff responses in a database. This data is then analyzed to determine user patterns throughout the store. By understanding the popularity of specific areas and products, as well as user movement patterns, it becomes possible to optimize layouts and product displays to suit the characteristics of each store. For example, based on data showing that a specific product area is frequently visited on weekends, it can propose changes to the layout of that area and the placement of related products.

[0953] Examples of concrete examples and prompts

[0954] Examples:

[0955] For example, a user enters a store and heads to the food section. The device receives video data via a camera installed in the device, which is then analyzed by a server. The server recognizes that the user spends a long time in front of the shelves in the food section, picking up and looking at several products. It combines this information with emotional data to determine that the user is interested in those products. The server then sends a notification to a salesperson, who then checks the user's location, products of interest, and emotional state on the device. The salesperson then contacts the user, explains the details of the products, and suggests related products.

[0956] Example prompt for a generative AI model:

[0957] "What is the appropriate way to respond when a customer lingers in front of a particular shelf in a store for a long time?"

[0958] "Please explain the specific method for recognizing user emotions and determining purchasing intent from video data."

[0959] As described above, the present invention provides a system and an operating method for analyzing user behavior and emotions in real time to maximize sales opportunities.

[0960] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0961] Step 1: Acquire camera footage

[0962] The terminal acquires video data from surveillance equipment installed in the store. The surveillance equipment uses network cameras, each positioned to cover a specific area or shelf. For example, a camera captures a food section, and the video data is streamed to the terminal. The input is real-time video data, and the output is video data sent to a server via the terminal.

[0963] Step 2: Analyzing the video data

[0964] The server receives video data sent from the device. The server uses Python's OpenCV library to extract frames of the video data and perform preprocessing such as noise removal. Next, it uses TensorFlow to analyze the user's movements and facial expressions. For example, the server analyzes frames every second to detect the movements of the user's face and hands. The input is the captured video data, and the output is the analyzed user's movements and facial expression data.

[0965] Step 3: Detecting user movements and facial expressions

[0966] The server uses OpenCV's Haar Cascade and TensorFlow to identify the position of the user's face and hands and detect changes in facial expressions and movements. For example, it recognizes the moment when the user smiles or picks up a specific product. The input is the video frame extracted in the previous step, and the output is specific data on the user's movements and facial expressions.

[0967] Step 4: Emotion Recognition with the Emotion Engine

[0968] The server uses Affectiva's API to send the facial expression data obtained in the previous step and recognize the user's emotions in real time. The emotion engine determines emotions (happiness, surprise, anger, sadness, etc.) from the user's facial muscle movements and facial features. The input is the user's facial expression data, and the output is the recognized emotion data.

[0969] Step 5: Track user behavior and learn patterns

[0970] The server tracks the user's time spent, location, and path in real time and records this data. For example, the database stores information such as how long the user stayed in front of a particular shelf and the path they took. The input is the user's location and path data, and the output is behavioral patterns and learned user data.

[0971] Step 6: Determine and communicate purchase intent

[0972] The server analyzes the user's behavioral patterns and emotional data, and identifies users who are determined to have a strong intention to purchase as "considering purchase users." For example, this applies when a user lingers in front of a particular product shelf for a long time, picks up the product to check its details, or shows positive emotions (joy). The input is behavioral pattern data and emotional data, and the output is a notification to the salesperson.

[0973] Step 7: Salesperson response

[0974] The salesperson, who is the user, receives the notification sent to the terminal and checks the content. The notification includes the user's current location, products of interest, and emotional state. Based on this, the salesperson contacts the user to provide detailed product information and suggest related products. The input is the notification data from the server, and the output is specific support for the user.

[0975] Step 8: Accumulating user behavior data and optimizing layout

[0976] The server stores user behavior data and the results of sales staff responses in a database. This data is analyzed to understand the popularity of specific areas and products, as well as user movement patterns. For example, if a specific product area is frequently visited on weekends, the server will suggest changes to the layout of that area and the placement of related products. The input is the stored behavioral data, and the output is specific proposals for layout optimization.

[0977] Through these steps, the system can analyze user behavior and emotions in real time, enabling effective sales strategies and store layout optimization.

[0978] (Application example 2)

[0979] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0980] In traditional brick-and-mortar stores, it was difficult to analyze customer behavior and emotions in real time and provide appropriate support at the right time. This made it difficult to effectively stimulate customer purchasing desire, limiting store sales. Furthermore, data analysis to improve customer experience was insufficient, and store layout and product display were not optimized.

[0981] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0982] In this invention, the server includes: means for acquiring video data from cameras installed in the store; means for analyzing the acquired video data to detect user movements and facial expressions; means for tracking user stay time, location, and flow to learn behavioral patterns; means for recognizing user emotions based on the analyzed facial recognition data; means for determining that a user exhibiting a specific behavioral pattern and emotional state is a user considering a purchase; and means for sending a notification to a salesperson when a user considering a purchase is identified. This allows for timely notification to a salesperson based on the customer's behavior and emotions, enabling effective customer service. Furthermore, it is possible to optimize the store layout and product display based on user behavior data and emotional data, improving the customer experience and increasing sales.

[0983] "In-store" refers to the physical location where customers can actually touch and purchase products.

[0984] A "camera" is a visual sensor device for acquiring video data.

[0985] "Video data" refers to the digital information of images or video captured by a camera.

[0986] A "user" is a customer who performs actions such as viewing products in a store or making a purchase.

[0987] "Movement" refers to the movement or change in position of the user's body.

[0988] "Facial expressions" are the emotional expressions and muscle movements that appear on the user's face.

[0989] "Dwell time" refers to the amount of time a user stays in a particular location or on a shelf.

[0990] "Location" is information that indicates which area of ​​the store the user is in.

[0991] "Pathways" refer to the routes and patterns that users take within a store.

[0992] "Behavioral patterns" refer to habitual behavior that can be inferred from a user's series of actions, changes in location, length of stay, etc.

[0993] A "purchase consideration user" is a user who is determined to have a strong intention to purchase based on their behavioral patterns and emotional state.

[0994] A "notification" is a message sent to convey specific information.

[0995] A "salesperson" is a staff member who explains products and provides support to customers in a store.

[0996] "Facial recognition data" is digital information used to identify and analyze a user's face.

[0997] "Emotion" refers to a user's psychological state as judged from their facial expressions and behavior.

[0998] A "server" is a computer system that analyzes video data, stores various data, and sends notifications.

[0999] "Optimization" refers to the efficient rearrangement of layout and product displays based on data analysis.

[1000] The present invention relates to a system that analyzes customer behavior and emotions in real time in a store and provides appropriate support at the appropriate time. This system is realized by capturing video data using cameras placed in the store and analyzing that video data.

[1001] Acquiring camera footage

[1002] The server collects real-time video data from multiple cameras placed throughout the store, which are positioned to cover specific areas and shelves, allowing for comprehensive monitoring of customer behavior.

[1003] Video data analysis

[1004] The server analyzes the captured video data to detect the customer's movements and facial expressions. This analysis uses open-source computer vision libraries and machine learning frameworks such as OpenCV and TensorFlow. Specifically, it uses a facial recognition model to detect the customer's facial expressions and recognizes the customer's emotions based on the facial expression data.

[1005] Introducing the Emotion Engine

[1006] The server recognizes the customer's emotions using an emotion engine based on the analyzed facial recognition data. The emotion engine analyzes the customer's facial muscle movements and other facial features to determine emotions such as happiness, surprise, anger, sadness, etc. This information, combined with the analysis of movements and facial expressions, is used to more precisely understand the customer's overall behavioral patterns.

[1007] Tracking user time, location, and path

[1008] The server tracks customer dwell time, location, and path in real time, allowing it to determine how long a customer spends at a particular shelf and the path they take. This data is used to learn user behavior patterns.

[1009] Identifying and notifying potential purchasers

[1010] Based on the user's behavioral patterns and emotional data, the server identifies users who are determined to have a high intent to purchase as "purchase-consideration users." For example, if a user spends a long time in front of a particular product shelf, repeatedly picks up products to check their details, or shows positive emotions (e.g., joy), the server determines that the user has a high intent to purchase. When a user is identified as a purchase-consideration user, the server immediately sends a notification to the salesperson. The notification includes information about the user's current location, the products they are interested in, and their current emotional state.

[1011] Salesperson's response

[1012] Salespeople receive notifications on their devices, review the content, and contact specific users based on the content of the notifications to provide appropriate support. For example, they can provide detailed product information or suggest other related products. Additionally, they can offer additional services or special offers to users who show particularly positive emotions.

[1013] Accumulating user behavior data and optimizing layout

[1014] The server stores user behavior data and sales staff response results in a database. Based on this data, it analyzes user patterns throughout the store, understanding the popularity of specific areas and products, and user movement patterns. This makes it possible to optimize layouts and product displays to suit the characteristics of each store.

[1015] Examples of concrete examples and prompts

[1016] As a specific example, if customer A is seen smiling in the in-store camera footage and spends a long time in front of a particular product, the application will notify the salesperson of this information so that the salesperson can provide appropriate assistance.

[1017] Example prompt for a generative AI model:

[1018] "We will develop a system that analyzes video footage from multiple cameras in real time, recognizes faces and emotions, determines the purchasing intent of customers in a specific area, and notifies sales staff at the appropriate time."

[1019] Thus, the present invention provides a system and an operating method thereof for analyzing customer behavior and emotions in real time and maximizing sales opportunities.

[1020] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1021] Step 1:

[1022] Acquiring video data from the camera

[1023] The server receives real-time video data from cameras placed around the store, capturing images of specific areas of the store and product shelves. It receives the video data from the cameras as input and converts it into a processable format as output.

[1024] Step 2:

[1025] Video data analysis

[1026] The server detects the user's movements and facial expressions based on the acquired video data. Specifically, it uses OpenCV to perform face detection and identify the user's location. The input is video data from the camera, and the output is the location information of the detected face.

[1027] Step 3:

[1028] Emotion recognition based on facial recognition data

[1029] The server uses facial recognition data to recognize the user's emotions. Using a TensorFlow model, it analyzes the user's facial muscle movements and other features to determine emotions such as joy, surprise, and sadness. The input is the facial position information and video data obtained in step 2, and the output is the user's emotional state.

[1030] Step 4:

[1031] Tracking time spent, location, and traffic

[1032] The server tracks users' time spent, location, and route in real time, allowing it to determine how long they stay in a particular area and the route they take. The input is video data from the camera and facial recognition data, and the output is user behavior pattern data.

[1033] Step 5:

[1034] Identifying and notifying potential purchasers

[1035] The server identifies users who are deemed to have a high intent to purchase based on their behavioral patterns and emotional data as "consideration users." If identified, it sends a notification to the salesperson containing information about the user's current location, products they are interested in, and their emotional state. The input is the data obtained in Step 4 and Step 3, and the output is a notification to the salesperson.

[1036] Step 6:

[1037] Salesperson's response

[1038] The terminal allows the salesperson to receive the notification and provide support to the user based on its content. Specifically, it provides detailed product information and suggests other related products to the user. In particular, it suggests additional services and benefits to users who show positive emotions. The input is the notification sent to the salesperson's terminal, and the output is the result of the support provided.

[1039] Step 7:

[1040] Accumulating user behavior data and optimizing layout

[1041] The server accumulates user behavior data and salesperson response results in a database and analyzes user patterns throughout the store. This allows the popularity of specific areas within the store and user movement patterns to be understood, and layout optimization proposals are generated. The input is the data obtained at each step, and the output is layout and display optimization proposals.

[1042] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1043] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1044] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1045] [Fourth embodiment]

[1046] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1047] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1048] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1049] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1050] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1051] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1052] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1053] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1054] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1055] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1056] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1057] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1058] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1059] The present invention provides a system that analyzes the behavior of users (customers) in a store in real time and sends a notification to sales staff at an appropriate time when it is determined that the user has a purchase intention.The system also makes suggestions for optimizing store layout and product displays based on the accumulated customer behavior data.Detailed embodiments of the present invention are described below.

[1060] Camera image acquisition and video data analysis

[1061] The device collects video data in real time from multiple cameras in the store. This video data is sent from cameras positioned to cover specific areas and shelves in the store. The server receives this video data and analyzes it in real time.

[1062] Detecting user movements and facial expressions

[1063] The server analyzes the user's movements and facial expressions based on the received video data. Specifically, it uses facial recognition technology and object detection algorithms to determine where the user is and what products they are interested in. This analysis can be performed using open-source computer vision libraries (e.g., OpenCV) and machine learning frameworks (e.g., TensorFlow).

[1064] Tracking time spent, location, and path, and learning behavioral patterns

[1065] The server tracks users' time spent, location, and route in real time. This allows it to determine how long a user stays at a particular shelf and the path they take. This data is used to learn user behavior patterns. For example, if a user spends a long time in a certain area, it can be determined that they are interested in that area.

[1066] Identifying and notifying potential purchasers

[1067] Based on the user's behavioral patterns, the server identifies users who are deemed to have a high intent to purchase as "purchase-consideration users." This identification is performed using behavioral analysis algorithms and machine learning models (e.g., random forests and support vector machines). When a user is identified as a purchase-consideration user, the server immediately sends a notification to the salesperson via the terminal. This notification includes information about the user's current location and the products they are interested in.

[1068] Salesperson's response

[1069] The salesperson, who is the user, receives a notification on their device. Upon receiving the notification, the salesperson will go to the user's designated location and provide appropriate support, such as providing detailed explanations about a specific product or suggesting additional products.

[1070] Accumulating user behavior data and optimizing layout

[1071] The server stores user behavior data and the results of sales staff responses in a database. Based on this data, it analyzes user patterns throughout the store, understanding the popularity of specific areas and products, and user movement patterns. This makes it possible to optimize layouts and product displays to suit the characteristics of each store. For example, if a specific product shelf is frequently visited, it can suggest adding products to the surrounding area or changing the layout.

[1072] Layout optimization example

[1073] For example, suppose there is a specific product area where many users gather on weekends. Based on this data, the server can suggest placing related products around that area or recommend expanding aisles to make user flow smoother. This can improve the user's shopping experience and increase store sales.

[1074] In this way, the present invention provides a system and an operating method for analyzing user behavior in a store in real time and maximizing sales opportunities.

[1075] The processing flow will be explained below.

[1076] Step 1:

[1077] The device captures real-time video data from cameras installed in the store, covering different areas and shelves within the store, and stores multiple video streams in an internal buffer.

[1078] Step 2:

[1079] The video data acquired by the terminal is compressed and sent to the server via the network. The data is compressed using a video codec to improve communication efficiency.

[1080] Step 3:

[1081] The server analyzes the received video data, uses computer vision algorithms to detect the user's movements and facial expressions, uses facial recognition technology to identify the user's face, and uses object recognition technology to identify the product the user is holding.

[1082] Step 4:

[1083] The server tracks the user's dwell time, location, and path, collecting data on how long the user spends in a particular area or in front of a shelf, and recording the user's path.

[1084] Step 5:

[1085] The server learns user behavior patterns based on the collected data, and uses machine learning algorithms to analyze user behavior trends and patterns to identify users with strong purchasing intent.

[1086] Step 6:

[1087] The server determines that users who exhibit certain behavioral patterns are considering a purchase. For example, if a user repeatedly spends a long time in front of a particular product shelf, picking up a product and checking its details, it is determined that the user has a strong intention to purchase.

[1088] Step 7:

[1089] If the server identifies a potential buyer, it immediately sends a notification to a salesperson, including information about the user's current location and the product they are interested in.

[1090] Step 8:

[1091] The salesperson, who is the user, receives the notification, reviews the content, and then contacts the specific user based on the content of the notification to provide appropriate support, such as providing the user with detailed information about the product or suggesting other related products.

[1092] Step 9:

[1093] The server records the results of salespeople's responses and data on interactions with users, which allows the effectiveness of sales activities to be evaluated and a database to be built for future use.

[1094] Step 10:

[1095] Based on the accumulated user behavior data, the server generates proposals to optimize the layout and display of each store. It analyzes whether specific areas or products are frequently visited and uses an optimization algorithm to propose layout changes.

[1096] Step 11:

[1097] The store manager, who is the user, reviews the proposals and takes steps to optimize the store layout and displays, thereby achieving efficient store operations and improving customer satisfaction.

[1098] Example 1

[1099] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1100] In conventional stores, there are limited means of understanding customer behavior patterns and purchasing intentions in real time, making it difficult to provide efficient sales support and effectively optimize store layouts. Therefore, there is a need for a method that can accurately grasp customer purchasing intentions to avoid missing sales opportunities, as well as to create effective store layout plans based on customer behavior data and provide a better shopping experience.

[1101] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1102] In this invention, the server includes a means for acquiring video data from a camera installed in the store, a means for analyzing the acquired video data to detect customer movements and facial expressions, a means for tracking customer stay time, location, and flow to learn behavioral patterns, a means for determining whether a customer exhibiting a specific behavioral pattern is a potential purchaser, and a means for sending a notification to a salesperson when a potential purchaser is identified. This allows for efficient sales support by grasping customer purchasing intentions in real time and notifying salespersons at the appropriate time. Furthermore, the server can propose optimization of store layout and product display based on customer behavior data, which is expected to improve customer satisfaction and increase store sales.

[1103] A "camera" is a device installed in a store to monitor and record customer behavior.

[1104] "Video data" refers to images and video information captured by a photographing device.

[1105] "Customer" means a person browsing or purchasing products in a store.

[1106] "Movement" refers to the physical movements and actions that customers perform in the store.

[1107] "Facial expressions" refer to the customer's facial expressions and gestures that show emotion.

[1108] "Dwell time" refers to the amount of time a customer spends in a particular area or in front of a shelf.

[1109] "Location" is information indicating where the customer is located within the store.

[1110] "Flow" refers to the route or pattern that customers take within a store.

[1111] "Behavioral patterns" refer to the trends and characteristics of customer behavior analyzed based on data such as customer length of stay, location, and route.

[1112] "Considering purchase customers" are customers whose behavioral patterns indicate a high level of purchasing intent.

[1113] A "salesperson" is an employee who guides and explains products to customers in a store.

[1114] A "notification" is a message or alert sent from the server to a salesperson.

[1115] The "server" is a central processing unit that analyzes video data and tracks and records customer behavior.

[1116] "Layout" refers to the layout plan within the store and the placement and arrangement of products.

[1117] "Product display" refers to the display and arrangement of products within a store.

[1118] The present invention provides a system that analyzes customer behavior in a store in real time and sends a notification to a salesperson at an appropriate time when it is determined that the customer has a purchase intention. The system also proposes optimization of store layout and product display based on the accumulated customer behavior data. Detailed embodiments of the present invention are described below.

[1119] Acquiring camera footage

[1120] The device acquires video data in real time from multiple cameras placed in the store. These cameras are positioned to cover specific areas or product shelves within the store. For example, cameras can be placed near key product shelves or store entrances to monitor customer behavior from all directions.

[1121] Video data analysis

[1122] The server receives and analyzes video data sent from the device in real time. The video data is processed frame by frame, and pre-processing is performed to identify customer movements and positions. Specifically, the analysis is performed using open-source computer vision libraries (e.g., OpenCV) and machine learning frameworks (e.g., TensorFlow).

[1123] User behavior and facial expression detection

[1124] The server detects the customer's movements and facial expressions from the video data. It uses facial recognition technology to identify the customer's face, and an object detection algorithm to identify the customer's movements and products of interest. For example, by combining facial recognition using OpenCV and object detection using TensorFlow, it can analyze which product the customer is in front of and how they are moving.

[1125] Tracking time spent, location, and traffic

[1126] The server tracks the customer's time spent, location, and movement path (pathway). This makes it possible to determine how long a customer stayed at a particular shelf and which path they took. For example, if a customer stays in front of a particular shelf for five minutes, the server can use this data to analyze the customer's behavioral patterns.

[1127] Determining users considering purchasing

[1128] Based on the customer's behavioral patterns, the server identifies customers who are judged to have a high intent to purchase as "considering purchase." This identification is performed using behavioral analysis algorithms and machine learning models (e.g., random forests and support vector machines). When a considering purchase customer is identified, the server immediately sends a notification to the salesperson via the terminal.

[1129] Notification to sales staff

[1130] The server notifies the salesperson via the terminal of information about the identified potential purchaser, including details about the customer's current location and the product they are interested in. For example, the terminal may be notified that "Customer A is interested in the product on shelf B."

[1131] Salesperson's response

[1132] The salesperson, who is the user, receives a notification from the terminal. Upon receiving the notification, the salesperson quickly goes to the specified customer's location and provides appropriate support, such as providing detailed explanations about the product or suggesting additional items.

[1133] Accumulation and analysis of user behavior data

[1134] The server stores all customer behavior data and the results of sales staff responses in a database, making it possible to analyze customer behavior patterns across the entire store.

[1135] Layout optimization proposals

[1136] The server generates suggestions for improving store layout and product displays based on the accumulated behavioral data, such as "Since a particular area is crowded on weekends, place related products around that area" or "Expand aisles to streamline customer flow."

[1137] Examples of prompt statements

[1138] The following is a specific example of a prompt sentence to be input to the generative AI model:

[1139] "Write a program that analyzes video data obtained from a camera and uses an algorithm to identify behavioral patterns based on the customer's movements and facial expressions to detect customers who intend to make a purchase in real time. Include a function to send a notification to a salesperson if a customer shows interest in a product on the shelf. Use the OpenCV and TensorFlow libraries."

[1140] As described above, the present invention provides a system and an operating method thereof for analyzing customer behavior in a store in real time and maximizing sales opportunities.

[1141] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1142] System program processing flow

[1143] Step 1: Acquire camera footage

[1144] The terminal acquires video data in real time from multiple cameras placed in the store. These cameras are positioned to cover specific areas and product shelves, monitoring customer behavior from all directions. The input is video data from the cameras, and the output is raw video data stored in the terminal.

[1145] Step 2: Sending video data

[1146] The video data acquired by the device is sent to the server in real time. Here, the input is the raw video data stored on the device, and the output is the video data to be sent to the server. For example, the setting is to send the data to the server periodically, frame by frame.

[1147] Step 3: Analyzing the video data

[1148] The server processes the received video data frame by frame and performs preprocessing to identify the customer's movements and position. Specifically, it uses OpenCV to preprocess the images, remove noise, and resize the images. The input is the video data sent to the server, and the output is the preprocessed image data.

[1149] Step 4: Detect customer behavior and facial expressions

[1150] The server detects the customer's movements and facial expressions from the preprocessed image data. It uses facial recognition technology to identify the customer's face and TensorFlow to perform object detection and movement recognition. The input is the preprocessed image data, and the output is analytical data on the customer's movements and facial expressions. For example, it detects whether the customer is smiling or looking at a specific product.

[1151] Step 5: Tracking dwell time, location, and path

[1152] The server records customer location information in real time and tracks their time spent and their path (pathway). The input is the detected customer location data, and the output is behavioral data including the customer's time spent, location, and path. For example, it tracks how long a customer stays in front of a particular product.

[1153] Step 6: Identify potential buyers

[1154] Based on customer behavior data, the server identifies customers who are determined to have a high intent to purchase as "considering purchase customers." It uses behavioral analysis algorithms such as random forests and support vector machines. The input is customer behavior data, and the output is the identification data of customers considering purchase. For example, it may identify "Customer A is interested in Product B."

[1155] Step 7: Notify salespeople

[1156] The server notifies the salesperson via the terminal of the information about the identified customer who is considering purchasing. The input is the specific data of the customer who is considering purchasing, and the output is the notification data sent to the salesperson's terminal. For example, the notification may say, "Customer A is interested in the product on shelf B."

[1157] Step 8: Salesperson response

[1158] The salesperson, who is the user, receives a notification from the terminal and quickly goes to the specified customer's location to provide appropriate support. The input is the notification data, and the output is the action of providing support to the customer, such as providing detailed explanations about the product or suggesting additional products.

[1159] Step 9: Collect and analyze user behavior data

[1160] The server stores all customer behavior data and salesperson response results in a database. The input is customer behavior data and salesperson response data, and the output is the stored database.

[1161] Step 10: Layout optimization proposal

[1162] The server generates suggestions for improving store layout and product displays based on the accumulated behavioral data. The input is the accumulated behavioral data, and the output is a layout optimization proposal. For example, it may suggest, "Since a particular area is crowded on weekends, place related products around that area."

[1163] As a result, the system of the present invention provides a system and an operating method for analyzing customer behavior in a store in real time and maximizing sales opportunities.

[1164] (Application example 1)

[1165] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1166] In recent years, there has been a demand for optimizing customer service and improving the customer experience in physical stores. However, the current situation lacks a system that can analyze customer behavior and purchasing intent in real time and send notifications to sales staff at the appropriate time. This can lead to missed sales opportunities and dissatisfaction among customers who do not receive the support they need. Furthermore, store layout and product display are not sufficiently optimized, making it difficult to increase sales and operate efficiently.

[1167] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1168] In this invention, the server includes: means for acquiring video data from cameras installed in the store; means for analyzing the acquired video data to detect user movements and facial expressions; means for tracking user time spent, location, and flow to learn behavioral patterns; means for determining whether a user exhibiting a specific behavioral pattern is a potential purchaser; means for sending a notification to a salesperson when a potential purchaser is identified; and means for sending a notification to the salesperson in real time using a smart device. This allows salespersons to provide support to customers in a timely manner. Furthermore, store layout and product display can be optimized based on user behavior data, improving store operational efficiency and leading to increased sales.

[1169] "Video data" refers to visual information acquired by a camera or other imaging device, and is data in the form of images or videos.

[1170] "Analysis" refers to processing acquired data to extract and understand specific information.

[1171] A "user" is a customer who is considering products in a store or thinking about making a purchase.

[1172] "Movement" refers to the physical movements and actions that users perform within the store.

[1173] "Facial expressions" refer to changes in the user's face that indicate emotions or intentions.

[1174] "Dwell time" refers to how long a user stays in front of a particular area or product.

[1175] "Location" is information indicating where the user is located within the store.

[1176] A "path" is the route a user follows when moving around a store.

[1177] "Behavioral patterns" are specific trends that are identified from a user's series of actions and location information.

[1178] "Purchase intent" refers to the possibility or willingness of a user to purchase a product.

[1179] A "user considering purchasing" is a user whose behavioral patterns indicate a high level of purchasing intent.

[1180] "Notification" refers to a message or alert that the system sends to a salesperson to inform them of specific information.

[1181] "Salesperson" refers to a staff member who explains products and provides support to customers in a store.

[1182] "Smart devices" refers to mobile information terminals, eyeglass-type devices, head-mounted displays, etc. that have communication functions and can connect to the Internet.

[1183] "Layout optimization" means arranging products and customer flow in a store in the most efficient way possible based on user behavior data.

[1184] "Product display optimization" refers to improving the way products are arranged based on product sales and customer interest.

[1185] The present invention provides a system that analyzes customer behavior in a physical store in real time and notifies sales staff of customers with high purchasing intent, thereby improving store management efficiency and customer satisfaction.

[1186] Acquiring camera footage and analyzing video data

[1187] The server collects video data in real time from multiple cameras placed in the store. These cameras are positioned to cover specific areas and shelves, and the video data from each camera is sent to the server. This video data is then analyzed using a computer vision library (e.g., OpenCV) or a machine learning framework (e.g., TensorFlow).

[1188] Detecting user movements and facial expressions

[1189] The server analyzes the user's movements and facial expressions based on the received video data. It uses facial recognition technology and object detection algorithms to determine where the user is and what products they are interested in. For example, if the user is paying attention to a particular product shelf, it can retrieve information about that product from the database.

[1190] Tracking user time spent, location, and path, and learning behavioral patterns

[1191] The server tracks users' time spent and locations in real time. Based on this information, it learns their behavioral patterns and determines whether a particular pattern indicates a purchasing intent. For example, if a user spends a long time on a particular shelf, it determines that the user is interested in that product.

[1192] Identifying and notifying potential purchasers

[1193] The server identifies users who are deemed to have a high intent to purchase based on their behavioral patterns as "purchase-consideration users." This determination is made using behavioral analysis algorithms and machine learning models (e.g., random forests and support vector machines). When a purchase-consideration user is identified, the server immediately sends a notification to a salesperson via a smart device (e.g., smart glasses or smartphone).

[1194] Salesperson's response

[1195] The salesperson receives a notification on the smart device, which includes the user's current location and product information of interest. Upon receiving the notification, the salesperson will go to the user's specified location and provide appropriate assistance. For example, the salesperson may provide detailed explanations about a specific product or suggest additional products.

[1196] Accumulating user behavior data and optimizing layout

[1197] The server stores user behavior data and the results of sales staff responses in a database. Based on this data, it analyzes user patterns throughout the store, understanding the popularity of specific areas and products, and user movement patterns. This makes it possible to optimize layouts and product displays to suit the characteristics of each store. For example, if a specific product shelf is frequently visited, it can suggest adding products to the surrounding area or changing the layout.

[1198] Practical examples and prompts

[1199] For example, if a customer spends a long time considering a particular product, "Product XYZ," in "Aisle 3" of a store, a notification will be sent to a sales associate wearing smart glasses who will provide appropriate assistance.

[1200] Example prompt sentence:

[1201] "Create a system that analyzes customer behavior in real time and sends notifications about customers who are deemed to have high purchasing intent."

[1202] In this way, the present invention provides a system and an operating method for analyzing user behavior in a physical store in real time and maximizing sales opportunities.

[1203] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1204] Step 1:

[1205] The server acquires video data in real time from multiple cameras installed in the store. The video data sent from the cameras is temporarily stored in the server's storage. This processing uses the camera devices, network connections, and storage functions. The input is video data from the cameras, and the output is video data stored on the server.

[1206] Step 2:

[1207] The server analyzes the captured video data to detect the user's movements and facial expressions. This uses a computer vision library (e.g., OpenCV) or a machine learning framework (e.g., TensorFlow). The input is the stored video data, and the output is analytical data on the user's position, movements, and facial expressions. Based on the analysis, the server determines which area the user is in.

[1208] Step 3:

[1209] The server performs processing to track users' time spent, location, and path in real time. Based on this information, it learns user behavior patterns. The inputs are location data, time spent, and path data, and the output is user behavior pattern data. The server uses storage data and real-time analysis data to track user paths.

[1210] Step 4:

[1211] The server analyzes behavioral patterns and runs an algorithm to determine whether users who exhibit certain behavioral patterns are "considerers." This process uses behavioral analysis algorithms and machine learning models (e.g., random forests and support vector machines). The input is behavioral pattern data, and the output is identification data for users considering purchasing.

[1212] Step 5:

[1213] When a "purchase-consideration user" is identified, the server immediately sends a notification to the smart device (e.g., smart glasses, smartphone). The input is the user's identification data, and the output is the notification data sent to the smart device. The server uses a network connection to send the notification.

[1214] Step 6:

[1215] The terminal receives the notification and displays an alert to the salesperson. The salesperson checks the notification on their smart device, goes to the specified user's location, and takes appropriate action to provide support. The input is the notification data sent from the server, and the output is the alert display to the salesperson.

[1216] Step 7:

[1217] The server stores user behavior data and the results of sales staff responses in a database. The stored data is used to optimize store layout and product displays. The input is user behavior data and response result data, and the output is optimization proposal data stored in the database. The server analyzes the stored data and generates optimization proposals for layout and display.

[1218] In this way, specific operations are clarified at each step, and the input and output data are defined for each step. Following this flow allows the entire system to operate effectively.

[1219] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1220] The present invention combines an emotion engine with a system that analyzes the behavior of users (customers) in a store in real time and sends a notification to a salesperson at the appropriate time if it is determined that the user has an intention to purchase. This system recognizes the user's emotions and can provide more accurate support based on that information. Detailed embodiments of the present invention are described below.

[1221] Camera image acquisition and video data analysis

[1222] The device collects video data in real time from multiple cameras in the store. This video data is sent from cameras positioned to cover specific areas and shelves in the store. The server receives this video data and analyzes it in real time.

[1223] Detecting user movements and facial expressions

[1224] The server analyzes the user's movements and facial expressions based on the received video data. Specifically, it uses facial recognition technology and object detection algorithms to determine where the user is and what products they are interested in. This analysis can be performed using open-source computer vision libraries (e.g., OpenCV) and machine learning frameworks (e.g., TensorFlow).

[1225] Introducing the Emotion Engine

[1226] Based on the analyzed facial recognition data, the server uses an emotion engine to recognize the user's emotions. The emotion engine analyzes the user's facial muscle movements and other facial features to determine emotions such as happiness, surprise, anger, sadness, etc. This information, combined with the analysis of movements and facial expressions, is used to more precisely understand the user's overall behavioral patterns.

[1227] Tracking time spent, location, and path, and learning behavioral patterns

[1228] The server tracks users' time spent, location, and route in real time. This allows it to determine how long a user stays at a particular shelf and the path they take. This data is used to learn user behavior patterns. For example, if a user spends a long time in a certain area, it can be determined that they are interested in that area.

[1229] Identifying and notifying potential purchasers

[1230] Based on the user's behavioral patterns and emotional data, the server identifies users who are deemed to have a high intent to purchase as "considering purchase users." For example, if a user spends a long time in front of a particular product shelf, repeatedly picks up a product to check its details, or shows positive emotions (e.g., joy), the server determines that the user has a high intent to purchase.

[1231] When a potential buyer is identified, the server immediately sends a notification to a salesperson, containing information about the user's current location, the product they are interested in, and their current emotional state.

[1232] Salesperson's response

[1233] The salesperson, who is the user, receives the notification on their device and checks the content. Based on the content of the notification, they can contact the specific user and provide appropriate support. For example, they can provide the user with detailed information about the product or suggest other related products. In addition, they can suggest additional services or benefits to users who show particularly positive emotions.

[1234] Accumulating user behavior data and optimizing layout

[1235] The server stores user behavior data and the results of sales staff responses in a database. Based on this data, it analyzes user patterns throughout the store, understanding the popularity of specific areas and products, and user movement patterns. This makes it possible to optimize layouts and product displays to suit the characteristics of each store. For example, if a specific product shelf is frequently visited, it can suggest adding products to the surrounding area or changing the layout.

[1236] Layout optimization example

[1237] For example, suppose there is a specific product area where many users gather on weekends. Based on this data, the server can suggest placing related products around that area or recommend expanding aisles to make user flow smoother. This can improve the user's shopping experience and increase store sales.

[1238] In this way, the present invention provides a system and an operating method for analyzing user behavior and emotions in a store in real time to maximize sales opportunities.

[1239] The processing flow will be explained below.

[1240] Step 1:

[1241] The device captures real-time video data from cameras installed in the store, covering different areas and shelves within the store, and stores multiple video streams in an internal buffer.

[1242] Step 2:

[1243] The video data acquired by the terminal is compressed and sent to the server via the network. The data is compressed using a video codec to improve communication efficiency.

[1244] Step 3:

[1245] The server analyzes the received video data, uses computer vision algorithms to detect the user's movements and facial expressions, uses facial recognition technology to identify the user's face, and uses object recognition technology to identify the product the user is holding.

[1246] Step 4:

[1247] The server uses an emotion engine to recognize the user's emotions from the analyzed facial recognition data. The emotion engine analyzes the user's facial muscle movements and facial features to determine emotions such as joy, surprise, anger, and sadness.

[1248] Step 5:

[1249] The server tracks the user's dwell time, location, and path, collecting data on how long the user spends in a particular area or in front of a shelf, and recording the user's path.

[1250] Step 6:

[1251] The server learns user behavior patterns based on the collected data, and uses machine learning algorithms to analyze user behavior trends and patterns to identify users with strong purchasing intent.

[1252] Step 7:

[1253] The server combines users who exhibit specific behavioral patterns with positive emotions generated by the emotion engine to determine whether they are considering a purchase. For example, if a user spends a long time in front of a particular product shelf, checking product details, or expresses happy emotions, it determines that the user has a strong intention to purchase.

[1254] Step 8:

[1255] When the server identifies a potential buyer, it immediately sends a notification to the salesperson, including the user's current location, the products they are interested in, and their current emotional state.

[1256] Step 9:

[1257] The salesperson, who is the user, receives the notification, reviews its contents, and then contacts the specific user based on the content of the notification to provide appropriate support. For example, the salesperson can provide the user with more information about the product or suggest other related products. Additionally, the salesperson can offer additional services or special offers to users who show particularly positive emotions.

[1258] Step 10:

[1259] The server records the results of salespeople's responses and data on interactions with users, which allows the effectiveness of sales activities to be evaluated and a database to be built for future use.

[1260] Step 11:

[1261] The server generates proposals to optimize the store's overall layout and display based on accumulated user behavior and emotion data. It analyzes whether specific areas or products are frequently visited and uses an optimization algorithm to propose layout changes.

[1262] Step 12:

[1263] The store manager, who is the user, reviews the proposals and takes steps to optimize the store layout and displays, thereby achieving efficient store operations and improving customer satisfaction.

[1264] Example 2

[1265] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1266] In modern retail stores, it is difficult for sales staff to accurately grasp users' purchasing intentions and provide timely support. Furthermore, there is a lack of methods to effectively optimize sales strategies and layouts by analyzing and utilizing user emotion and behavior data in real time. These problems need to be solved.

[1267] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for acquiring video data from a monitoring device installed in the store, a means for analyzing the acquired video data to detect the user's movements and facial expressions, a means for recognizing emotions based on the user's facial expression data, a means for tracking the user's stay time, location, and flow to learn behavioral patterns, a means for determining that a user exhibiting a specific behavioral pattern or emotion is a user considering purchasing, and a means for sending a notification to a notification target when a user considering purchasing is identified. This allows the user's purchasing intention to be properly understood and sales staff to provide timely support. Furthermore, analyzing user behavioral data can optimize store layout and improve sales strategies.

[1268] A "monitoring device" is a device that is placed in a store and captures users' movements and facial expressions in real time.

[1269] "Video data" refers to visual information including the user's movements and facial expressions acquired by a monitoring device.

[1270] "Analysis" is the process of processing video data, recognizing the user's movements and facial expressions, and extracting this as data.

[1271] "Facial expression data" is information that indicates emotions extracted from the user's facial movements and features.

[1272] "Emotion recognition" is the process of analyzing facial expression data to identify the emotion a user is expressing (e.g., joy, surprise, anger, sadness, etc.).

[1273] "Behavioral patterns" are the tendencies and characteristics of user behavior learned based on data such as the user's length of stay, location, and path.

[1274] A "purchase consideration user" is a user who is determined to have a high desire to purchase because they exhibit specific behavioral patterns and emotions based on analysis results and emotion recognition data.

[1275] The "notification recipient" is a person (e.g., a salesperson) who should receive a notification when a user considering purchase is identified.

[1276] A "notification" is a message containing information about a user considering purchasing that is sent from the server to the notification target.

[1277] "Layout optimization" is the process of optimizing the placement and display within a location based on tracked user behavior data.

[1278] This invention is a system that analyzes the behavior of users (customers) in a store in real time, identifies the user's purchase intentions, and notifies the salesperson. By combining this system with an emotion engine that recognizes the user's emotions, it is possible to provide more accurate support.

[1279] Camera image acquisition and video data analysis

[1280] The terminal acquires video data in real time from multiple monitoring devices (e.g., network cameras) placed in the store. The monitoring devices are placed to cover specific areas or product shelves. The server receives this video data and performs analysis. The analysis is performed using Python's OpenCV library and TensorFlow.

[1281] Detecting user movements and facial expressions

[1282] The server analyzes the user's movements and facial expressions based on the received video data. Specifically, it uses facial recognition technology and object detection algorithms to identify the user's location and products they are interested in. This analysis is performed using open-source computer vision libraries (e.g., OpenCV) and machine learning frameworks (e.g., TensorFlow).

[1283] Introducing the Emotion Engine

[1284] Based on the analyzed facial recognition data, the server uses an emotion engine to recognize the user's emotions. The emotion engine analyzes the user's facial muscle movements and other facial features to determine emotions such as happiness, surprise, anger, sadness, etc. This information, combined with the analysis of movements and facial expressions, is used to more precisely understand the user's overall behavioral patterns.

[1285] Tracking user behavior and learning patterns

[1286] The server tracks the user's time spent, location, and path in real time, allowing it to determine how long the user spent at a particular shelf and the path they took. For example, if a user spends 10 minutes in front of a particular shelf, it can determine that they are interested in that product.

[1287] Determining and notifying purchase intent

[1288] Based on the user's behavioral patterns and emotional data, the server identifies users who are deemed to have a strong intent to purchase as "considering purchase users." For example, if a user lingers in front of a particular product shelf for a long time, repeatedly picks up a product to check its details, or shows positive emotions (e.g., joy). When a considering purchase user is identified, the server immediately sends a notification to the salesperson.

[1289] Salesperson's response

[1290] The salesperson, who is the user, receives the notification sent to their device and checks its contents. The notification includes the user's current location, products of interest, and the user's current emotional state. Based on this, the salesperson contacts the specific user and provides detailed product information or suggests related products.

[1291] Accumulating user behavior data and optimizing layout

[1292] The server stores user behavior data and the results of sales staff responses in a database. This data is then analyzed to determine user patterns throughout the store. By understanding the popularity of specific areas and products, as well as user movement patterns, it becomes possible to optimize layouts and product displays to suit the characteristics of each store. For example, based on data showing that a specific product area is frequently visited on weekends, it can propose changes to the layout of that area and the placement of related products.

[1293] Examples of concrete examples and prompts

[1294] Examples:

[1295] For example, a user enters a store and heads to the food section. The device receives video data via a camera installed in the device, which is then analyzed by a server. The server recognizes that the user spends a long time in front of the shelves in the food section, picking up and looking at several products. It combines this information with emotional data to determine that the user is interested in those products. The server then sends a notification to a salesperson, who then checks the user's location, products of interest, and emotional state on the device. The salesperson then contacts the user, explains the details of the products, and suggests related products.

[1296] Example prompt for a generative AI model:

[1297] "What is the appropriate way to respond when a customer lingers in front of a particular shelf in a store for a long time?"

[1298] "Please explain the specific method for recognizing user emotions and determining purchasing intent from video data."

[1299] As described above, the present invention provides a system and an operating method for analyzing user behavior and emotions in real time to maximize sales opportunities.

[1300] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1301] Step 1: Acquire camera footage

[1302] The terminal acquires video data from surveillance equipment installed in the store. The surveillance equipment uses network cameras, each positioned to cover a specific area or shelf. For example, a camera captures a food section, and the video data is streamed to the terminal. The input is real-time video data, and the output is video data sent to a server via the terminal.

[1303] Step 2: Analyzing the video data

[1304] The server receives video data sent from the device. The server uses Python's OpenCV library to extract frames of the video data and perform preprocessing such as noise removal. Next, it uses TensorFlow to analyze the user's movements and facial expressions. For example, the server analyzes frames every second to detect the movements of the user's face and hands. The input is the captured video data, and the output is the analyzed user's movements and facial expression data.

[1305] Step 3: Detecting user movements and facial expressions

[1306] The server uses OpenCV's Haar Cascade and TensorFlow to identify the position of the user's face and hands and detect changes in facial expressions and movements. For example, it recognizes the moment when the user smiles or picks up a specific product. The input is the video frame extracted in the previous step, and the output is specific data on the user's movements and facial expressions.

[1307] Step 4: Emotion Recognition with the Emotion Engine

[1308] The server uses Affectiva's API to send the facial expression data obtained in the previous step and recognize the user's emotions in real time. The emotion engine determines emotions (happiness, surprise, anger, sadness, etc.) from the user's facial muscle movements and facial features. The input is the user's facial expression data, and the output is the recognized emotion data.

[1309] Step 5: Track user behavior and learn patterns

[1310] The server tracks the user's time spent, location, and path in real time and records this data. For example, the database stores information such as how long the user stayed in front of a particular shelf and the path they took. The input is the user's location and path data, and the output is behavioral patterns and learned user data.

[1311] Step 6: Determine and communicate purchase intent

[1312] The server analyzes the user's behavioral patterns and emotional data, and identifies users who are determined to have a strong intention to purchase as "considering purchase users." For example, this applies when a user lingers in front of a particular product shelf for a long time, picks up the product to check its details, or shows positive emotions (joy). The input is behavioral pattern data and emotional data, and the output is a notification to the salesperson.

[1313] Step 7: Salesperson response

[1314] The salesperson, who is the user, receives the notification sent to the terminal and checks the content. The notification includes the user's current location, products of interest, and emotional state. Based on this, the salesperson contacts the user to provide detailed product information and suggest related products. The input is the notification data from the server, and the output is specific support for the user.

[1315] Step 8: Accumulating user behavior data and optimizing layout

[1316] The server stores user behavior data and the results of sales staff responses in a database. This data is analyzed to understand the popularity of specific areas and products, as well as user movement patterns. For example, if a specific product area is frequently visited on weekends, the server will suggest changes to the layout of that area and the placement of related products. The input is the stored behavioral data, and the output is specific proposals for layout optimization.

[1317] Through these steps, the system can analyze user behavior and emotions in real time, enabling effective sales strategies and store layout optimization.

[1318] (Application example 2)

[1319] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1320] In traditional brick-and-mortar stores, it was difficult to analyze customer behavior and emotions in real time and provide appropriate support at the right time. This made it difficult to effectively stimulate customer purchasing desire, limiting store sales. Furthermore, data analysis to improve customer experience was insufficient, and store layout and product display were not optimized.

[1321] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1322] In this invention, the server includes: means for acquiring video data from cameras installed in the store; means for analyzing the acquired video data to detect user movements and facial expressions; means for tracking user stay time, location, and flow to learn behavioral patterns; means for recognizing user emotions based on the analyzed facial recognition data; means for determining that a user exhibiting a specific behavioral pattern and emotional state is a user considering a purchase; and means for sending a notification to a salesperson when a user considering a purchase is identified. This allows for timely notification to a salesperson based on the customer's behavior and emotions, enabling effective customer service. Furthermore, it is possible to optimize the store layout and product display based on user behavior data and emotional data, improving the customer experience and increasing sales.

[1323] "In-store" refers to the physical location where customers can actually touch and purchase products.

[1324] A "camera" is a visual sensor device for acquiring video data.

[1325] "Video data" refers to the digital information of images or video captured by a camera.

[1326] A "user" is a customer who performs actions such as viewing products in a store or making a purchase.

[1327] "Movement" refers to the movement or change in position of the user's body.

[1328] "Facial expressions" are the emotional expressions and muscle movements that appear on the user's face.

[1329] "Dwell time" refers to the amount of time a user stays in a particular location or on a shelf.

[1330] "Location" is information that indicates which area of ​​the store the user is in.

[1331] "Pathways" refer to the routes and patterns that users take within a store.

[1332] "Behavioral patterns" refer to habitual behavior that can be inferred from a user's series of actions, changes in location, length of stay, etc.

[1333] A "purchase consideration user" is a user who is determined to have a strong intention to purchase based on their behavioral patterns and emotional state.

[1334] A "notification" is a message sent to convey specific information.

[1335] A "salesperson" is a staff member who explains products and provides support to customers in a store.

[1336] "Facial recognition data" is digital information used to identify and analyze a user's face.

[1337] "Emotion" refers to a user's psychological state as judged from their facial expressions and behavior.

[1338] A "server" is a computer system that analyzes video data, stores various data, and sends notifications.

[1339] "Optimization" refers to the efficient rearrangement of layout and product displays based on data analysis.

[1340] The present invention relates to a system that analyzes customer behavior and emotions in real time in a store and provides appropriate support at the appropriate time. This system is realized by capturing video data using cameras placed in the store and analyzing that video data.

[1341] Acquiring camera footage

[1342] The server collects real-time video data from multiple cameras placed throughout the store, which are positioned to cover specific areas and shelves, allowing for comprehensive monitoring of customer behavior.

[1343] Video data analysis

[1344] The server analyzes the captured video data to detect the customer's movements and facial expressions. This analysis uses open-source computer vision libraries and machine learning frameworks such as OpenCV and TensorFlow. Specifically, it uses a facial recognition model to detect the customer's facial expressions and recognizes the customer's emotions based on the facial expression data.

[1345] Introducing the Emotion Engine

[1346] The server recognizes the customer's emotions using an emotion engine based on the analyzed facial recognition data. The emotion engine analyzes the customer's facial muscle movements and other facial features to determine emotions such as happiness, surprise, anger, sadness, etc. This information, combined with the analysis of movements and facial expressions, is used to more precisely understand the customer's overall behavioral patterns.

[1347] Tracking user time, location, and path

[1348] The server tracks customer dwell time, location, and path in real time, allowing it to determine how long a customer spends at a particular shelf and the path they take. This data is used to learn user behavior patterns.

[1349] Identifying and notifying potential purchasers

[1350] Based on the user's behavioral patterns and emotional data, the server identifies users who are determined to have a high intent to purchase as "purchase-consideration users." For example, if a user spends a long time in front of a particular product shelf, repeatedly picks up products to check their details, or shows positive emotions (e.g., joy), the server determines that the user has a high intent to purchase. When a user is identified as a purchase-consideration user, the server immediately sends a notification to the salesperson. The notification includes information about the user's current location, the products they are interested in, and their current emotional state.

[1351] Salesperson's response

[1352] Salespeople receive notifications on their devices, review the content, and contact specific users based on the content of the notifications to provide appropriate support. For example, they can provide detailed product information or suggest other related products. Additionally, they can offer additional services or special offers to users who show particularly positive emotions.

[1353] Accumulating user behavior data and optimizing layout

[1354] The server stores user behavior data and sales staff response results in a database. Based on this data, it analyzes user patterns throughout the store, understanding the popularity of specific areas and products, and user movement patterns. This makes it possible to optimize layouts and product displays to suit the characteristics of each store.

[1355] Examples of concrete examples and prompts

[1356] As a specific example, if customer A is seen smiling in the in-store camera footage and spends a long time in front of a particular product, the application will notify the salesperson of this information so that the salesperson can provide appropriate assistance.

[1357] Example prompt for a generative AI model:

[1358] "We will develop a system that analyzes video footage from multiple cameras in real time, recognizes faces and emotions, determines the purchasing intent of customers in a specific area, and notifies sales staff at the appropriate time."

[1359] Thus, the present invention provides a system and an operating method thereof for analyzing customer behavior and emotions in real time and maximizing sales opportunities.

[1360] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1361] Step 1:

[1362] Acquiring video data from the camera

[1363] The server receives real-time video data from cameras placed around the store, capturing images of specific areas of the store and product shelves. It receives the video data from the cameras as input and converts it into a processable format as output.

[1364] Step 2:

[1365] Video data analysis

[1366] The server detects the user's movements and facial expressions based on the acquired video data. Specifically, it uses OpenCV to perform face detection and identify the user's location. The input is video data from the camera, and the output is the location information of the detected face.

[1367] Step 3:

[1368] Emotion recognition based on facial recognition data

[1369] The server uses facial recognition data to recognize the user's emotions. Using a TensorFlow model, it analyzes the user's facial muscle movements and other features to determine emotions such as joy, surprise, and sadness. The input is the facial position information and video data obtained in step 2, and the output is the user's emotional state.

[1370] Step 4:

[1371] Tracking time spent, location, and traffic

[1372] The server tracks users' time spent, location, and route in real time, allowing it to determine how long they stay in a particular area and the route they take. The input is video data from the camera and facial recognition data, and the output is user behavior pattern data.

[1373] Step 5:

[1374] Identifying and notifying potential purchasers

[1375] The server identifies users who are deemed to have a high intent to purchase based on their behavioral patterns and emotional data as "consideration users." If identified, it sends a notification to the salesperson containing information about the user's current location, products they are interested in, and their emotional state. The input is the data obtained in Step 4 and Step 3, and the output is a notification to the salesperson.

[1376] Step 6:

[1377] Salesperson's response

[1378] The terminal allows the salesperson to receive the notification and provide support to the user based on its content. Specifically, it provides detailed product information and suggests other related products to the user. In particular, it suggests additional services and benefits to users who show positive emotions. The input is the notification sent to the salesperson's terminal, and the output is the result of the support provided.

[1379] Step 7:

[1380] Accumulating user behavior data and optimizing layout

[1381] The server accumulates user behavior data and salesperson response results in a database and analyzes user patterns throughout the store. This allows the popularity of specific areas within the store and user movement patterns to be understood, and layout optimization proposals are generated. The input is the data obtained at each step, and the output is layout and display optimization proposals.

[1382] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1383] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1384] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1385] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1386] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1387] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1388] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1389] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1390] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1391] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1392] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1393] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1394] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1395] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1396] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1397] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1398] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1399] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1400] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1401] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1402] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1403] The following is further disclosed regarding the above embodiment.

[1404] (Claim 1)

[1405] A means for acquiring video data from cameras installed in the store;

[1406] A means for analyzing the acquired video data to detect the user's movements and facial expressions;

[1407] A means of tracking user time spent, location, and navigation to learn behavioral patterns;

[1408] A method to determine whether users who exhibit specific behavioral patterns are potential purchasers, and

[1409] and means for sending a notification to a salesperson when a user considering a purchase is identified.

[1410] (Claim 2)

[1411] 2. The system according to claim 1, further comprising means for a salesperson who receives the notification to provide support to the user based on the content of the notification.

[1412] (Claim 3)

[1413] 10. The system of claim 1, further comprising means for generating recommendations for optimizing in-store layout and displays based on the tracked user behavior data.

[1414] (Claim 4)

[1415] 4. The system of claim 3, further comprising means for providing assistance to a store manager in modifying the layout and displays within the store based on the optimization suggestions.

[1416] "Example 1"

[1417] (Claim 1)

[1418] means for acquiring video data from a camera installed in the store;

[1419] A means for analyzing the acquired video data to detect the customer's movements and facial expressions;

[1420] A means to track customer dwell time, location, and path to learn behavioral patterns;

[1421] A means for determining whether a customer exhibiting a specific behavioral pattern is a potential purchaser;

[1422] and means for sending a notification to a salesperson when a potential purchaser is identified.

[1423] (Claim 2)

[1424] 2. The system according to claim 1, further comprising means for a salesperson who receives the notification to provide support to the customer based on the content of the notification.

[1425] (Claim 3)

[1426] 10. The system of claim 1, further comprising means for generating recommendations for optimizing in-store layout and display based on the tracked customer behavior data.

[1427] "Application Example 1"

[1428] (Claim 1)

[1429] A means for acquiring video data from cameras installed in the store;

[1430] A means for analyzing the acquired video data to detect the user's movements and facial expressions;

[1431] A means of tracking user time spent, location, and navigation to learn behavioral patterns;

[1432] A method to determine whether users who exhibit specific behavioral patterns are potential purchasers, and

[1433] a means for sending a notification to a salesperson when a potential purchaser is identified;

[1434] a means of sending real-time notifications to sales associates using smart devices;

[1435] A system including:

[1436] (Claim 2)

[1437] 2. The system according to claim 1, further comprising means for a salesperson who receives the notification to provide support to the user based on the content of the notification.

[1438] (Claim 3)

[1439] 10. The system of claim 1, further comprising means for generating recommendations for optimizing in-store layout and displays based on the tracked user behavior data.

[1440] "Example 2: Combining Emotion Engines"

[1441] (Claim 1)

[1442] means for acquiring video data from a monitoring device installed in the store;

[1443] A means for analyzing the acquired video data to detect the user's movements and facial expressions;

[1444] A means for recognizing emotions based on the user's facial expression data;

[1445] A means of tracking user time spent, location, and navigation to learn behavioral patterns;

[1446] A method to identify users who exhibit specific behavioral patterns or emotions as potential purchasers, and

[1447] A means for sending a notification to a notification target when a user considering purchase is identified;

[1448] A system including:

[1449] (Claim 2)

[1450] 2. The system according to claim 1, further comprising means for a notification target that receives the notification to provide support to the user based on the content of the notification.

[1451] (Claim 3)

[1452] 10. The system of claim 1, further comprising means for generating recommendations for optimizing placement and display within the location based on the tracked user behavior data.

[1453] "Application example 2 when combining emotion engines"

[1454] (Claim 1)

[1455] A means for acquiring video data from cameras installed in the store;

[1456] A means for analyzing the acquired video data to detect the user's movements and facial expressions;

[1457] A means of tracking user time spent, location, and navigation to learn behavioral patterns;

[1458] means for recognizing a user's emotion based on the analyzed facial recognition data;

[1459] A means for determining whether a user exhibiting a particular behavioral pattern and emotional state is a potential purchaser;

[1460] and means for sending a notification to a salesperson when a user considering a purchase is identified.

[1461] (Claim 2)

[1462] 2. The system according to claim 1, further comprising means for a sales representative who receives the notification to provide support to the user based on the content of the notification.

[1463] (Claim 3)

[1464] 10. The system of claim 1, further comprising means for generating recommendations for optimizing in-store layout and displays based on the tracked user behavioral and sentiment data. [Explanation of symbols]

[1465] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. A means for acquiring video data from cameras installed in the store; A means for analyzing the acquired video data to detect the user's movements and facial expressions; A means of tracking user time spent, location, and navigation to learn behavioral patterns; A method to determine whether users who exhibit specific behavioral patterns are potential purchasers, and and means for sending a notification to a salesperson when a user considering a purchase is identified.

2. 2. The system according to claim 1, further comprising means for a salesperson who receives the notification to provide support to the user based on the content of the notification.

3. The system of claim 1 further comprising means for generating recommendations for optimizing in-store layout and displays based on the tracked user behavior data.

4. The system of claim 3 further comprising means for providing assistance to a store manager in modifying the layout and displays within the store based on the optimization suggestions.

Images