System

Abstract

A system is provided.SOLUTION: The system includes a means for acquiring video data of a visitor, a means for extracting appearance information of the visitor from the acquired video data, a means for estimating the annual income of the visitor on the basis of the extracted appearance information, a means for storing estimated annual income data in a database, and a means for analyzing the stored annual income data to support decision making related to store operation.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] For brick-and-mortar store managers, annual income data of visitors is useful for marketing purposes, as it allows them to understand the potential purchase amount. However, if no products are purchased or if payment data about visitors is not available, it is difficult to collect information about visitors. Therefore, an effective data collection and analysis method for estimating the annual income of visitors is required. [Means for solving the problem]

[0005] The present invention solves the above-mentioned problems by providing a system including means for acquiring video data of visitors and extracting appearance information of the visitors from the video data, means for estimating the annual income of the visitors based on the extracted appearance information, means for storing the estimated annual income data in a database, and means for analyzing the stored annual income data to support decision-making regarding store management. Furthermore, by further including means for recommending products suitable for the visitors based on the annual income data and means for estimating the age and gender of the visitors from the video data, the accuracy of the estimation and practical use in physical stores are improved.

[0006] "Video data" refers to a digital video signal containing images of visitors captured by a video device such as a surveillance camera.

[0007] "Appearance information" refers to visual characteristics extracted from video data, such as the visitor's age, gender, clothing, belongings, and type of shoes.

[0008] "Annual income" refers to the total amount of income a visitor earns in a year, and is a value estimated using machine learning models based on appearance information.

[0009] A "database" is an information system that allows for the systematic management, analysis, and utilization of accumulated data.

[0010] A "machine learning model" is an algorithm or mathematical framework that uses training data to learn a specific task and then makes inferences or predictions about new data.

[0011] "Age estimation" is the process of estimating a visitor's age based on appearance information obtained from video data.

[0012] "Gender estimation" is the process of guessing a visitor's gender based on appearance information obtained from video data.

[0013] "Product recommendation" is a function that selects and suggests suitable products based on visitor information such as annual income data. [Brief explanation of the drawings]

[0014] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0015] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0016] First, the terms used in the following description will be explained.

[0017] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0018] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0019] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0020] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0021] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0022] [First embodiment]

[0023] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0024] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0025] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0026] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0027] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0028] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0029] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0030] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0031] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0032] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0033] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0034] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0035] This invention is a system for efficiently collecting annual income data of customers and utilizing it for decision-making in store management. This system is realized by analyzing video data acquired from video devices such as surveillance cameras, extracting information about the appearance of customers, and estimating their annual income.

[0036] System Overview

[0037] The system mainly consists of the following modules:

[0038] 1. Video data collection module (terminal)

[0039] 2. Video analysis module (server)

[0040] 3. Annual income estimation module (server)

[0041] 4. Database storage module (server)

[0042] 5. Data analysis module (server)

[0043] 6. Product recommendation module (terminal)

[0044] 7. Customer Service Support Module (User)

[0045] Program processing

[0046] Video data collection module (terminal)

[0047] The device acquires video data in real time from surveillance cameras installed in the store. This video data records how customers move around the store after entering. For example, video of a customer entering the store at 10:00 a.m. is acquired.

[0048] Video analysis module (server)

[0049] The server receives the video data sent from the device and uses an image recognition algorithm to extract information about the visitor's appearance, including their age, gender, clothing, belongings, and type of shoes. For example, the captured video may identify the visitor as a man around 35 years old wearing a business suit.

[0050] Annual income estimation module (server)

[0051] The server uses a machine learning model to estimate the visitor's annual income based on the extracted appearance information. This machine learning model is pre-trained and has learned the correlation between appearance information and annual income. For example, a visitor wearing a luxury watch is estimated to have an annual income of 8 million yen.

[0052] Database storage module (server)

[0053] The server stores the estimated income data in a database, which systematically stores income data and other appearance information for each visitor, for example, the income data of visitors who visited at a specific date and time.

[0054] Data analysis module (server)

[0055] The server statistically analyzes the accumulated annual income data to determine what income brackets are attracting customers to a particular store and what products are being purchased. The results of this analysis are used to improve the store's product lineup and formulate new store opening plans. For example, it can be determined that a particular store is frequented by customers with an annual income of 8 million yen or more.

[0056] Product recommendation module (terminal)

[0057] The device recommends products suitable for visitors based on the annual income data sent from the server. It lists expensive products for visitors with high annual incomes and affordable products for visitors with low annual incomes. For example, a visitor with an annual income of 5 million yen will be shown a list of mid-priced products.

[0058] Customer service support module (user)

[0059] The user, a store clerk, checks the display on the device and suggests appropriate products to the customer. This creates an environment where customers can easily make purchases. For example, a store clerk suggests a luxury bag to a customer wearing an expensive suit.

[0060] Specific examples

[0061] Suppose a woman in her 30s visits a store one afternoon. The device acquires video data from the surveillance camera and sends it to the server. The server analyzes the video and extracts information about the woman's appearance, such as her age, gender, clothing, and belongings. Based on this information, a machine learning model estimates her annual income, calculating it as 4 million yen, for example. The server stores this information in a database and analyzes it together with data from other visitors. The device then lists products suitable for a visitor with an annual income of 4 million yen and displays it to the user, who is also a store clerk. The clerk uses this information to suggest products to the visitor.

[0062] This system allows annual income data to be collected even if visitors do not purchase products, and can be used to make decisions about store management.

[0063] The processing flow will be explained below.

[0064] Step 1:

[0065] The device captures real-time video data from the store's surveillance cameras, recording customers entering and moving around the store.

[0066] Step 2:

[0067] The device temporarily stores the acquired video data and periodically transmits it to the server, for example, transmitting frame data every second.

[0068] Step 3:

[0069] The server receives the video data sent from the terminal, and the received video data is input to the image analysis module.

[0070] Step 4:

[0071] The server uses video analysis algorithms to extract information about the visitor's facial and physical appearance from the video data, including facial recognition, gender estimation, and age estimation. For example, facial recognition technology is used to identify the visitor's face and estimate their gender as "male" and their age as "around 35 years old."

[0072] Step 5:

[0073] The server extracts further appearance information such as the visitor's clothing, belongings, and type of shoes, including attributes such as color, brand, and fashion style. For example, it may be recognized that the visitor is wearing a business suit and a luxury watch.

[0074] Step 6:

[0075] The server inputs the extracted appearance information into a machine learning model. The machine learning model learns the correlation between appearance information and annual income based on past data, and estimates the visitor's annual income. For example, if a visitor is wearing a business suit and a luxury watch, it can estimate that the visitor's annual income is 8 million yen.

[0076] Step 7:

[0077] The server stores the estimated annual income data in a database. The stored data includes the visitor's age, gender, appearance information, and estimated annual income. For example, data such as "male," "around 35 years old," and "8 million yen" might be stored.

[0078] Step 8:

[0079] The server statistically analyzes the income data stored in the database to determine which income brackets are most frequent visitors to a particular store and what products are being purchased. For example, it may analyze whether a particular store is frequented by a large number of people in the high income bracket.

[0080] Step 9:

[0081] The device generates a product list that recommends the most suitable products to each visitor based on the estimated annual income data. For example, it would recommend luxury bags to a visitor with an annual income of 8 million yen.

[0082] Step 10:

[0083] The terminal displays the generated product list to the user, who is also a store clerk. The store clerk checks the displayed product list and suggests suitable products to the customer. For example, they may explain where to find luxury bags and recommend trying them on.

[0084] This allows the system to collect annual income data even if visitors do not purchase products, and use it to improve the store's product lineup and plan new stores.It can also increase purchasing motivation by suggesting products that are suitable for visitors.

[0085] Example 1

[0086] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0087] In current store operations, it is difficult to obtain data on the annual income of customers, limiting the effectiveness of marketing strategies and product recommendations. It has also been difficult to establish a system that estimates annual income based on information about customers' appearance and then uses that data to recommend specific products and make operational decisions.

[0088] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0089] In this invention, the server includes means for collecting video data of visitors from a video capture device, means for compressing the collected video data and transmitting it to a data server, means for analyzing appearance information of visitors from the video data in the data server, means for estimating the annual income of visitors from the appearance information using a machine learning model, means for storing the estimated annual income data in a database, means for statistically analyzing the stored annual income data and utilizing it in store management, and means for recommending products suitable for visitors based on the analysis results. This makes it possible to efficiently collect annual income data of visitors, improve the accuracy of marketing strategies and product recommendations, and support decision-making in store management.

[0090] The "image capture device" is a device for collecting image data of visitors.

[0091] The "means for compressing collected video data and transmitting it to a data server" refers to a means for effectively compressing collected video data and transmitting it to a data server using a secure communication method.

[0092] The "means for analyzing visitor appearance information from video data using a data server" refers to a means for analyzing video data and extracting visitor appearance information (age, gender, clothing, belongings, etc.).

[0093] The "means for estimating the annual income of a visitor from appearance information using a machine learning model" is a means for estimating the annual income of a visitor based on acquired appearance information using machine learning technology.

[0094] "Means for storing in a database" means a means for systematically storing all relevant information, including estimated annual income data.

[0095] "Means of statistically analyzing accumulated annual income data and utilizing it in store management" refers to a means of conducting statistical analysis based on accumulated annual income data and using the results to help with store management strategies and decision-making.

[0096] The "means for recommending products suitable for visitors based on the analysis results" refers to a means for selecting and recommending the most suitable products to each visitor using the results of statistical analysis.

[0097] The present invention provides a system for efficiently collecting annual income data of customers and utilizing it for decision-making in store management. The system includes an image capture device, a data server, a machine learning model, a database, a statistical analysis means, and a product recommendation means.

[0098] First, the terminal collects real-time video data of visitors using a video capture device installed in the store. The collected video data is compressed using video compression formats such as H.264 or H.265 and sent to a data server via HTTPS. The data server analyzes the received video data and extracts information about the visitor's appearance using image recognition algorithms such as OpenCV and TensorFlow. This information includes age, gender, clothing, and belongings.

[0099] The server then uses a machine learning model to estimate the visitor's annual income from the extracted appearance information. This machine learning model has been trained in advance on a large dataset and has learned the correlation between appearance information and annual income. For example, it can estimate the annual income of a visitor wearing a luxury watch to be 8 million yen.

[0100] The estimated annual income data is stored in a database on the server. This database systematically stores information such as the age, gender, and annual income of each visitor. The server analyzes the accumulated data using statistical analysis libraries such as Pandas, providing useful insights into store operations.

[0101] Furthermore, the server generates a list of recommended products suitable for the visitor based on the analysis results and sends this information to the terminal. The terminal displays the received product recommendation list, and the user (store clerk) suggests appropriate products to the visitor based on this list. Specifically, the server displays a list of expensive products for visitors with high annual incomes and affordable products for visitors with low annual incomes to the user.

[0102] Specific examples

[0103] Suppose a woman in her 30s visits a store one afternoon. The device acquires video data from the surveillance camera and sends it to the server. The server analyzes the video and extracts information about the woman's appearance, such as her age, gender, clothing, and belongings. Based on this information, a machine learning model estimates her annual income, calculating it as 4 million yen, for example. The server stores this information in a database and analyzes it together with data from other visitors. The device then lists products suitable for a visitor with an annual income of 4 million yen and displays it to the user, who is also a store clerk. The clerk uses this information to suggest products to the visitor.

[0104] Prompt Sentence Examples

[0105] Here are some example prompts to explain the system's operations to the generative AI model:

[0106] "Surveillance cameras installed in stores capture footage of customers, and the video data is analyzed to extract information about the customer's appearance (age, gender, clothing, belongings, etc.). Next, a machine learning model is used to estimate the customer's annual income from this appearance information and the information is stored in a database. Based on the stored data, appropriate products are recommended to the customer, and finally, a store associate suggests appropriate products to the customer. Please explain this process step by step."

[0107] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0108] Step 1:

[0109] The terminal collects video data of visitors in real time from a video capture device installed in the store. This input data records how customers move around the store after entering. Specifically, the surveillance camera captures video at about 60 frames per second and stores it on the terminal.

[0110] Step 2:

[0111] The terminal compresses the collected video data using video compression methods such as H.264 or H.265 and sends it to the data server using the HTTPS protocol. This reduces the amount of data traffic and ensures secure transmission. Specifically, the video data is compressed using batch processing and uploaded to the server at regular intervals.

[0112] Step 3:

[0113] The server receives the video data sent from the device and analyzes the visitor's appearance using image recognition algorithms such as OpenCV and TensorFlow. The input data is video frames, and the output extracts appearance information such as the visitor's age, gender, clothing, and belongings. Specifically, a deep learning model for video analysis processes the image data for each frame and extracts specific features.

[0114] Step 4:

[0115] The server uses a machine learning model (e.g., a sparse ridge regression model) to estimate the visitor's annual income from the appearance information. This model is trained in advance using a training dataset. Based on the appearance information extracted as input, the visitor's annual income is estimated as output. Specifically, the model performs a regression analysis using the appearance information variables to calculate an estimated annual income.

[0116] Step 5:

[0117] The server accumulates the estimated annual income data in a database. The input data is the estimated annual income data, and the output is saved in the database. Specifically, the server uses an SQL query to register the annual income data in the database, linking it with the visitor's ID.

[0118] Step 6:

[0119] The server statistically analyzes the accumulated annual income data. The input data is the annual income data stored in the database, and the output provides analysis reports and data trends. Specifically, it converts the data into a data frame using the Pandas library and calculates various statistics (such as the mean, median, and standard deviation).

[0120] Step 7:

[0121] The server generates a list of recommended products suitable for the visitor based on the analysis results. The input data is annual income data and the analysis results, and the output is a list of recommended products. Specifically, the server filters products from the product database according to annual income category and generates the list.

[0122] Step 8:

[0123] The terminal receives the product recommendation list sent from the server and displays it to the user, who is a store clerk. The input data is the product recommendation list, and the output is displayed on the terminal's display. Specifically, the terminal parses the received data and updates the interface displayed on the screen.

[0124] Step 9:

[0125] The user, a store clerk, refers to the product recommendation list displayed on the terminal and suggests appropriate products to the visitor. The input data is the information displayed on the terminal, and the output is specific product suggestions to the visitor. In concrete terms, the store clerk explains the features and benefits of the products to the visitor based on the displayed list, encouraging them to make a purchase.

[0126] (Application example 1)

[0127] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0128] In store operations, efficiently collecting annual income data of customers and using that data to recommend products and provide customer service support contributes to improving store profits and customer satisfaction. However, conventional methods have made it difficult to efficiently collect such data before customers make a purchase, and there has been a lack of concrete means to utilize the collected data in store management decision-making. Furthermore, there has been no system that can make product recommendations tailored to customers in real time.

[0129] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0130] In this invention, the server includes means for acquiring video data of visitors, means for extracting appearance information of the visitors from the acquired video data, means for estimating the annual income of the visitors based on the extracted appearance information, means for storing the estimated annual income data in a database, means for recommending products suitable for the visitors based on the stored annual income data, means for displaying information about the recommended products on a display device and supporting store staff in making suggestions to the visitors, and means for statistically analyzing the annual income data stored in the database and analyzing specific income brackets. This makes it possible to collect annual income data even if the visitors do not purchase products, and to utilize the data to recommend products and provide customer service support in real time.

[0131] "Visitor" refers to an individual who visits a store with the intent of using its services or products.

[0132] "Video data" refers to image and video information acquired from video devices such as surveillance cameras within a store.

[0133] "Appearance information" refers to visual characteristics of visitors, such as their age, gender, clothing, and belongings, extracted from video data.

[0134] "Annual income" refers to the estimated annual income of the visitor.

[0135] "Estimation" refers to predicting a visitor's annual income using a machine learning model based on acquired appearance information, etc.

[0136] "Database" refers to a system for storing and managing accumulated annual income data and other information.

[0137] "Product recommendation" refers to presenting the most suitable products to visitors based on accumulated annual income data.

[0138] "Display device" refers to a device such as a smartphone or tablet that displays product recommendation information and other notifications.

[0139] "Store staff" refers to employees who support sales activities in stores.

[0140] "Customer service support" refers to assistance provided to store staff to efficiently recommend products and provide services to customers.

[0141] "Statistical analysis" refers to a method for analyzing accumulated data and deriving specific patterns or trends.

[0142] This invention is a system for efficiently collecting annual income data of customers and using it to make decisions about store management, and includes the following components. Specifically, we will explain in detail the roles of the server and terminals.

[0143] Video data collection module (terminal)

[0144] The device captures real-time video data from high-resolution IP cameras installed in the store, recording customers as they enter and move around the store. The video data is then sent to a server via Wi-Fi.

[0145] Video analysis module (server)

[0146] The server receives the video data sent from the device and uses an image recognition algorithm to extract information about the visitor's appearance, including their age, gender, clothing, belongings, etc. Specifically, the server performs image analysis using OpenCV, TensorFlow, and the Google Cloud Vision API.

[0147] Annual income estimation module (server)

[0148] The server uses a machine learning model to estimate the visitor's annual income based on the extracted appearance information. This machine learning model is pre-trained and has learned the correlation between appearance information and annual income. Specific technologies used are scikit-learn and Python.

[0149] Database storage module (server)

[0150] The server stores the estimated annual income data in a database, which stores the annual income data and other appearance information of each visitor in an organized manner. The database is managed using MySQL or SQLite.

[0151] Data analysis module (server)

[0152] The server statistically analyzes the accumulated income data to determine what income brackets are attracting customers to specific stores and what products are being purchased. The analysis is performed using R and Python (Pandas, NumPy).

[0153] Product recommendation module (terminal)

[0154] The device recommends products suitable for visitors based on the annual income data sent from the server. It lists expensive products for visitors with high annual incomes and affordable products for visitors with low annual incomes. This is done using cloud-based data communication using Firebase.

[0155] Customer service support module (user)

[0156] Store staff, who are the users, can use their smartphones or tablets to check the displayed product recommendation information and help suggest appropriate products to customers. The built-in notification function and user interface enable efficient customer service.

[0157] Specific examples

[0158] For example, suppose a woman in her 30s visits a store one afternoon. Video data is acquired from a surveillance camera and sent to a server. The server analyzes the video and extracts information about the woman's appearance, such as her age, gender, clothing, and belongings. Based on this information, a machine learning model estimates her annual income, calculating it as, say, 4 million yen. The server stores this information in a database and analyzes it together with data from other visitors. The device then lists products suitable for a visitor with an annual income of 4 million yen and displays it to the store clerk. The clerk then uses this information to suggest products to the visitor.

[0159] Example prompts for generative AI models

[0160] "Estimate the annual income of women in their 30s and recommend products": Build a system that estimates the annual income of visitors from their appearance information (age, gender, clothing, belongings) and recommends appropriate products based on that income. Using the movie example, when a woman in her 30s enters a store, the system analyzes her age and clothing, estimates her annual income to be 4 million yen, and displays mid-priced products on a smartphone app. The store clerk uses this information to suggest products that suit her.

[0161] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0162] Step 1:

[0163] The terminal acquires video data in real time from high-resolution IP cameras installed in the store. Specifically, it captures images and videos of visitors through the camera lens and transmits the video data to a server via Wi-Fi connection. The input is the video data from the camera, and the output is the video data transmitted to the server via Wi-Fi.

[0164] Step 2:

[0165] The server receives the video data sent from the device. It uses an image recognition algorithm to extract the visitor's appearance information from the received video data. Specifically, it uses OpenCV, TensorFlow, and Google Cloud Vision API to analyze and extract characteristic information such as the visitor's age, gender, clothing, and belongings. The input is the video data, and the output is the extracted appearance information.

[0166] Step 3:

[0167] The server uses a machine learning model to estimate the visitor's annual income based on the extracted appearance information. Specifically, a pre-trained model using scikit-learn and Python is used to evaluate the correlation between appearance information and annual income and predict annual income. The input is appearance information, and the output is estimated annual income data.

[0168] Step 4:

[0169] The server stores the estimated annual income data and appearance information in a database. This database stores the annual income data and appearance information for each visitor in an organized manner. Specifically, data is managed efficiently using MySQL and SQLite. The input is annual income data and appearance information, and the output is data storage in the database.

[0170] Step 5:

[0171] The server statistically analyzes the income data stored in the database and analyzes the trends of visitors in specific income brackets. Specifically, it performs data analysis using R and Python (Pandas, NumPy). The input is the income data from the database, and the output is the analysis results.

[0172] Step 6:

[0173] The device recommends suitable products to visitors based on the analysis results sent from the server. Specifically, Firebase is used for cloud-based data communication and a list of recommended products is displayed on the smartphone or tablet. The input is the analysis results, and the output is the display data of the recommended products.

[0174] Step 7:

[0175] Store staff, who are users, check the product recommendation information displayed on their smartphones or tablets and suggest suitable products to visitors. Specifically, they check product information through the built-in notification function and user interface and provide appropriate customer service. The input is the display data of recommended products, and the output is the product suggestions to visitors.

[0176] By following the above steps, annual income data can be collected even if the visitor does not purchase a product, and this data can be used to recommend products and provide customer service support in real time.

[0177] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0178] This invention is a system for efficiently collecting annual income data and emotional data of customers and using it to help with store management decision-making and customer service. This system is realized by analyzing video data acquired from video devices such as surveillance cameras, extracting information on the appearance and emotional information of customers, and estimating their annual income.

[0179] System Overview

[0180] The system mainly consists of the following modules:

[0181] 1. Video data collection module (terminal)

[0182] 2. Video analysis module (server)

[0183] 3. Annual income estimation module (server)

[0184] 4. Emotion engine (server)

[0185] 5. Database storage module (server)

[0186] 6. Data analysis module (server)

[0187] 7. Product recommendation module (terminal)

[0188] 8. Customer Service Support Module (User)

[0189] Program processing

[0190] Video data collection module (terminal)

[0191] The device acquires video data in real time from surveillance cameras installed in the store. This video data records how customers move around the store after entering. For example, video of a customer entering the store at 10:00 a.m. is acquired.

[0192] Video analysis module (server)

[0193] The server receives the video data sent from the device and uses an image recognition algorithm to extract information about the visitor's face and body appearance. This information includes the visitor's age, gender, clothing, belongings, and type of shoes. For example, the captured video may identify the visitor as a man around 35 years old and wearing a business suit.

[0194] Annual income estimation module (server)

[0195] The server uses a machine learning model to estimate the visitor's annual income based on the extracted appearance information. This machine learning model is pre-trained and has learned the correlation between appearance information and annual income. For example, a visitor wearing a luxury watch is estimated to have an annual income of 8 million yen.

[0196] Emotion engine (server)

[0197] The server is equipped with an emotion engine that recognizes visitors' emotions from video data. This emotion engine identifies the visitor's emotional state (e.g., happiness, surprise, irritation, etc.) through facial expression analysis. For example, if a visitor smiles while looking at a product, the emotion engine recognizes this as "happiness."

[0198] Database storage module (server)

[0199] The server stores the estimated annual income data and emotion data in a database. This database systematically stores annual income data, emotion data, and other appearance information for each visitor. For example, data such as "male," "around 35 years old," "8 million yen," and "happiness" might be stored.

[0200] Data analysis module (server)

[0201] The server statistically analyzes the accumulated income data and emotional data to determine which income brackets are most frequent visitors to a particular store and what emotional state they are in when looking at products. The results of this analysis are used to improve the store's product lineup and develop new store plans. For example, it may be possible to determine that a particular store is visited by many high-income earners, or that visitors feel "happy" when looking at a particular product.

[0202] Product recommendation module (terminal)

[0203] The device generates a product list that recommends the most suitable products to the visitor based on the annual income data and emotional data sent from the server. Expensive products are listed for visitors with high annual incomes, while affordable products are listed for visitors with low annual incomes, and the device further recommends the most suitable products according to the visitor's emotional state. For example, if a visitor with an annual income of 8 million yen has a happy expression, a list of recommended luxury bags will be displayed first.

[0204] Customer service support module (user)

[0205] The user, a store clerk, checks the display on the device and suggests appropriate products to the customer. This creates an environment where customers can easily make a purchase. For example, if a customer smiles while looking at a luxury bag, the clerk can suggest a luxury bag and encourage them to try on the luxurious bag.

[0206] Specific examples

[0207] Suppose a woman in her 30s visits a store one afternoon. The device acquires video data from the surveillance camera and sends it to the server. The server analyzes the video and extracts information about the woman's appearance, such as her age, gender, clothing, and belongings. It also uses an emotion engine to recognize emotions such as "happiness" from the woman's facial expression. Based on this information, a machine learning model estimates her annual income, calculating it as 4 million yen, for example. The server stores this information in a database and analyzes it together with data from other visitors. The device then lists products suitable for a visitor with an annual income of 4 million yen who is emotionally happy, and displays this to the user, who is also a store clerk. The store clerk then uses this information to suggest products to the visitor.

[0208] This system can collect data on customers' annual income and emotions, even if they do not purchase products, and use this data to improve the store's product lineup and plan new stores. It can also provide more effective customer service by suggesting products based on the customer's emotional state.

[0209] The processing flow will be explained below.

[0210] Step 1:

[0211] The device captures real-time video data from the store's surveillance cameras, recording customers entering and moving around the store.

[0212] Step 2:

[0213] The device temporarily stores the acquired video data and periodically transmits it to the server, for example, transmitting frame data every second.

[0214] Step 3:

[0215] The server receives the video data sent from the terminal, and the received video data is input to the image analysis module.

[0216] Step 4:

[0217] The server uses video analysis algorithms to extract information about the visitor's facial and physical appearance from the video data, including facial recognition, gender estimation, and age estimation. For example, facial recognition technology is used to identify the visitor's face and estimate their gender as "male" and their age as "around 35 years old."

[0218] Step 5:

[0219] The server extracts further appearance information such as the visitor's clothing, belongings, and type of shoes, including attributes such as color, brand, and fashion style. For example, it may be recognized that the visitor is wearing a business suit and a luxury watch.

[0220] Step 6:

[0221] The server inputs the extracted appearance information into a machine learning model. The machine learning model learns the correlation between appearance information and annual income based on past data, and estimates the visitor's annual income. For example, if a visitor is wearing a business suit and a luxury watch, it can estimate that the visitor's annual income is 8 million yen.

[0222] Step 7:

[0223] The server runs an emotion engine that recognizes the visitor's emotions from the video data. The emotion engine identifies the visitor's emotional state (e.g., happiness, surprise, irritation, etc.) through facial expression analysis. For example, if a visitor is looking at a product and smiling, the emotion engine recognizes the visitor as "happy."

[0224] Step 8:

[0225] The server stores the estimated annual income data and emotional data in a database. The stored data includes the visitor's age, gender, appearance information, annual income, and emotional state. For example, the data stored might be "male," "around 35 years old," "8 million yen," and "happy."

[0226] Step 9:

[0227] The server statistically analyzes the income data and emotional data stored in the database to determine which income brackets are most frequent visitors to a particular store and what emotional state they are in when looking at products. The results of this analysis are used to improve the store's product lineup and develop new store plans. For example, it may be possible to determine that a particular store is visited by many high-income earners, or that visitors feel "happy" when looking at a particular product.

[0228] Step 10:

[0229] The device generates a product list that recommends the most suitable products to the visitor based on the annual income data and emotional data sent from the server. Expensive products are listed for visitors with high annual incomes, while affordable products are listed for visitors with low annual incomes, and the device further recommends the most suitable products according to the visitor's emotional state. For example, if a visitor with an annual income of 8 million yen has a happy expression, a list of recommended luxury bags will be displayed first.

[0230] Step 11:

[0231] The terminal displays the generated product list to the user, who is also a store clerk. The store clerk checks the displayed product list and suggests appropriate products to the visitor. For example, if a visitor smiles while looking at a luxury bag, the store clerk will suggest a luxury bag and recommend that the visitor try on a luxurious bag.

[0232] Example 2

[0233] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0234] In conventional store operations, the collection of information about customers was limited, making it difficult to grasp details such as the annual income and emotional state of customers. As a result, effective decision-making and product recommendations suited to customers were not fully implemented in store operations. In addition, it was difficult to provide individual support to stimulate customers' purchasing desire, resulting in a situation where sales could not be expected to increase.

[0235] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0236] In this invention, the server includes means for extracting appearance information of visitors from video data, means for estimating annual income based on the extracted appearance information, and means for analyzing the emotional state of visitors. This makes it possible to grasp annual income and emotions based on detailed information about visitors, support decision-making for store management, and suggest products suitable for visitors.

[0237] A "terminal" is a device that acquires video data of visitors and transmits it to a server.

[0238] A "server" is a computer system that analyzes video data, processes data such as appearance information, annual income, and emotional state, and stores and analyzes this data in a database.

[0239] "Video Data" refers to image and video data captured to record the appearance and actions of visitors.

[0240] "Appearance information" is data extracted from a visitor's age, gender, clothing, belongings, type of shoes, and other visual characteristics.

[0241] "Annual income" is the amount of income a visitor is estimated to earn per year.

[0242] "Emotional state" refers to the emotional state analyzed from the visitor's facial expression, and includes happiness, surprise, irritation, etc.

[0243] A "database" is a digital storage device for systematically storing information such as visitors' appearance, annual income, and emotional state.

[0244] "Data analysis" is the process of using data stored in a database to analyze trends and patterns related to customer visitation and store operations.

[0245] A "product list" is a list of products generated to recommend the most suitable products to visitors.

[0246] "User" refers to a store employee or salesperson who uses the system to suggest suitable products to visitors.

[0247] MODE FOR CARRYING OUT THE INVENTION

[0248] This invention is a system that efficiently collects annual income data and emotional data of customers and uses it to help with decision-making in store management and customer service. This system functions in cooperation with three entities: terminals, servers, and users.

[0249] Hardware and software used

[0250] The device includes a surveillance camera installed in the store that captures video data in real time. This video data is used to record how customers move around the store after entering. For example, a video of a customer entering the store at 10:00 AM is captured.

[0251] The server receives the video data sent from the device and uses an image recognition algorithm (such as TensorFlow or OpenCV) to analyze it. The server extracts the visitor's appearance information (age, gender, clothing, belongings, etc.). For example, the server recognizes from the captured video that the visitor is a man around 35 years old wearing a business suit.

[0252] Next, the server uses a machine learning model (for example, a model using scikit-learn or TensorFlow) to estimate the visitor's annual income based on the extracted appearance information. For example, a visitor wearing a luxury watch is estimated to have an annual income of 8 million yen.

[0253] The server is also equipped with an emotion engine that recognizes visitors' emotions from video data. This emotion engine identifies the visitor's emotional state (happiness, surprise, irritation, etc.) through facial expression analysis. For example, if a visitor smiles while looking at a product, the emotion engine will recognize this as "happiness."

[0254] The server stores the estimated annual income data and emotion data in a database. This database stores the annual income data, emotion data, and other appearance information for each visitor in an organized format. For example, data such as "male," "around 35 years old," "8 million yen," and "happy" might be stored.

[0255] The data stored in the database is statistically analyzed by a server. The analysis results provide information such as which income brackets are most frequent visitors to a particular store and the emotional state customers are in when looking at products. These analysis results are used to improve the store's product lineup and formulate new store opening plans. For example, it may be discovered that a particular store is visited by many high-income earners, or that visitors feel "happy" when looking at a particular product.

[0256] Specific examples

[0257] Let's say a woman in her 30s visits a store one afternoon. The device acquires video data from the surveillance camera and sends it to the server. The server analyzes the video and extracts information about the woman's appearance, such as her age, gender, clothing, and belongings. It also uses an emotion engine to recognize emotions such as "happiness" from the woman's facial expression. Based on this information, a machine learning model estimates her annual income, calculating it as 4 million yen, for example. The server stores this information in a database and analyzes it together with data from other visitors. The device then lists products suitable for a visitor with an annual income of 4 million yen who is emotionally happy, and displays this to the user, the store clerk. The store clerk then uses this information to suggest products that the visitor might like.

[0258] Prompt Sentence Examples

[0259] "Write a program that generates the best product listings for a visitor with a specific facial expression and annual income."

[0260] This system can collect data on customers' annual income and emotions, even if they do not purchase products, and use this data to improve the store's product lineup and plan new stores. It can also provide more effective customer service by suggesting products based on the customer's emotional state.

[0261] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0262] Step 1:

[0263] The terminal acquires video data in real time from surveillance cameras installed in the store. The input data is the video footage sent from the surveillance cameras, and the output data is the video footage itself. Specifically, the surveillance cameras capture the movement of customers within the store from the moment they enter. For example, the terminal acquires video of Visitor A, who entered the store at 10:00 AM.

[0264] Step 2:

[0265] The video data captured by the device is sent to the server in real time. The input data is the video captured by the device, and the output data is the video sent to the server. The video data is encrypted to protect privacy. For example, the video data of visitor A is transferred from the camera to the server within a few seconds.

[0266] Step 3:

[0267] The server analyzes the video data received from the device and extracts the visitor's appearance information using an image recognition algorithm. The input data is the transmitted video data, and the output data is the extracted appearance information. Specifically, the server uses a deep learning model to identify the visitor's age, gender, clothing, belongings, etc. For example, it recognizes that Visitor A is a man around 35 years old wearing a business suit.

[0268] Step 4:

[0269] The server estimates the visitor's annual income using a pre-trained machine learning model based on the extracted appearance information. The input data is the extracted appearance information, and the output data is the estimated annual income. Specifically, annual income is predicted based on personal belongings such as luxury watches and designer bags. For example, visitor A is wearing a luxury watch, so his annual income is estimated to be 8 million yen.

[0270] Step 5:

[0271] The server uses an emotion engine to analyze the emotional state of the visitor. The input data is the transmitted video data, and the output data is the analyzed emotional state. Specifically, facial expression analysis technology is used to recognize emotions such as happiness, surprise, and irritation. For example, if visitor A is looking at a product and smiling, the emotion engine will recognize this as "happiness."

[0272] Step 6:

[0273] The server stores the estimated annual income data and emotional state data in a database. The input data is the estimated annual income data and emotional state data, and the output data is the data accumulated in the database. The database stores detailed information about each visitor in an organized manner. For example, data such as "male," "around 35 years old," "8 million yen," and "happy" is stored.

[0274] Step 7:

[0275] The server performs statistical analysis based on the accumulated data. The input data is visitor information stored in the database, and the output data is the results of the statistical analysis. The analysis results reveal the visitor demographics and emotional trends at a particular store, and are useful for improving store operations. For example, it can reveal that visitors with high incomes tend to feel "happy" with certain products.

[0276] Step 8:

[0277] The terminal generates a product list that recommends the most suitable products to the visitor based on the annual income data and emotion data sent from the server. The input data are the annual income data and emotion data sent from the server, and the output data is the generated product list. Specifically, appropriate products are listed according to the visitor's annual income and emotion. For example, a recommended list of luxury bags is displayed to visitor A.

[0278] Step 9:

[0279] The user, a store clerk, checks the display on the terminal and suggests appropriate products to the visitor. The input data is the product list displayed on the terminal, and the output data is the product suggestions to the visitor. The store clerk refers to the product list and provides customer service according to the visitor's emotional state. For example, if visitor A has a happy expression, the store clerk will recommend a luxury bag and offer a special try-on service.

[0280] This series of processing flows allows detailed annual income data and emotional data to be collected from visitors even if they do not purchase any products, enabling improved store operations and more effective customer service support.

[0281] (Application example 2)

[0282] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0283] In current brick-and-mortar store operations, it is difficult to grasp the attributes and emotional state of visitors in real time and immediately make appropriate product recommendations. This is particularly true when targeting a large number of visitors, as it is difficult to provide highly accurate product recommendations and customer service support for each individual customer. Furthermore, there are limited means for making store management decisions based on objective data, which can hinder improvements in sales and customer satisfaction. Furthermore, because customer service does not take into account the emotions of visitors, opportunities to stimulate purchasing motivation are often missed.

[0284] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0285] In this invention, the server includes means for acquiring video data of visitors, means for extracting appearance information of the visitors from the acquired video data, means for estimating the annual income of the visitors based on the extracted appearance information, means for storing the estimated annual income data and emotion data in a database, means for analyzing the accumulated annual income data and emotion data to support decision-making related to store management, means for recognizing the emotional state of the visitors from the video data, and means for transmitting data for display on a smart device. This enables highly accurate product recommendations and customer service support based on the annual income and emotion data of the visitors. Furthermore, store management decisions can be made based on objective data, which can result in expected improvements in sales and customer satisfaction.

[0286] A "customer" is someone who visits a physical store to receive a particular product or service.

[0287] "Video data" refers to video signals and image information obtained from surveillance cameras, smart devices, etc.

[0288] "Appearance information" refers to information such as a visitor's facial and physical features, age, gender, clothing, belongings, and type of shoes.

[0289] "Annual income" refers to the total income earned by visitors in a year.

[0290] "Emotional data" refers to information about the emotional state of a visitor as recognized from their facial expressions, actions, etc.

[0291] A "database" is a storage location for structured information that systematically stores collected information and makes it easy to search and analyze.

[0292] "Smart devices" refer to electronic devices such as mobile devices, smart glasses, and tablets that have internet connectivity and advanced processing capabilities.

[0293] "Store operations" refers to the daily operations of a store, such as store management, sales activities, customer service, and product management.

[0294] Decision making refers to choosing the best course of action or strategy to achieve a particular goal.

[0295] This invention provides a system that extracts appearance information and emotional data from video data of visitors in real time, estimates their annual income based on the extracted data, and is useful for store management and customer service support.

[0296] The system consists of the following modules:

[0297] 1. Video data collection module (terminal)

[0298] The device acquires video data in real time from surveillance cameras installed in the store. This video data records how customers enter the store and move around inside the store. For example, video of a customer entering the store at 10:00 AM is acquired.

[0299] 2. Video analysis module (server)

[0300] The server receives the video data sent from the device and uses an image recognition algorithm to extract information about the visitor's face and body appearance. This information includes the visitor's age, gender, clothing, belongings, and type of shoes. For example, the captured video may identify the visitor as a man around 35 years old and wearing a business suit.

[0301] 3. Annual income estimation module (server)

[0302] The server uses a machine learning model to estimate the visitor's annual income based on the extracted appearance information. This machine learning model is pre-trained and has learned the correlation between appearance information and annual income. For example, a visitor wearing a luxury watch is estimated to have an annual income of 8 million yen.

[0303] 4. Emotion engine (server)

[0304] The server is equipped with an emotion engine that recognizes visitors' emotions from video data. This emotion engine identifies the visitor's emotional state (e.g., happiness, surprise, irritation, etc.) through facial expression analysis. For example, if a visitor smiles while looking at a product, the emotion engine recognizes this as "happiness."

[0305] 5. Database storage module (server)

[0306] The server stores the estimated annual income data and emotion data in a database. This database systematically stores annual income data, emotion data, and other appearance information for each visitor. For example, data such as "male," "around 35 years old," "8 million yen," and "happiness" might be stored.

[0307] 6. Data analysis module (server)

[0308] The server statistically analyzes the accumulated income data and emotional data to determine which income brackets are most frequent visitors to a particular store and what emotional state they are in when looking at products. The results of this analysis are used to improve the store's product lineup and develop new store plans. For example, it may be possible to determine that a particular store is visited by many high-income earners, or that visitors feel "happy" when looking at a particular product.

[0309] 7. Product recommendation module (terminal)

[0310] The device generates a product list that recommends the most suitable products to the visitor based on the annual income data and emotional data sent from the server. Expensive products are listed for visitors with high annual incomes, while affordable products are listed for visitors with low annual incomes, and the device further recommends the most suitable products according to the visitor's emotional state. For example, if a visitor with an annual income of 8 million yen has a happy expression, a list of recommended luxury bags will be displayed first.

[0311] 8. Customer Service Support Module (User)

[0312] The user, a store clerk, checks the display on the device and suggests appropriate products to the customer. This creates an environment where customers can easily make a purchase. For example, if a customer smiles while looking at a luxury bag, the clerk can suggest a luxury bag and encourage them to try on the luxurious bag.

[0313] Specific examples

[0314] Suppose a woman in her 30s visits a store one afternoon. The device acquires video data from the surveillance camera and sends it to the server. The server analyzes the video and extracts information about the woman's appearance, such as her age, gender, clothing, and belongings. It also uses an emotion engine to recognize emotions such as "happiness" from the woman's facial expression. Based on this information, a machine learning model estimates her annual income, calculating it as 4 million yen, for example. The server stores this information in a database and analyzes it together with data from other visitors. The device then lists products suitable for a visitor with an annual income of 4 million yen who is emotionally happy, and displays this to the user, who is also a store clerk. The store clerk then uses this information to suggest products to the visitor.

[0315] Prompt Sentence Examples

[0316] We would like to implement a system that estimates annual income and recommends products based on visitors' appearance and emotional data.

[0317] Write a program to add a feature that displays real-time annual income estimates and emotions through an application installed on smart glasses when a customer enters a store.

[0318] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0319] Step 1:

[0320] Acquiring video data (device)

[0321] The terminal acquires video data in real time from surveillance cameras installed in the store. This video data records how customers enter the store and move around the store. For example, video of a customer entering the store at 10:00 AM is acquired. This video data becomes the input for the system.

[0322] Step 2:

[0323] Video data transmission (terminal)

[0324] The device transmits the acquired video data to the server. This data transmission process is performed in real time, and the video data arrives at the server without delay. The transmitted video data becomes the input for the next processing step.

[0325] Step 3:

[0326] Extraction of appearance information (server)

[0327] The server uses an image recognition algorithm to extract information about the visitor's face and body appearance based on the received video data. Specifically, it performs facial recognition on the video data and extracts facial features. The extracted appearance information includes the visitor's age, gender, clothing, belongings, and type of shoes. For example, it may recognize that the visitor is a man around 35 years old and wearing a business suit. This appearance information becomes the input for the next processing step.

[0328] Step 4:

[0329] Estimated annual income (server)

[0330] The server uses a pre-trained machine learning model to estimate the visitor's annual income based on the extracted appearance information. For example, a visitor wearing a luxury watch might be estimated to have an annual income of 8 million yen. This annual income data becomes the input for the next processing step.

[0331] Step 5:

[0332] Emotion Recognition (Server)

[0333] The server is equipped with an emotion engine that recognizes the emotions of visitors from video data. This emotion engine identifies the emotional state of the visitor (e.g., happiness, surprise, irritation, etc.) through facial expression analysis. For example, if a visitor smiles while looking at a product, the emotion engine recognizes this as "happiness." This emotion data becomes the input for the next processing step.

[0334] Step 6:

[0335] Data storage (server)

[0336] The server stores the estimated annual income data and emotion data in a database. This database systematically stores annual income data, emotion data, and other appearance information for each visitor. For example, data such as "male," "around 35 years old," "8 million yen," and "happy" is stored. This stored data becomes the input for the next processing step.

[0337] Step 7:

[0338] Data analysis (server)

[0339] The server statistically analyzes the accumulated income data and emotional data to analyze which income brackets are most frequent visitors to a particular store and what emotional state they are in when looking at products. The results of this analysis may reveal, for example, that a particular store is frequented by many high-income earners, or that visitors feel "happy" when looking at a particular product. These analysis results become the input for the next processing step.

[0340] Step 8:

[0341] Product recommendation generation (device)

[0342] The device generates a product list that recommends the most suitable products to the visitor based on the annual income data and emotional data sent from the server. Expensive products are listed for visitors with high annual incomes, and affordable products for visitors with low annual incomes, and the device further recommends the most suitable products according to the visitor's emotional state. For example, if a visitor with an annual income of 8 million yen has a happy expression, a list of recommended luxury bags will be displayed preferentially. This product list becomes the input for the next processing step.

[0343] Step 9:

[0344] Customer service support (user)

[0345] The user, a store clerk, checks the display on the terminal and suggests appropriate products to the customer. For example, if a customer smiles while looking at a luxury bag, the store clerk suggests a luxury bag and recommends that the customer try on a luxurious bag.

[0346] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0347] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0348] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0349] [Second embodiment]

[0350] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0351] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0352] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0353] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0354] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0355] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0356] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0357] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0358] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0359] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0360] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0361] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0362] This invention is a system for efficiently collecting annual income data of customers and utilizing it for decision-making in store management. This system is realized by analyzing video data acquired from video devices such as surveillance cameras, extracting information about the appearance of customers, and estimating their annual income.

[0363] System Overview

[0364] The system mainly consists of the following modules:

[0365] 1. Video data collection module (terminal)

[0366] 2. Video analysis module (server)

[0367] 3. Annual income estimation module (server)

[0368] 4. Database storage module (server)

[0369] 5. Data analysis module (server)

[0370] 6. Product recommendation module (terminal)

[0371] 7. Customer Service Support Module (User)

[0372] Program processing

[0373] Video data collection module (terminal)

[0374] The device acquires video data in real time from surveillance cameras installed in the store. This video data records how customers move around the store after entering. For example, video of a customer entering the store at 10:00 a.m. is acquired.

[0375] Video analysis module (server)

[0376] The server receives the video data sent from the device and uses an image recognition algorithm to extract information about the visitor's appearance, including their age, gender, clothing, belongings, and type of shoes. For example, the captured video may identify the visitor as a man around 35 years old wearing a business suit.

[0377] Annual income estimation module (server)

[0378] The server uses a machine learning model to estimate the visitor's annual income based on the extracted appearance information. This machine learning model is pre-trained and has learned the correlation between appearance information and annual income. For example, a visitor wearing a luxury watch is estimated to have an annual income of 8 million yen.

[0379] Database storage module (server)

[0380] The server stores the estimated income data in a database, which systematically stores income data and other appearance information for each visitor, for example, the income data of visitors who visited at a specific date and time.

[0381] Data analysis module (server)

[0382] The server statistically analyzes the accumulated annual income data to determine what income brackets are attracting customers to a particular store and what products are being purchased. The results of this analysis are used to improve the store's product lineup and formulate new store opening plans. For example, it can be determined that a particular store is frequented by customers with an annual income of 8 million yen or more.

[0383] Product recommendation module (terminal)

[0384] The device recommends products suitable for visitors based on the annual income data sent from the server. It lists expensive products for visitors with high annual incomes and affordable products for visitors with low annual incomes. For example, a visitor with an annual income of 5 million yen will be shown a list of mid-priced products.

[0385] Customer service support module (user)

[0386] The user, a store clerk, checks the display on the device and suggests appropriate products to the customer. This creates an environment where customers can easily make purchases. For example, a store clerk suggests a luxury bag to a customer wearing an expensive suit.

[0387] Specific examples

[0388] Suppose a woman in her 30s visits a store one afternoon. The device acquires video data from the surveillance camera and sends it to the server. The server analyzes the video and extracts information about the woman's appearance, such as her age, gender, clothing, and belongings. Based on this information, a machine learning model estimates her annual income, calculating it as 4 million yen, for example. The server stores this information in a database and analyzes it together with data from other visitors. The device then lists products suitable for a visitor with an annual income of 4 million yen and displays it to the user, who is also a store clerk. The clerk uses this information to suggest products to the visitor.

[0389] This system allows annual income data to be collected even if visitors do not purchase products, and can be used to make decisions about store management.

[0390] The processing flow will be explained below.

[0391] Step 1:

[0392] The device captures real-time video data from the store's surveillance cameras, recording customers entering and moving around the store.

[0393] Step 2:

[0394] The device temporarily stores the acquired video data and periodically transmits it to the server, for example, transmitting frame data every second.

[0395] Step 3:

[0396] The server receives the video data sent from the terminal, and the received video data is input to the image analysis module.

[0397] Step 4:

[0398] The server uses video analysis algorithms to extract information about the visitor's facial and physical appearance from the video data, including facial recognition, gender estimation, and age estimation. For example, facial recognition technology is used to identify the visitor's face and estimate their gender as "male" and their age as "around 35 years old."

[0399] Step 5:

[0400] The server extracts further appearance information such as the visitor's clothing, belongings, and type of shoes, including attributes such as color, brand, and fashion style. For example, it may be recognized that the visitor is wearing a business suit and a luxury watch.

[0401] Step 6:

[0402] The server inputs the extracted appearance information into a machine learning model. The machine learning model learns the correlation between appearance information and annual income based on past data, and estimates the visitor's annual income. For example, if a visitor is wearing a business suit and a luxury watch, it can estimate that the visitor's annual income is 8 million yen.

[0403] Step 7:

[0404] The server stores the estimated annual income data in a database. The stored data includes the visitor's age, gender, appearance information, and estimated annual income. For example, data such as "male," "around 35 years old," and "8 million yen" might be stored.

[0405] Step 8:

[0406] The server statistically analyzes the income data stored in the database to determine which income brackets are most frequent visitors to a particular store and what products are being purchased. For example, it may analyze whether a particular store is frequented by a large number of people in the high income bracket.

[0407] Step 9:

[0408] The device generates a product list that recommends the most suitable products to each visitor based on the estimated annual income data. For example, it would recommend luxury bags to a visitor with an annual income of 8 million yen.

[0409] Step 10:

[0410] The terminal displays the generated product list to the user, who is also a store clerk. The store clerk checks the displayed product list and suggests suitable products to the customer. For example, they may explain where to find luxury bags and recommend trying them on.

[0411] This allows the system to collect annual income data even if visitors do not purchase products, and use it to improve the store's product lineup and plan new stores.It can also increase purchasing motivation by suggesting products that are suitable for visitors.

[0412] Example 1

[0413] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0414] In current store operations, it is difficult to obtain data on the annual income of customers, limiting the effectiveness of marketing strategies and product recommendations. It has also been difficult to establish a system that estimates annual income based on information about customers' appearance and then uses that data to recommend specific products and make operational decisions.

[0415] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0416] In this invention, the server includes means for collecting video data of visitors from a video capture device, means for compressing the collected video data and transmitting it to a data server, means for analyzing appearance information of visitors from the video data in the data server, means for estimating the annual income of visitors from the appearance information using a machine learning model, means for storing the estimated annual income data in a database, means for statistically analyzing the stored annual income data and utilizing it in store management, and means for recommending products suitable for visitors based on the analysis results. This makes it possible to efficiently collect annual income data of visitors, improve the accuracy of marketing strategies and product recommendations, and support decision-making in store management.

[0417] The "image capture device" is a device for collecting image data of visitors.

[0418] The "means for compressing collected video data and transmitting it to a data server" refers to a means for effectively compressing collected video data and transmitting it to a data server using a secure communication method.

[0419] The "means for analyzing visitor appearance information from video data using a data server" refers to a means for analyzing video data and extracting visitor appearance information (age, gender, clothing, belongings, etc.).

[0420] The "means for estimating the annual income of a visitor from appearance information using a machine learning model" is a means for estimating the annual income of a visitor based on acquired appearance information using machine learning technology.

[0421] "Means for storing in a database" means a means for systematically storing all relevant information, including estimated annual income data.

[0422] "Means of statistically analyzing accumulated annual income data and utilizing it in store management" refers to a means of conducting statistical analysis based on accumulated annual income data and using the results to help with store management strategies and decision-making.

[0423] The "means for recommending products suitable for visitors based on the analysis results" refers to a means for selecting and recommending the most suitable products to each visitor using the results of statistical analysis.

[0424] The present invention provides a system for efficiently collecting annual income data of customers and utilizing it for decision-making in store management. The system includes an image capture device, a data server, a machine learning model, a database, a statistical analysis means, and a product recommendation means.

[0425] First, the terminal collects real-time video data of visitors using a video capture device installed in the store. The collected video data is compressed using video compression formats such as H.264 or H.265 and sent to a data server via HTTPS. The data server analyzes the received video data and extracts information about the visitor's appearance using image recognition algorithms such as OpenCV and TensorFlow. This information includes age, gender, clothing, and belongings.

[0426] The server then uses a machine learning model to estimate the visitor's annual income from the extracted appearance information. This machine learning model has been trained in advance on a large dataset and has learned the correlation between appearance information and annual income. For example, it can estimate the annual income of a visitor wearing a luxury watch to be 8 million yen.

[0427] The estimated annual income data is stored in a database on the server. This database systematically stores information such as the age, gender, and annual income of each visitor. The server analyzes the accumulated data using statistical analysis libraries such as Pandas, providing useful insights into store operations.

[0428] Furthermore, the server generates a list of recommended products suitable for the visitor based on the analysis results and sends this information to the terminal. The terminal displays the received product recommendation list, and the user (store clerk) suggests appropriate products to the visitor based on this list. Specifically, the server displays a list of expensive products for visitors with high annual incomes and affordable products for visitors with low annual incomes to the user.

[0429] Specific examples

[0430] Suppose a woman in her 30s visits a store one afternoon. The device acquires video data from the surveillance camera and sends it to the server. The server analyzes the video and extracts information about the woman's appearance, such as her age, gender, clothing, and belongings. Based on this information, a machine learning model estimates her annual income, calculating it as 4 million yen, for example. The server stores this information in a database and analyzes it together with data from other visitors. The device then lists products suitable for a visitor with an annual income of 4 million yen and displays it to the user, who is also a store clerk. The clerk uses this information to suggest products to the visitor.

[0431] Prompt Sentence Examples

[0432] Here are some example prompts to explain the system's operations to the generative AI model:

[0433] "Surveillance cameras installed in stores capture footage of customers, and the video data is analyzed to extract information about the customer's appearance (age, gender, clothing, belongings, etc.). Next, a machine learning model is used to estimate the customer's annual income from this appearance information and the information is stored in a database. Based on the stored data, appropriate products are recommended to the customer, and finally, a store associate suggests appropriate products to the customer. Please explain this process step by step."

[0434] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0435] Step 1:

[0436] The terminal collects video data of visitors in real time from a video capture device installed in the store. This input data records how customers move around the store after entering. Specifically, the surveillance camera captures video at about 60 frames per second and stores it on the terminal.

[0437] Step 2:

[0438] The terminal compresses the collected video data using video compression methods such as H.264 or H.265 and sends it to the data server using the HTTPS protocol. This reduces the amount of data traffic and ensures secure transmission. Specifically, the video data is compressed using batch processing and uploaded to the server at regular intervals.

[0439] Step 3:

[0440] The server receives the video data sent from the device and analyzes the visitor's appearance using image recognition algorithms such as OpenCV and TensorFlow. The input data is video frames, and the output extracts appearance information such as the visitor's age, gender, clothing, and belongings. Specifically, a deep learning model for video analysis processes the image data for each frame and extracts specific features.

[0441] Step 4:

[0442] The server uses a machine learning model (e.g., a sparse ridge regression model) to estimate the visitor's annual income from the appearance information. This model is trained in advance using a training dataset. Based on the appearance information extracted as input, the visitor's annual income is estimated as output. Specifically, the model performs a regression analysis using the appearance information variables to calculate an estimated annual income.

[0443] Step 5:

[0444] The server accumulates the estimated annual income data in a database. The input data is the estimated annual income data, and the output is saved in the database. Specifically, the server uses an SQL query to register the annual income data in the database, linking it with the visitor's ID.

[0445] Step 6:

[0446] The server statistically analyzes the accumulated annual income data. The input data is the annual income data stored in the database, and the output provides analysis reports and data trends. Specifically, it converts the data into a data frame using the Pandas library and calculates various statistics (such as the mean, median, and standard deviation).

[0447] Step 7:

[0448] The server generates a list of recommended products suitable for the visitor based on the analysis results. The input data is annual income data and the analysis results, and the output is a list of recommended products. Specifically, the server filters products from the product database according to annual income category and generates the list.

[0449] Step 8:

[0450] The terminal receives the product recommendation list sent from the server and displays it to the user, who is a store clerk. The input data is the product recommendation list, and the output is displayed on the terminal's display. Specifically, the terminal parses the received data and updates the interface displayed on the screen.

[0451] Step 9:

[0452] The user, a store clerk, refers to the product recommendation list displayed on the terminal and suggests appropriate products to the visitor. The input data is the information displayed on the terminal, and the output is specific product suggestions to the visitor. In concrete terms, the store clerk explains the features and benefits of the products to the visitor based on the displayed list, encouraging them to make a purchase.

[0453] (Application example 1)

[0454] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0455] In store operations, efficiently collecting annual income data of customers and using that data to recommend products and provide customer service support contributes to improving store profits and customer satisfaction. However, conventional methods have made it difficult to efficiently collect such data before customers make a purchase, and there has been a lack of concrete means to utilize the collected data in store management decision-making. Furthermore, there has been no system that can make product recommendations tailored to customers in real time.

[0456] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0457] In this invention, the server includes means for acquiring video data of visitors, means for extracting appearance information of the visitors from the acquired video data, means for estimating the annual income of the visitors based on the extracted appearance information, means for storing the estimated annual income data in a database, means for recommending products suitable for the visitors based on the stored annual income data, means for displaying information about the recommended products on a display device and supporting store staff in making suggestions to the visitors, and means for statistically analyzing the annual income data stored in the database and analyzing specific income brackets. This makes it possible to collect annual income data even if the visitors do not purchase products, and to utilize the data to recommend products and provide customer service support in real time.

[0458] "Visitor" refers to an individual who visits a store with the intent of using its services or products.

[0459] "Video data" refers to image and video information acquired from video devices such as surveillance cameras within a store.

[0460] "Appearance information" refers to visual characteristics of visitors, such as their age, gender, clothing, and belongings, extracted from video data.

[0461] "Annual income" refers to the estimated annual income of the visitor.

[0462] "Estimation" refers to predicting a visitor's annual income using a machine learning model based on acquired appearance information, etc.

[0463] "Database" refers to a system for storing and managing accumulated annual income data and other information.

[0464] "Product recommendation" refers to presenting the most suitable products to visitors based on accumulated annual income data.

[0465] "Display device" refers to a device such as a smartphone or tablet that displays product recommendation information and other notifications.

[0466] "Store staff" refers to employees who support sales activities in stores.

[0467] "Customer service support" refers to assistance provided to store staff to efficiently recommend products and provide services to customers.

[0468] "Statistical analysis" refers to a method for analyzing accumulated data and deriving specific patterns or trends.

[0469] This invention is a system for efficiently collecting annual income data of customers and using it to make decisions about store management, and includes the following components. Specifically, we will explain in detail the roles of the server and terminals.

[0470] Video data collection module (terminal)

[0471] The device captures real-time video data from high-resolution IP cameras installed in the store, recording customers as they enter and move around the store. The video data is then sent to a server via Wi-Fi.

[0472] Video analysis module (server)

[0473] The server receives the video data sent from the device and uses an image recognition algorithm to extract information about the visitor's appearance, including their age, gender, clothing, belongings, etc. Specifically, the server performs image analysis using OpenCV, TensorFlow, and the Google Cloud Vision API.

[0474] Annual income estimation module (server)

[0475] The server uses a machine learning model to estimate the visitor's annual income based on the extracted appearance information. This machine learning model is pre-trained and has learned the correlation between appearance information and annual income. Specific technologies used are scikit-learn and Python.

[0476] Database storage module (server)

[0477] The server stores the estimated annual income data in a database, which stores the annual income data and other appearance information of each visitor in an organized manner. The database is managed using MySQL or SQLite.

[0478] Data analysis module (server)

[0479] The server statistically analyzes the accumulated income data to determine what income brackets are attracting customers to specific stores and what products are being purchased. The analysis is performed using R and Python (Pandas, NumPy).

[0480] Product recommendation module (terminal)

[0481] The device recommends products suitable for visitors based on the annual income data sent from the server. It lists expensive products for visitors with high annual incomes and affordable products for visitors with low annual incomes. This is done using cloud-based data communication using Firebase.

[0482] Customer service support module (user)

[0483] Store staff, who are the users, can use their smartphones or tablets to check the displayed product recommendation information and help suggest appropriate products to customers. The built-in notification function and user interface enable efficient customer service.

[0484] Specific examples

[0485] For example, suppose a woman in her 30s visits a store one afternoon. Video data is acquired from a surveillance camera and sent to a server. The server analyzes the video and extracts information about the woman's appearance, such as her age, gender, clothing, and belongings. Based on this information, a machine learning model estimates her annual income, calculating it as, say, 4 million yen. The server stores this information in a database and analyzes it together with data from other visitors. The device then lists products suitable for a visitor with an annual income of 4 million yen and displays it to the store clerk. The clerk then uses this information to suggest products to the visitor.

[0486] Example prompts for generative AI models

[0487] "Estimate the annual income of women in their 30s and recommend products": Build a system that estimates the annual income of visitors from their appearance information (age, gender, clothing, belongings) and recommends appropriate products based on that income. Using the movie example, when a woman in her 30s enters a store, the system analyzes her age and clothing, estimates her annual income to be 4 million yen, and displays mid-priced products on a smartphone app. The store clerk uses this information to suggest products that suit her.

[0488] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0489] Step 1:

[0490] The terminal acquires video data in real time from high-resolution IP cameras installed in the store. Specifically, it captures images and videos of visitors through the camera lens and transmits the video data to a server via Wi-Fi connection. The input is the video data from the camera, and the output is the video data transmitted to the server via Wi-Fi.

[0491] Step 2:

[0492] The server receives the video data sent from the device. It uses an image recognition algorithm to extract the visitor's appearance information from the received video data. Specifically, it uses OpenCV, TensorFlow, and Google Cloud Vision API to analyze and extract characteristic information such as the visitor's age, gender, clothing, and belongings. The input is the video data, and the output is the extracted appearance information.

[0493] Step 3:

[0494] The server uses a machine learning model to estimate the visitor's annual income based on the extracted appearance information. Specifically, a pre-trained model using scikit-learn and Python is used to evaluate the correlation between appearance information and annual income and predict annual income. The input is appearance information, and the output is estimated annual income data.

[0495] Step 4:

[0496] The server stores the estimated annual income data and appearance information in a database. This database stores the annual income data and appearance information for each visitor in an organized manner. Specifically, data is managed efficiently using MySQL and SQLite. The input is annual income data and appearance information, and the output is data storage in the database.

[0497] Step 5:

[0498] The server statistically analyzes the income data stored in the database and analyzes the trends of visitors in specific income brackets. Specifically, it performs data analysis using R and Python (Pandas, NumPy). The input is the income data from the database, and the output is the analysis results.

[0499] Step 6:

[0500] The device recommends suitable products to visitors based on the analysis results sent from the server. Specifically, Firebase is used for cloud-based data communication and a list of recommended products is displayed on the smartphone or tablet. The input is the analysis results, and the output is the display data of the recommended products.

[0501] Step 7:

[0502] Store staff, who are users, check the product recommendation information displayed on their smartphones or tablets and suggest suitable products to visitors. Specifically, they check product information through the built-in notification function and user interface and provide appropriate customer service. The input is the display data of recommended products, and the output is the product suggestions to visitors.

[0503] By following the above steps, annual income data can be collected even if the visitor does not purchase a product, and this data can be used to recommend products and provide customer service support in real time.

[0504] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0505] This invention is a system for efficiently collecting annual income data and emotional data of customers and using it to help with store management decision-making and customer service. This system is realized by analyzing video data acquired from video devices such as surveillance cameras, extracting information on the appearance and emotional information of customers, and estimating their annual income.

[0506] System Overview

[0507] The system mainly consists of the following modules:

[0508] 1. Video data collection module (terminal)

[0509] 2. Video analysis module (server)

[0510] 3. Annual income estimation module (server)

[0511] 4. Emotion engine (server)

[0512] 5. Database storage module (server)

[0513] 6. Data analysis module (server)

[0514] 7. Product recommendation module (terminal)

[0515] 8. Customer Service Support Module (User)

[0516] Program processing

[0517] Video data collection module (terminal)

[0518] The device acquires video data in real time from surveillance cameras installed in the store. This video data records how customers move around the store after entering. For example, video of a customer entering the store at 10:00 a.m. is acquired.

[0519] Video analysis module (server)

[0520] The server receives the video data sent from the device and uses an image recognition algorithm to extract information about the visitor's face and body appearance. This information includes the visitor's age, gender, clothing, belongings, and type of shoes. For example, the captured video may identify the visitor as a man around 35 years old and wearing a business suit.

[0521] Annual income estimation module (server)

[0522] The server uses a machine learning model to estimate the visitor's annual income based on the extracted appearance information. This machine learning model is pre-trained and has learned the correlation between appearance information and annual income. For example, a visitor wearing a luxury watch is estimated to have an annual income of 8 million yen.

[0523] Emotion engine (server)

[0524] The server is equipped with an emotion engine that recognizes visitors' emotions from video data. This emotion engine identifies the visitor's emotional state (e.g., happiness, surprise, irritation, etc.) through facial expression analysis. For example, if a visitor smiles while looking at a product, the emotion engine recognizes this as "happiness."

[0525] Database storage module (server)

[0526] The server stores the estimated annual income data and emotion data in a database. This database systematically stores annual income data, emotion data, and other appearance information for each visitor. For example, data such as "male," "around 35 years old," "8 million yen," and "happiness" might be stored.

[0527] Data analysis module (server)

[0528] The server statistically analyzes the accumulated income data and emotional data to determine which income brackets are most frequent visitors to a particular store and what emotional state they are in when looking at products. The results of this analysis are used to improve the store's product lineup and develop new store plans. For example, it may be possible to determine that a particular store is visited by many high-income earners, or that visitors feel "happy" when looking at a particular product.

[0529] Product recommendation module (terminal)

[0530] The device generates a product list that recommends the most suitable products to the visitor based on the annual income data and emotional data sent from the server. Expensive products are listed for visitors with high annual incomes, while affordable products are listed for visitors with low annual incomes, and the device further recommends the most suitable products according to the visitor's emotional state. For example, if a visitor with an annual income of 8 million yen has a happy expression, a list of recommended luxury bags will be displayed first.

[0531] Customer service support module (user)

[0532] The user, a store clerk, checks the display on the device and suggests appropriate products to the customer. This creates an environment where customers can easily make a purchase. For example, if a customer smiles while looking at a luxury bag, the clerk can suggest a luxury bag and encourage them to try on the luxurious bag.

[0533] Specific examples

[0534] Suppose a woman in her 30s visits a store one afternoon. The device acquires video data from the surveillance camera and sends it to the server. The server analyzes the video and extracts information about the woman's appearance, such as her age, gender, clothing, and belongings. It also uses an emotion engine to recognize emotions such as "happiness" from the woman's facial expression. Based on this information, a machine learning model estimates her annual income, calculating it as 4 million yen, for example. The server stores this information in a database and analyzes it together with data from other visitors. The device then lists products suitable for a visitor with an annual income of 4 million yen who is emotionally happy, and displays this to the user, who is also a store clerk. The store clerk then uses this information to suggest products to the visitor.

[0535] This system can collect data on customers' annual income and emotions, even if they do not purchase products, and use this data to improve the store's product lineup and plan new stores. It can also provide more effective customer service by suggesting products based on the customer's emotional state.

[0536] The processing flow will be explained below.

[0537] Step 1:

[0538] The device captures real-time video data from the store's surveillance cameras, recording customers entering and moving around the store.

[0539] Step 2:

[0540] The device temporarily stores the acquired video data and periodically transmits it to the server, for example, transmitting frame data every second.

[0541] Step 3:

[0542] The server receives the video data sent from the terminal, and the received video data is input to the image analysis module.

[0543] Step 4:

[0544] The server uses video analysis algorithms to extract information about the visitor's facial and physical appearance from the video data, including facial recognition, gender estimation, and age estimation. For example, facial recognition technology is used to identify the visitor's face and estimate their gender as "male" and their age as "around 35 years old."

[0545] Step 5:

[0546] The server extracts further appearance information such as the visitor's clothing, belongings, and type of shoes, including attributes such as color, brand, and fashion style. For example, it may be recognized that the visitor is wearing a business suit and a luxury watch.

[0547] Step 6:

[0548] The server inputs the extracted appearance information into a machine learning model. The machine learning model learns the correlation between appearance information and annual income based on past data, and estimates the visitor's annual income. For example, if a visitor is wearing a business suit and a luxury watch, it can estimate that the visitor's annual income is 8 million yen.

[0549] Step 7:

[0550] The server runs an emotion engine that recognizes the visitor's emotions from the video data. The emotion engine identifies the visitor's emotional state (e.g., happiness, surprise, irritation, etc.) through facial expression analysis. For example, if a visitor is looking at a product and smiling, the emotion engine recognizes the visitor as "happy."

[0551] Step 8:

[0552] The server stores the estimated annual income data and emotional data in a database. The stored data includes the visitor's age, gender, appearance information, annual income, and emotional state. For example, the data stored might be "male," "around 35 years old," "8 million yen," and "happy."

[0553] Step 9:

[0554] The server statistically analyzes the income data and emotional data stored in the database to determine which income brackets are most frequent visitors to a particular store and what emotional state they are in when looking at products. The results of this analysis are used to improve the store's product lineup and develop new store plans. For example, it may be possible to determine that a particular store is visited by many high-income earners, or that visitors feel "happy" when looking at a particular product.

[0555] Step 10:

[0556] The device generates a product list that recommends the most suitable products to the visitor based on the annual income data and emotional data sent from the server. Expensive products are listed for visitors with high annual incomes, while affordable products are listed for visitors with low annual incomes, and the device further recommends the most suitable products according to the visitor's emotional state. For example, if a visitor with an annual income of 8 million yen has a happy expression, a list of recommended luxury bags will be displayed first.

[0557] Step 11:

[0558] The terminal displays the generated product list to the user, who is also a store clerk. The store clerk checks the displayed product list and suggests appropriate products to the visitor. For example, if a visitor smiles while looking at a luxury bag, the store clerk will suggest a luxury bag and recommend that the visitor try on a luxurious bag.

[0559] Example 2

[0560] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0561] In conventional store operations, the collection of information about customers was limited, making it difficult to grasp details such as the annual income and emotional state of customers. As a result, effective decision-making and product recommendations suited to customers were not fully implemented in store operations. In addition, it was difficult to provide individual support to stimulate customers' purchasing desire, resulting in a situation where sales could not be expected to increase.

[0562] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0563] In this invention, the server includes means for extracting appearance information of visitors from video data, means for estimating annual income based on the extracted appearance information, and means for analyzing the emotional state of visitors. This makes it possible to grasp annual income and emotions based on detailed information about visitors, support decision-making for store management, and suggest products suitable for visitors.

[0564] A "terminal" is a device that acquires video data of visitors and transmits it to a server.

[0565] A "server" is a computer system that analyzes video data, processes data such as appearance information, annual income, and emotional state, and stores and analyzes this data in a database.

[0566] "Video Data" refers to image and video data captured to record the appearance and actions of visitors.

[0567] "Appearance information" is data extracted from a visitor's age, gender, clothing, belongings, type of shoes, and other visual characteristics.

[0568] "Annual income" is the amount of income a visitor is estimated to earn per year.

[0569] "Emotional state" refers to the emotional state analyzed from the visitor's facial expression, and includes happiness, surprise, irritation, etc.

[0570] A "database" is a digital storage device for systematically storing information such as visitors' appearance, annual income, and emotional state.

[0571] "Data analysis" is the process of using data stored in a database to analyze trends and patterns related to customer visitation and store operations.

[0572] A "product list" is a list of products generated to recommend the most suitable products to visitors.

[0573] "User" refers to a store employee or salesperson who uses the system to suggest suitable products to visitors.

[0574] MODE FOR CARRYING OUT THE INVENTION

[0575] This invention is a system that efficiently collects annual income data and emotional data of customers and uses it to help with decision-making in store management and customer service. This system functions in cooperation with three entities: terminals, servers, and users.

[0576] Hardware and software used

[0577] The device includes a surveillance camera installed in the store that captures video data in real time. This video data is used to record how customers move around the store after entering. For example, a video of a customer entering the store at 10:00 AM is captured.

[0578] The server receives the video data sent from the device and uses an image recognition algorithm (such as TensorFlow or OpenCV) to analyze it. The server extracts the visitor's appearance information (age, gender, clothing, belongings, etc.). For example, the server recognizes from the captured video that the visitor is a man around 35 years old wearing a business suit.

[0579] Next, the server uses a machine learning model (for example, a model using scikit-learn or TensorFlow) to estimate the visitor's annual income based on the extracted appearance information. For example, a visitor wearing a luxury watch is estimated to have an annual income of 8 million yen.

[0580] The server is also equipped with an emotion engine that recognizes visitors' emotions from video data. This emotion engine identifies the visitor's emotional state (happiness, surprise, irritation, etc.) through facial expression analysis. For example, if a visitor smiles while looking at a product, the emotion engine will recognize this as "happiness."

[0581] The server stores the estimated annual income data and emotion data in a database. This database stores the annual income data, emotion data, and other appearance information for each visitor in an organized format. For example, data such as "male," "around 35 years old," "8 million yen," and "happy" might be stored.

[0582] The data stored in the database is statistically analyzed by a server. The analysis results provide information such as which income brackets are most frequent visitors to a particular store and the emotional state customers are in when looking at products. These analysis results are used to improve the store's product lineup and formulate new store opening plans. For example, it may be discovered that a particular store is visited by many high-income earners, or that visitors feel "happy" when looking at a particular product.

[0583] Specific examples

[0584] Let's say a woman in her 30s visits a store one afternoon. The device acquires video data from the surveillance camera and sends it to the server. The server analyzes the video and extracts information about the woman's appearance, such as her age, gender, clothing, and belongings. It also uses an emotion engine to recognize emotions such as "happiness" from the woman's facial expression. Based on this information, a machine learning model estimates her annual income, calculating it as 4 million yen, for example. The server stores this information in a database and analyzes it together with data from other visitors. The device then lists products suitable for a visitor with an annual income of 4 million yen who is emotionally happy, and displays this to the user, the store clerk. The store clerk then uses this information to suggest products that the visitor might like.

[0585] Prompt Sentence Examples

[0586] "Write a program that generates the best product listings for a visitor with a specific facial expression and annual income."

[0587] This system can collect data on customers' annual income and emotions, even if they do not purchase products, and use this data to improve the store's product lineup and plan new stores. It can also provide more effective customer service by suggesting products based on the customer's emotional state.

[0588] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0589] Step 1:

[0590] The terminal acquires video data in real time from surveillance cameras installed in the store. The input data is the video footage sent from the surveillance cameras, and the output data is the video footage itself. Specifically, the surveillance cameras capture the movement of customers within the store from the moment they enter. For example, the terminal acquires video of Visitor A, who entered the store at 10:00 AM.

[0591] Step 2:

[0592] The video data captured by the device is sent to the server in real time. The input data is the video captured by the device, and the output data is the video sent to the server. The video data is encrypted to protect privacy. For example, the video data of visitor A is transferred from the camera to the server within a few seconds.

[0593] Step 3:

[0594] The server analyzes the video data received from the device and extracts the visitor's appearance information using an image recognition algorithm. The input data is the transmitted video data, and the output data is the extracted appearance information. Specifically, the server uses a deep learning model to identify the visitor's age, gender, clothing, belongings, etc. For example, it recognizes that Visitor A is a man around 35 years old wearing a business suit.

[0595] Step 4:

[0596] The server estimates the visitor's annual income using a pre-trained machine learning model based on the extracted appearance information. The input data is the extracted appearance information, and the output data is the estimated annual income. Specifically, annual income is predicted based on personal belongings such as luxury watches and designer bags. For example, visitor A is wearing a luxury watch, so his annual income is estimated to be 8 million yen.

[0597] Step 5:

[0598] The server uses an emotion engine to analyze the emotional state of the visitor. The input data is the transmitted video data, and the output data is the analyzed emotional state. Specifically, facial expression analysis technology is used to recognize emotions such as happiness, surprise, and irritation. For example, if visitor A is looking at a product and smiling, the emotion engine will recognize this as "happiness."

[0599] Step 6:

[0600] The server stores the estimated annual income data and emotional state data in a database. The input data is the estimated annual income data and emotional state data, and the output data is the data accumulated in the database. The database stores detailed information about each visitor in an organized manner. For example, data such as "male," "around 35 years old," "8 million yen," and "happy" is stored.

[0601] Step 7:

[0602] The server performs statistical analysis based on the accumulated data. The input data is visitor information stored in the database, and the output data is the results of the statistical analysis. The analysis results reveal the visitor demographics and emotional trends at a particular store, and are useful for improving store operations. For example, it can reveal that visitors with high incomes tend to feel "happy" with certain products.

[0603] Step 8:

[0604] The terminal generates a product list that recommends the most suitable products to the visitor based on the annual income data and emotion data sent from the server. The input data are the annual income data and emotion data sent from the server, and the output data is the generated product list. Specifically, appropriate products are listed according to the visitor's annual income and emotion. For example, a recommended list of luxury bags is displayed to visitor A.

[0605] Step 9:

[0606] The user, a store clerk, checks the display on the terminal and suggests appropriate products to the visitor. The input data is the product list displayed on the terminal, and the output data is the product suggestions to the visitor. The store clerk refers to the product list and provides customer service according to the visitor's emotional state. For example, if visitor A has a happy expression, the store clerk will recommend a luxury bag and offer a special try-on service.

[0607] This series of processing flows allows detailed annual income data and emotional data to be collected from visitors even if they do not purchase any products, enabling improved store operations and more effective customer service support.

[0608] (Application example 2)

[0609] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0610] In current brick-and-mortar store operations, it is difficult to grasp the attributes and emotional state of visitors in real time and immediately make appropriate product recommendations. This is particularly true when targeting a large number of visitors, as it is difficult to provide highly accurate product recommendations and customer service support for each individual customer. Furthermore, there are limited means for making store management decisions based on objective data, which can hinder improvements in sales and customer satisfaction. Furthermore, because customer service does not take into account the emotions of visitors, opportunities to stimulate purchasing motivation are often missed.

[0611] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0612] In this invention, the server includes means for acquiring video data of visitors, means for extracting appearance information of the visitors from the acquired video data, means for estimating the annual income of the visitors based on the extracted appearance information, means for storing the estimated annual income data and emotion data in a database, means for analyzing the accumulated annual income data and emotion data to support decision-making related to store management, means for recognizing the emotional state of the visitors from the video data, and means for transmitting data for display on a smart device. This enables highly accurate product recommendations and customer service support based on the annual income and emotion data of the visitors. Furthermore, store management decisions can be made based on objective data, which can result in expected improvements in sales and customer satisfaction.

[0613] A "customer" is someone who visits a physical store to receive a particular product or service.

[0614] "Video data" refers to video signals and image information obtained from surveillance cameras, smart devices, etc.

[0615] "Appearance information" refers to information such as a visitor's facial and physical features, age, gender, clothing, belongings, and type of shoes.

[0616] "Annual income" refers to the total income earned by visitors in a year.

[0617] "Emotional data" refers to information about the emotional state of a visitor as recognized from their facial expressions, actions, etc.

[0618] A "database" is a storage location for structured information that systematically stores collected information and makes it easy to search and analyze.

[0619] "Smart devices" refer to electronic devices such as mobile devices, smart glasses, and tablets that have internet connectivity and advanced processing capabilities.

[0620] "Store operations" refers to the daily operations of a store, such as store management, sales activities, customer service, and product management.

[0621] Decision making refers to choosing the best course of action or strategy to achieve a particular goal.

[0622] This invention provides a system that extracts appearance information and emotional data from video data of visitors in real time, estimates their annual income based on the extracted data, and is useful for store management and customer service support.

[0623] The system consists of the following modules:

[0624] 1. Video data collection module (terminal)

[0625] The device acquires video data in real time from surveillance cameras installed in the store. This video data records how customers enter the store and move around inside the store. For example, video of a customer entering the store at 10:00 AM is acquired.

[0626] 2. Video analysis module (server)

[0627] The server receives the video data sent from the device and uses an image recognition algorithm to extract information about the visitor's face and body appearance. This information includes the visitor's age, gender, clothing, belongings, and type of shoes. For example, the captured video may identify the visitor as a man around 35 years old and wearing a business suit.

[0628] 3. Annual income estimation module (server)

[0629] The server uses a machine learning model to estimate the visitor's annual income based on the extracted appearance information. This machine learning model is pre-trained and has learned the correlation between appearance information and annual income. For example, a visitor wearing a luxury watch is estimated to have an annual income of 8 million yen.

[0630] 4. Emotion engine (server)

[0631] The server is equipped with an emotion engine that recognizes visitors' emotions from video data. This emotion engine identifies the visitor's emotional state (e.g., happiness, surprise, irritation, etc.) through facial expression analysis. For example, if a visitor smiles while looking at a product, the emotion engine recognizes this as "happiness."

[0632] 5. Database storage module (server)

[0633] The server stores the estimated annual income data and emotion data in a database. This database systematically stores annual income data, emotion data, and other appearance information for each visitor. For example, data such as "male," "around 35 years old," "8 million yen," and "happiness" might be stored.

[0634] 6. Data analysis module (server)

[0635] The server statistically analyzes the accumulated income data and emotional data to determine which income brackets are most frequent visitors to a particular store and what emotional state they are in when looking at products. The results of this analysis are used to improve the store's product lineup and develop new store plans. For example, it may be possible to determine that a particular store is visited by many high-income earners, or that visitors feel "happy" when looking at a particular product.

[0636] 7. Product recommendation module (terminal)

[0637] The device generates a product list that recommends the most suitable products to the visitor based on the annual income data and emotional data sent from the server. Expensive products are listed for visitors with high annual incomes, while affordable products are listed for visitors with low annual incomes, and the device further recommends the most suitable products according to the visitor's emotional state. For example, if a visitor with an annual income of 8 million yen has a happy expression, a list of recommended luxury bags will be displayed first.

[0638] 8. Customer Service Support Module (User)

[0639] The user, a store clerk, checks the display on the device and suggests appropriate products to the customer. This creates an environment where customers can easily make a purchase. For example, if a customer smiles while looking at a luxury bag, the clerk can suggest a luxury bag and encourage them to try on the luxurious bag.

[0640] Specific examples

[0641] Suppose a woman in her 30s visits a store one afternoon. The device acquires video data from the surveillance camera and sends it to the server. The server analyzes the video and extracts information about the woman's appearance, such as her age, gender, clothing, and belongings. It also uses an emotion engine to recognize emotions such as "happiness" from the woman's facial expression. Based on this information, a machine learning model estimates her annual income, calculating it as 4 million yen, for example. The server stores this information in a database and analyzes it together with data from other visitors. The device then lists products suitable for a visitor with an annual income of 4 million yen who is emotionally happy, and displays this to the user, who is also a store clerk. The store clerk then uses this information to suggest products to the visitor.

[0642] Prompt Sentence Examples

[0643] We would like to implement a system that estimates annual income and recommends products based on visitors' appearance and emotional data.

[0644] Write a program to add a feature that displays real-time annual income estimates and emotions through an application installed on smart glasses when a customer enters a store.

[0645] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0646] Step 1:

[0647] Acquiring video data (device)

[0648] The terminal acquires video data in real time from surveillance cameras installed in the store. This video data records how customers enter the store and move around the store. For example, video of a customer entering the store at 10:00 AM is acquired. This video data becomes the input for the system.

[0649] Step 2:

[0650] Video data transmission (terminal)

[0651] The device transmits the acquired video data to the server. This data transmission process is performed in real time, and the video data arrives at the server without delay. The transmitted video data becomes the input for the next processing step.

[0652] Step 3:

[0653] Extraction of appearance information (server)

[0654] The server uses an image recognition algorithm to extract information about the visitor's face and body appearance based on the received video data. Specifically, it performs facial recognition on the video data and extracts facial features. The extracted appearance information includes the visitor's age, gender, clothing, belongings, and type of shoes. For example, it may recognize that the visitor is a man around 35 years old and wearing a business suit. This appearance information becomes the input for the next processing step.

[0655] Step 4:

[0656] Estimated annual income (server)

[0657] The server uses a pre-trained machine learning model to estimate the visitor's annual income based on the extracted appearance information. For example, a visitor wearing a luxury watch might be estimated to have an annual income of 8 million yen. This annual income data becomes the input for the next processing step.

[0658] Step 5:

[0659] Emotion Recognition (Server)

[0660] The server is equipped with an emotion engine that recognizes the emotions of visitors from video data. This emotion engine identifies the emotional state of the visitor (e.g., happiness, surprise, irritation, etc.) through facial expression analysis. For example, if a visitor smiles while looking at a product, the emotion engine recognizes this as "happiness." This emotion data becomes the input for the next processing step.

[0661] Step 6:

[0662] Data storage (server)

[0663] The server stores the estimated annual income data and emotion data in a database. This database systematically stores annual income data, emotion data, and other appearance information for each visitor. For example, data such as "male," "around 35 years old," "8 million yen," and "happy" is stored. This stored data becomes the input for the next processing step.

[0664] Step 7:

[0665] Data analysis (server)

[0666] The server statistically analyzes the accumulated income data and emotional data to analyze which income brackets are most frequent visitors to a particular store and what emotional state they are in when looking at products. The results of this analysis may reveal, for example, that a particular store is frequented by many high-income earners, or that visitors feel "happy" when looking at a particular product. These analysis results become the input for the next processing step.

[0667] Step 8:

[0668] Product recommendation generation (device)

[0669] The device generates a product list that recommends the most suitable products to the visitor based on the annual income data and emotional data sent from the server. Expensive products are listed for visitors with high annual incomes, and affordable products for visitors with low annual incomes, and the device further recommends the most suitable products according to the visitor's emotional state. For example, if a visitor with an annual income of 8 million yen has a happy expression, a list of recommended luxury bags will be displayed preferentially. This product list becomes the input for the next processing step.

[0670] Step 9:

[0671] Customer service support (user)

[0672] The user, a store clerk, checks the display on the terminal and suggests appropriate products to the customer. For example, if a customer smiles while looking at a luxury bag, the store clerk suggests a luxury bag and recommends that the customer try on a luxurious bag.

[0673] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0674] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0675] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0676] [Third embodiment]

[0677] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0678] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0679] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0680] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0681] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0682] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0683] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0684] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0685] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0686] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0687] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0688] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0689] This invention is a system for efficiently collecting annual income data of customers and utilizing it for decision-making in store management. This system is realized by analyzing video data acquired from video devices such as surveillance cameras, extracting information about the appearance of customers, and estimating their annual income.

[0690] System Overview

[0691] The system mainly consists of the following modules:

[0692] 1. Video data collection module (terminal)

[0693] 2. Video analysis module (server)

[0694] 3. Annual income estimation module (server)

[0695] 4. Database storage module (server)

[0696] 5. Data analysis module (server)

[0697] 6. Product recommendation module (terminal)

[0698] 7. Customer Service Support Module (User)

[0699] Program processing

[0700] Video data collection module (terminal)

[0701] The device acquires video data in real time from surveillance cameras installed in the store. This video data records how customers move around the store after entering. For example, video of a customer entering the store at 10:00 a.m. is acquired.

[0702] Video analysis module (server)

[0703] The server receives the video data sent from the device and uses an image recognition algorithm to extract information about the visitor's appearance, including their age, gender, clothing, belongings, and type of shoes. For example, the captured video may identify the visitor as a man around 35 years old wearing a business suit.

[0704] Annual income estimation module (server)

[0705] The server uses a machine learning model to estimate the visitor's annual income based on the extracted appearance information. This machine learning model is pre-trained and has learned the correlation between appearance information and annual income. For example, a visitor wearing a luxury watch is estimated to have an annual income of 8 million yen.

[0706] Database storage module (server)

[0707] The server stores the estimated income data in a database, which systematically stores income data and other appearance information for each visitor, for example, the income data of visitors who visited at a specific date and time.

[0708] Data analysis module (server)

[0709] The server statistically analyzes the accumulated annual income data to determine what income brackets are attracting customers to a particular store and what products are being purchased. The results of this analysis are used to improve the store's product lineup and formulate new store opening plans. For example, it can be determined that a particular store is frequented by customers with an annual income of 8 million yen or more.

[0710] Product recommendation module (terminal)

[0711] The device recommends products suitable for visitors based on the annual income data sent from the server. It lists expensive products for visitors with high annual incomes and affordable products for visitors with low annual incomes. For example, a visitor with an annual income of 5 million yen will be shown a list of mid-priced products.

[0712] Customer service support module (user)

[0713] The user, a store clerk, checks the display on the device and suggests appropriate products to the customer. This creates an environment where customers can easily make purchases. For example, a store clerk suggests a luxury bag to a customer wearing an expensive suit.

[0714] Specific examples

[0715] Suppose a woman in her 30s visits a store one afternoon. The device acquires video data from the surveillance camera and sends it to the server. The server analyzes the video and extracts information about the woman's appearance, such as her age, gender, clothing, and belongings. Based on this information, a machine learning model estimates her annual income, calculating it as 4 million yen, for example. The server stores this information in a database and analyzes it together with data from other visitors. The device then lists products suitable for a visitor with an annual income of 4 million yen and displays it to the user, who is also a store clerk. The clerk uses this information to suggest products to the visitor.

[0716] This system allows annual income data to be collected even if visitors do not purchase products, and can be used to make decisions about store management.

[0717] The processing flow will be explained below.

[0718] Step 1:

[0719] The device captures real-time video data from the store's surveillance cameras, recording customers entering and moving around the store.

[0720] Step 2:

[0721] The device temporarily stores the acquired video data and periodically transmits it to the server, for example, transmitting frame data every second.

[0722] Step 3:

[0723] The server receives the video data sent from the terminal, and the received video data is input to the image analysis module.

[0724] Step 4:

[0725] The server uses video analysis algorithms to extract information about the visitor's facial and physical appearance from the video data, including facial recognition, gender estimation, and age estimation. For example, facial recognition technology is used to identify the visitor's face and estimate their gender as "male" and their age as "around 35 years old."

[0726] Step 5:

[0727] The server extracts further appearance information such as the visitor's clothing, belongings, and type of shoes, including attributes such as color, brand, and fashion style. For example, it may be recognized that the visitor is wearing a business suit and a luxury watch.

[0728] Step 6:

[0729] The server inputs the extracted appearance information into a machine learning model. The machine learning model learns the correlation between appearance information and annual income based on past data, and estimates the visitor's annual income. For example, if a visitor is wearing a business suit and a luxury watch, it can estimate that the visitor's annual income is 8 million yen.

[0730] Step 7:

[0731] The server stores the estimated annual income data in a database. The stored data includes the visitor's age, gender, appearance information, and estimated annual income. For example, data such as "male," "around 35 years old," and "8 million yen" might be stored.

[0732] Step 8:

[0733] The server statistically analyzes the income data stored in the database to determine which income brackets are most frequent visitors to a particular store and what products are being purchased. For example, it may analyze whether a particular store is frequented by a large number of people in the high income bracket.

[0734] Step 9:

[0735] The device generates a product list that recommends the most suitable products to each visitor based on the estimated annual income data. For example, it would recommend luxury bags to a visitor with an annual income of 8 million yen.

[0736] Step 10:

[0737] The terminal displays the generated product list to the user, who is also a store clerk. The store clerk checks the displayed product list and suggests suitable products to the customer. For example, they may explain where to find luxury bags and recommend trying them on.

[0738] This allows the system to collect annual income data even if visitors do not purchase products, and use it to improve the store's product lineup and plan new stores.It can also increase purchasing motivation by suggesting products that are suitable for visitors.

[0739] Example 1

[0740] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0741] In current store operations, it is difficult to obtain data on the annual income of customers, limiting the effectiveness of marketing strategies and product recommendations. It has also been difficult to establish a system that estimates annual income based on information about customers' appearance and then uses that data to recommend specific products and make operational decisions.

[0742] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0743] In this invention, the server includes means for collecting video data of visitors from a video capture device, means for compressing the collected video data and transmitting it to a data server, means for analyzing appearance information of visitors from the video data in the data server, means for estimating the annual income of visitors from the appearance information using a machine learning model, means for storing the estimated annual income data in a database, means for statistically analyzing the stored annual income data and utilizing it in store management, and means for recommending products suitable for visitors based on the analysis results. This makes it possible to efficiently collect annual income data of visitors, improve the accuracy of marketing strategies and product recommendations, and support decision-making in store management.

[0744] The "image capture device" is a device for collecting image data of visitors.

[0745] The "means for compressing collected video data and transmitting it to a data server" refers to a means for effectively compressing collected video data and transmitting it to a data server using a secure communication method.

[0746] The "means for analyzing visitor appearance information from video data using a data server" refers to a means for analyzing video data and extracting visitor appearance information (age, gender, clothing, belongings, etc.).

[0747] The "means for estimating the annual income of a visitor from appearance information using a machine learning model" is a means for estimating the annual income of a visitor based on acquired appearance information using machine learning technology.

[0748] "Means for storing in a database" means a means for systematically storing all relevant information, including estimated annual income data.

[0749] "Means of statistically analyzing accumulated annual income data and utilizing it in store management" refers to a means of conducting statistical analysis based on accumulated annual income data and using the results to help with store management strategies and decision-making.

[0750] The "means for recommending products suitable for visitors based on the analysis results" refers to a means for selecting and recommending the most suitable products to each visitor using the results of statistical analysis.

[0751] The present invention provides a system for efficiently collecting annual income data of customers and utilizing it for decision-making in store management. The system includes an image capture device, a data server, a machine learning model, a database, a statistical analysis means, and a product recommendation means.

[0752] First, the terminal collects real-time video data of visitors using a video capture device installed in the store. The collected video data is compressed using video compression formats such as H.264 or H.265 and sent to a data server via HTTPS. The data server analyzes the received video data and extracts information about the visitor's appearance using image recognition algorithms such as OpenCV and TensorFlow. This information includes age, gender, clothing, and belongings.

[0753] The server then uses a machine learning model to estimate the visitor's annual income from the extracted appearance information. This machine learning model has been trained in advance on a large dataset and has learned the correlation between appearance information and annual income. For example, it can estimate the annual income of a visitor wearing a luxury watch to be 8 million yen.

[0754] The estimated annual income data is stored in a database on the server. This database systematically stores information such as the age, gender, and annual income of each visitor. The server analyzes the accumulated data using statistical analysis libraries such as Pandas, providing useful insights into store operations.

[0755] Furthermore, the server generates a list of recommended products suitable for the visitor based on the analysis results and sends this information to the terminal. The terminal displays the received product recommendation list, and the user (store clerk) suggests appropriate products to the visitor based on this list. Specifically, the server displays a list of expensive products for visitors with high annual incomes and affordable products for visitors with low annual incomes to the user.

[0756] Specific examples

[0757] Suppose a woman in her 30s visits a store one afternoon. The device acquires video data from the surveillance camera and sends it to the server. The server analyzes the video and extracts information about the woman's appearance, such as her age, gender, clothing, and belongings. Based on this information, a machine learning model estimates her annual income, calculating it as 4 million yen, for example. The server stores this information in a database and analyzes it together with data from other visitors. The device then lists products suitable for a visitor with an annual income of 4 million yen and displays it to the user, who is also a store clerk. The clerk uses this information to suggest products to the visitor.

[0758] Prompt Sentence Examples

[0759] Here are some example prompts to explain the system's operations to the generative AI model:

[0760] "Surveillance cameras installed in stores capture footage of customers, and the video data is analyzed to extract information about the customer's appearance (age, gender, clothing, belongings, etc.). Next, a machine learning model is used to estimate the customer's annual income from this appearance information and the information is stored in a database. Based on the stored data, appropriate products are recommended to the customer, and finally, a store associate suggests appropriate products to the customer. Please explain this process step by step."

[0761] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0762] Step 1:

[0763] The terminal collects video data of visitors in real time from a video capture device installed in the store. This input data records how customers move around the store after entering. Specifically, the surveillance camera captures video at about 60 frames per second and stores it on the terminal.

[0764] Step 2:

[0765] The terminal compresses the collected video data using video compression methods such as H.264 or H.265 and sends it to the data server using the HTTPS protocol. This reduces the amount of data traffic and ensures secure transmission. Specifically, the video data is compressed using batch processing and uploaded to the server at regular intervals.

[0766] Step 3:

[0767] The server receives the video data sent from the device and analyzes the visitor's appearance using image recognition algorithms such as OpenCV and TensorFlow. The input data is video frames, and the output extracts appearance information such as the visitor's age, gender, clothing, and belongings. Specifically, a deep learning model for video analysis processes the image data for each frame and extracts specific features.

[0768] Step 4:

[0769] The server uses a machine learning model (e.g., a sparse ridge regression model) to estimate the visitor's annual income from the appearance information. This model is trained in advance using a training dataset. Based on the appearance information extracted as input, the visitor's annual income is estimated as output. Specifically, the model performs a regression analysis using the appearance information variables to calculate an estimated annual income.

[0770] Step 5:

[0771] The server accumulates the estimated annual income data in a database. The input data is the estimated annual income data, and the output is saved in the database. Specifically, the server uses an SQL query to register the annual income data in the database, linking it with the visitor's ID.

[0772] Step 6:

[0773] The server statistically analyzes the accumulated annual income data. The input data is the annual income data stored in the database, and the output provides analysis reports and data trends. Specifically, it converts the data into a data frame using the Pandas library and calculates various statistics (such as the mean, median, and standard deviation).

[0774] Step 7:

[0775] The server generates a list of recommended products suitable for the visitor based on the analysis results. The input data is annual income data and the analysis results, and the output is a list of recommended products. Specifically, the server filters products from the product database according to annual income category and generates the list.

[0776] Step 8:

[0777] The terminal receives the product recommendation list sent from the server and displays it to the user, who is a store clerk. The input data is the product recommendation list, and the output is displayed on the terminal's display. Specifically, the terminal parses the received data and updates the interface displayed on the screen.

[0778] Step 9:

[0779] The user, a store clerk, refers to the product recommendation list displayed on the terminal and suggests appropriate products to the visitor. The input data is the information displayed on the terminal, and the output is specific product suggestions to the visitor. In concrete terms, the store clerk explains the features and benefits of the products to the visitor based on the displayed list, encouraging them to make a purchase.

[0780] (Application example 1)

[0781] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0782] In store operations, efficiently collecting annual income data of customers and using that data to recommend products and provide customer service support contributes to improving store profits and customer satisfaction. However, conventional methods have made it difficult to efficiently collect such data before customers make a purchase, and there has been a lack of concrete means to utilize the collected data in store management decision-making. Furthermore, there has been no system that can make product recommendations tailored to customers in real time.

[0783] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0784] In this invention, the server includes means for acquiring video data of visitors, means for extracting appearance information of the visitors from the acquired video data, means for estimating the annual income of the visitors based on the extracted appearance information, means for storing the estimated annual income data in a database, means for recommending products suitable for the visitors based on the stored annual income data, means for displaying information about the recommended products on a display device and supporting store staff in making suggestions to the visitors, and means for statistically analyzing the annual income data stored in the database and analyzing specific income brackets. This makes it possible to collect annual income data even if the visitors do not purchase products, and to utilize the data to recommend products and provide customer service support in real time.

[0785] "Visitor" refers to an individual who visits a store with the intent of using its services or products.

[0786] "Video data" refers to image and video information acquired from video devices such as surveillance cameras within a store.

[0787] "Appearance information" refers to visual characteristics of visitors, such as their age, gender, clothing, and belongings, extracted from video data.

[0788] "Annual income" refers to the estimated annual income of the visitor.

[0789] "Estimation" refers to predicting a visitor's annual income using a machine learning model based on acquired appearance information, etc.

[0790] "Database" refers to a system for storing and managing accumulated annual income data and other information.

[0791] "Product recommendation" refers to presenting the most suitable products to visitors based on accumulated annual income data.

[0792] "Display device" refers to a device such as a smartphone or tablet that displays product recommendation information and other notifications.

[0793] "Store staff" refers to employees who support sales activities in stores.

[0794] "Customer service support" refers to assistance provided to store staff to efficiently recommend products and provide services to customers.

[0795] "Statistical analysis" refers to a method for analyzing accumulated data and deriving specific patterns or trends.

[0796] This invention is a system for efficiently collecting annual income data of customers and using it to make decisions about store management, and includes the following components. Specifically, we will explain in detail the roles of the server and terminals.

[0797] Video data collection module (terminal)

[0798] The device captures real-time video data from high-resolution IP cameras installed in the store, recording customers as they enter and move around the store. The video data is then sent to a server via Wi-Fi.

[0799] Video analysis module (server)

[0800] The server receives the video data sent from the device and uses an image recognition algorithm to extract information about the visitor's appearance, including their age, gender, clothing, belongings, etc. Specifically, the server performs image analysis using OpenCV, TensorFlow, and the Google Cloud Vision API.

[0801] Annual income estimation module (server)

[0802] The server uses a machine learning model to estimate the visitor's annual income based on the extracted appearance information. This machine learning model is pre-trained and has learned the correlation between appearance information and annual income. Specific technologies used are scikit-learn and Python.

[0803] Database storage module (server)

[0804] The server stores the estimated annual income data in a database, which stores the annual income data and other appearance information of each visitor in an organized manner. The database is managed using MySQL or SQLite.

[0805] Data analysis module (server)

[0806] The server statistically analyzes the accumulated income data to determine what income brackets are attracting customers to specific stores and what products are being purchased. The analysis is performed using R and Python (Pandas, NumPy).

[0807] Product recommendation module (terminal)

[0808] The device recommends products suitable for visitors based on the annual income data sent from the server. It lists expensive products for visitors with high annual incomes and affordable products for visitors with low annual incomes. This is done using cloud-based data communication using Firebase.

[0809] Customer service support module (user)

[0810] Store staff, who are the users, can use their smartphones or tablets to check the displayed product recommendation information and help suggest appropriate products to customers. The built-in notification function and user interface enable efficient customer service.

[0811] Specific examples

[0812] For example, suppose a woman in her 30s visits a store one afternoon. Video data is acquired from a surveillance camera and sent to a server. The server analyzes the video and extracts information about the woman's appearance, such as her age, gender, clothing, and belongings. Based on this information, a machine learning model estimates her annual income, calculating it as, say, 4 million yen. The server stores this information in a database and analyzes it together with data from other visitors. The device then lists products suitable for a visitor with an annual income of 4 million yen and displays it to the store clerk. The clerk then uses this information to suggest products to the visitor.

[0813] Example prompts for generative AI models

[0814] "Estimate the annual income of women in their 30s and recommend products": Build a system that estimates the annual income of visitors from their appearance information (age, gender, clothing, belongings) and recommends appropriate products based on that income. Using the movie example, when a woman in her 30s enters a store, the system analyzes her age and clothing, estimates her annual income to be 4 million yen, and displays mid-priced products on a smartphone app. The store clerk uses this information to suggest products that suit her.

[0815] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0816] Step 1:

[0817] The terminal acquires video data in real time from high-resolution IP cameras installed in the store. Specifically, it captures images and videos of visitors through the camera lens and transmits the video data to a server via Wi-Fi connection. The input is the video data from the camera, and the output is the video data transmitted to the server via Wi-Fi.

[0818] Step 2:

[0819] The server receives the video data sent from the device. It uses an image recognition algorithm to extract the visitor's appearance information from the received video data. Specifically, it uses OpenCV, TensorFlow, and Google Cloud Vision API to analyze and extract characteristic information such as the visitor's age, gender, clothing, and belongings. The input is the video data, and the output is the extracted appearance information.

[0820] Step 3:

[0821] The server uses a machine learning model to estimate the visitor's annual income based on the extracted appearance information. Specifically, a pre-trained model using scikit-learn and Python is used to evaluate the correlation between appearance information and annual income and predict annual income. The input is appearance information, and the output is estimated annual income data.

[0822] Step 4:

[0823] The server stores the estimated annual income data and appearance information in a database. This database stores the annual income data and appearance information for each visitor in an organized manner. Specifically, data is managed efficiently using MySQL and SQLite. The input is annual income data and appearance information, and the output is data storage in the database.

[0824] Step 5:

[0825] The server statistically analyzes the income data stored in the database and analyzes the trends of visitors in specific income brackets. Specifically, it performs data analysis using R and Python (Pandas, NumPy). The input is the income data from the database, and the output is the analysis results.

[0826] Step 6:

[0827] The device recommends suitable products to visitors based on the analysis results sent from the server. Specifically, Firebase is used for cloud-based data communication and a list of recommended products is displayed on the smartphone or tablet. The input is the analysis results, and the output is the display data of the recommended products.

[0828] Step 7:

[0829] Store staff, who are users, check the product recommendation information displayed on their smartphones or tablets and suggest suitable products to visitors. Specifically, they check product information through the built-in notification function and user interface and provide appropriate customer service. The input is the display data of recommended products, and the output is the product suggestions to visitors.

[0830] By following the above steps, annual income data can be collected even if the visitor does not purchase a product, and this data can be used to recommend products and provide customer service support in real time.

[0831] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0832] This invention is a system for efficiently collecting annual income data and emotional data of customers and using it to help with store management decision-making and customer service. This system is realized by analyzing video data acquired from video devices such as surveillance cameras, extracting information on the appearance and emotional information of customers, and estimating their annual income.

[0833] System Overview

[0834] The system mainly consists of the following modules:

[0835] 1. Video data collection module (terminal)

[0836] 2. Video analysis module (server)

[0837] 3. Annual income estimation module (server)

[0838] 4. Emotion engine (server)

[0839] 5. Database storage module (server)

[0840] 6. Data analysis module (server)

[0841] 7. Product recommendation module (terminal)

[0842] 8. Customer Service Support Module (User)

[0843] Program processing

[0844] Video data collection module (terminal)

[0845] The device acquires video data in real time from surveillance cameras installed in the store. This video data records how customers move around the store after entering. For example, video of a customer entering the store at 10:00 a.m. is acquired.

[0846] Video analysis module (server)

[0847] The server receives the video data sent from the device and uses an image recognition algorithm to extract information about the visitor's face and body appearance. This information includes the visitor's age, gender, clothing, belongings, and type of shoes. For example, the captured video may identify the visitor as a man around 35 years old and wearing a business suit.

[0848] Annual income estimation module (server)

[0849] The server uses a machine learning model to estimate the visitor's annual income based on the extracted appearance information. This machine learning model is pre-trained and has learned the correlation between appearance information and annual income. For example, a visitor wearing a luxury watch is estimated to have an annual income of 8 million yen.

[0850] Emotion engine (server)

[0851] The server is equipped with an emotion engine that recognizes visitors' emotions from video data. This emotion engine identifies the visitor's emotional state (e.g., happiness, surprise, irritation, etc.) through facial expression analysis. For example, if a visitor smiles while looking at a product, the emotion engine recognizes this as "happiness."

[0852] Database storage module (server)

[0853] The server stores the estimated annual income data and emotion data in a database. This database systematically stores annual income data, emotion data, and other appearance information for each visitor. For example, data such as "male," "around 35 years old," "8 million yen," and "happiness" might be stored.

[0854] Data analysis module (server)

[0855] The server statistically analyzes the accumulated income data and emotional data to determine which income brackets are most frequent visitors to a particular store and what emotional state they are in when looking at products. The results of this analysis are used to improve the store's product lineup and develop new store plans. For example, it may be possible to determine that a particular store is visited by many high-income earners, or that visitors feel "happy" when looking at a particular product.

[0856] Product recommendation module (terminal)

[0857] The device generates a product list that recommends the most suitable products to the visitor based on the annual income data and emotional data sent from the server. Expensive products are listed for visitors with high annual incomes, while affordable products are listed for visitors with low annual incomes, and the device further recommends the most suitable products according to the visitor's emotional state. For example, if a visitor with an annual income of 8 million yen has a happy expression, a list of recommended luxury bags will be displayed first.

[0858] Customer service support module (user)

[0859] The user, a store clerk, checks the display on the device and suggests appropriate products to the customer. This creates an environment where customers can easily make a purchase. For example, if a customer smiles while looking at a luxury bag, the clerk can suggest a luxury bag and encourage them to try on the luxurious bag.

[0860] Specific examples

[0861] Suppose a woman in her 30s visits a store one afternoon. The device acquires video data from the surveillance camera and sends it to the server. The server analyzes the video and extracts information about the woman's appearance, such as her age, gender, clothing, and belongings. It also uses an emotion engine to recognize emotions such as "happiness" from the woman's facial expression. Based on this information, a machine learning model estimates her annual income, calculating it as 4 million yen, for example. The server stores this information in a database and analyzes it together with data from other visitors. The device then lists products suitable for a visitor with an annual income of 4 million yen who is emotionally happy, and displays this to the user, who is also a store clerk. The store clerk then uses this information to suggest products to the visitor.

[0862] This system can collect data on customers' annual income and emotions, even if they do not purchase products, and use this data to improve the store's product lineup and plan new stores. It can also provide more effective customer service by suggesting products based on the customer's emotional state.

[0863] The processing flow will be explained below.

[0864] Step 1:

[0865] The device captures real-time video data from the store's surveillance cameras, recording customers entering and moving around the store.

[0866] Step 2:

[0867] The device temporarily stores the acquired video data and periodically transmits it to the server, for example, transmitting frame data every second.

[0868] Step 3:

[0869] The server receives the video data sent from the terminal, and the received video data is input to the image analysis module.

[0870] Step 4:

[0871] The server uses video analysis algorithms to extract information about the visitor's facial and physical appearance from the video data, including facial recognition, gender estimation, and age estimation. For example, facial recognition technology is used to identify the visitor's face and estimate their gender as "male" and their age as "around 35 years old."

[0872] Step 5:

[0873] The server extracts further appearance information such as the visitor's clothing, belongings, and type of shoes, including attributes such as color, brand, and fashion style. For example, it may be recognized that the visitor is wearing a business suit and a luxury watch.

[0874] Step 6:

[0875] The server inputs the extracted appearance information into a machine learning model. The machine learning model learns the correlation between appearance information and annual income based on past data, and estimates the visitor's annual income. For example, if a visitor is wearing a business suit and a luxury watch, it can estimate that the visitor's annual income is 8 million yen.

[0876] Step 7:

[0877] The server runs an emotion engine that recognizes the visitor's emotions from the video data. The emotion engine identifies the visitor's emotional state (e.g., happiness, surprise, irritation, etc.) through facial expression analysis. For example, if a visitor is looking at a product and smiling, the emotion engine recognizes the visitor as "happy."

[0878] Step 8:

[0879] The server stores the estimated annual income data and emotional data in a database. The stored data includes the visitor's age, gender, appearance information, annual income, and emotional state. For example, the data stored might be "male," "around 35 years old," "8 million yen," and "happy."

[0880] Step 9:

[0881] The server statistically analyzes the income data and emotional data stored in the database to determine which income brackets are most frequent visitors to a particular store and what emotional state they are in when looking at products. The results of this analysis are used to improve the store's product lineup and develop new store plans. For example, it may be possible to determine that a particular store is visited by many high-income earners, or that visitors feel "happy" when looking at a particular product.

[0882] Step 10:

[0883] The device generates a product list that recommends the most suitable products to the visitor based on the annual income data and emotional data sent from the server. Expensive products are listed for visitors with high annual incomes, while affordable products are listed for visitors with low annual incomes, and the device further recommends the most suitable products according to the visitor's emotional state. For example, if a visitor with an annual income of 8 million yen has a happy expression, a list of recommended luxury bags will be displayed first.

[0884] Step 11:

[0885] The terminal displays the generated product list to the user, who is also a store clerk. The store clerk checks the displayed product list and suggests appropriate products to the visitor. For example, if a visitor smiles while looking at a luxury bag, the store clerk will suggest a luxury bag and recommend that the visitor try on a luxurious bag.

[0886] Example 2

[0887] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0888] In conventional store operations, the collection of information about customers was limited, making it difficult to grasp details such as the annual income and emotional state of customers. As a result, effective decision-making and product recommendations suited to customers were not fully implemented in store operations. In addition, it was difficult to provide individual support to stimulate customers' purchasing desire, resulting in a situation where sales could not be expected to increase.

[0889] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0890] In this invention, the server includes means for extracting appearance information of visitors from video data, means for estimating annual income based on the extracted appearance information, and means for analyzing the emotional state of visitors. This makes it possible to grasp annual income and emotions based on detailed information about visitors, support decision-making for store management, and suggest products suitable for visitors.

[0891] A "terminal" is a device that acquires video data of visitors and transmits it to a server.

[0892] A "server" is a computer system that analyzes video data, processes data such as appearance information, annual income, and emotional state, and stores and analyzes this data in a database.

[0893] "Video Data" refers to image and video data captured to record the appearance and actions of visitors.

[0894] "Appearance information" is data extracted from a visitor's age, gender, clothing, belongings, type of shoes, and other visual characteristics.

[0895] "Annual income" is the amount of income a visitor is estimated to earn per year.

[0896] "Emotional state" refers to the emotional state analyzed from the visitor's facial expression, and includes happiness, surprise, irritation, etc.

[0897] A "database" is a digital storage device for systematically storing information such as visitors' appearance, annual income, and emotional state.

[0898] "Data analysis" is the process of using data stored in a database to analyze trends and patterns related to customer visitation and store operations.

[0899] A "product list" is a list of products generated to recommend the most suitable products to visitors.

[0900] "User" refers to a store employee or salesperson who uses the system to suggest suitable products to visitors.

[0901] MODE FOR CARRYING OUT THE INVENTION

[0902] This invention is a system that efficiently collects annual income data and emotional data of customers and uses it to help with decision-making in store management and customer service. This system functions in cooperation with three entities: terminals, servers, and users.

[0903] Hardware and software used

[0904] The device includes a surveillance camera installed in the store that captures video data in real time. This video data is used to record how customers move around the store after entering. For example, a video of a customer entering the store at 10:00 AM is captured.

[0905] The server receives the video data sent from the device and uses an image recognition algorithm (such as TensorFlow or OpenCV) to analyze it. The server extracts the visitor's appearance information (age, gender, clothing, belongings, etc.). For example, the server recognizes from the captured video that the visitor is a man around 35 years old wearing a business suit.

[0906] Next, the server uses a machine learning model (for example, a model using scikit-learn or TensorFlow) to estimate the visitor's annual income based on the extracted appearance information. For example, a visitor wearing a luxury watch is estimated to have an annual income of 8 million yen.

[0907] The server is also equipped with an emotion engine that recognizes visitors' emotions from video data. This emotion engine identifies the visitor's emotional state (happiness, surprise, irritation, etc.) through facial expression analysis. For example, if a visitor smiles while looking at a product, the emotion engine will recognize this as "happiness."

[0908] The server stores the estimated annual income data and emotion data in a database. This database stores the annual income data, emotion data, and other appearance information for each visitor in an organized format. For example, data such as "male," "around 35 years old," "8 million yen," and "happy" might be stored.

[0909] The data stored in the database is statistically analyzed by a server. The analysis results provide information such as which income brackets are most frequent visitors to a particular store and the emotional state customers are in when looking at products. These analysis results are used to improve the store's product lineup and formulate new store opening plans. For example, it may be discovered that a particular store is visited by many high-income earners, or that visitors feel "happy" when looking at a particular product.

[0910] Specific examples

[0911] Let's say a woman in her 30s visits a store one afternoon. The device acquires video data from the surveillance camera and sends it to the server. The server analyzes the video and extracts information about the woman's appearance, such as her age, gender, clothing, and belongings. It also uses an emotion engine to recognize emotions such as "happiness" from the woman's facial expression. Based on this information, a machine learning model estimates her annual income, calculating it as 4 million yen, for example. The server stores this information in a database and analyzes it together with data from other visitors. The device then lists products suitable for a visitor with an annual income of 4 million yen who is emotionally happy, and displays this to the user, the store clerk. The store clerk then uses this information to suggest products that the visitor might like.

[0912] Prompt Sentence Examples

[0913] "Write a program that generates the best product listings for a visitor with a specific facial expression and annual income."

[0914] This system can collect data on customers' annual income and emotions, even if they do not purchase products, and use this data to improve the store's product lineup and plan new stores. It can also provide more effective customer service by suggesting products based on the customer's emotional state.

[0915] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0916] Step 1:

[0917] The terminal acquires video data in real time from surveillance cameras installed in the store. The input data is the video footage sent from the surveillance cameras, and the output data is the video footage itself. Specifically, the surveillance cameras capture the movement of customers within the store from the moment they enter. For example, the terminal acquires video of Visitor A, who entered the store at 10:00 AM.

[0918] Step 2:

[0919] The video data captured by the device is sent to the server in real time. The input data is the video captured by the device, and the output data is the video sent to the server. The video data is encrypted to protect privacy. For example, the video data of visitor A is transferred from the camera to the server within a few seconds.

[0920] Step 3:

[0921] The server analyzes the video data received from the device and extracts the visitor's appearance information using an image recognition algorithm. The input data is the transmitted video data, and the output data is the extracted appearance information. Specifically, the server uses a deep learning model to identify the visitor's age, gender, clothing, belongings, etc. For example, it recognizes that Visitor A is a man around 35 years old wearing a business suit.

[0922] Step 4:

[0923] The server estimates the visitor's annual income using a pre-trained machine learning model based on the extracted appearance information. The input data is the extracted appearance information, and the output data is the estimated annual income. Specifically, annual income is predicted based on personal belongings such as luxury watches and designer bags. For example, visitor A is wearing a luxury watch, so his annual income is estimated to be 8 million yen.

[0924] Step 5:

[0925] The server uses an emotion engine to analyze the emotional state of the visitor. The input data is the transmitted video data, and the output data is the analyzed emotional state. Specifically, facial expression analysis technology is used to recognize emotions such as happiness, surprise, and irritation. For example, if visitor A is looking at a product and smiling, the emotion engine will recognize this as "happiness."

[0926] Step 6:

[0927] The server stores the estimated annual income data and emotional state data in a database. The input data is the estimated annual income data and emotional state data, and the output data is the data accumulated in the database. The database stores detailed information about each visitor in an organized manner. For example, data such as "male," "around 35 years old," "8 million yen," and "happy" is stored.

[0928] Step 7:

[0929] The server performs statistical analysis based on the accumulated data. The input data is visitor information stored in the database, and the output data is the results of the statistical analysis. The analysis results reveal the visitor demographics and emotional trends at a particular store, and are useful for improving store operations. For example, it can reveal that visitors with high incomes tend to feel "happy" with certain products.

[0930] Step 8:

[0931] The terminal generates a product list that recommends the most suitable products to the visitor based on the annual income data and emotion data sent from the server. The input data are the annual income data and emotion data sent from the server, and the output data is the generated product list. Specifically, appropriate products are listed according to the visitor's annual income and emotion. For example, a recommended list of luxury bags is displayed to visitor A.

[0932] Step 9:

[0933] The user, a store clerk, checks the display on the terminal and suggests appropriate products to the visitor. The input data is the product list displayed on the terminal, and the output data is the product suggestions to the visitor. The store clerk refers to the product list and provides customer service according to the visitor's emotional state. For example, if visitor A has a happy expression, the store clerk will recommend a luxury bag and offer a special try-on service.

[0934] This series of processing flows allows detailed annual income data and emotional data to be collected from visitors even if they do not purchase any products, enabling improved store operations and more effective customer service support.

[0935] (Application example 2)

[0936] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0937] In current brick-and-mortar store operations, it is difficult to grasp the attributes and emotional state of visitors in real time and immediately make appropriate product recommendations. This is particularly true when targeting a large number of visitors, as it is difficult to provide highly accurate product recommendations and customer service support for each individual customer. Furthermore, there are limited means for making store management decisions based on objective data, which can hinder improvements in sales and customer satisfaction. Furthermore, because customer service does not take into account the emotions of visitors, opportunities to stimulate purchasing motivation are often missed.

[0938] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0939] In this invention, the server includes means for acquiring video data of visitors, means for extracting appearance information of the visitors from the acquired video data, means for estimating the annual income of the visitors based on the extracted appearance information, means for storing the estimated annual income data and emotion data in a database, means for analyzing the accumulated annual income data and emotion data to support decision-making related to store management, means for recognizing the emotional state of the visitors from the video data, and means for transmitting data for display on a smart device. This enables highly accurate product recommendations and customer service support based on the annual income and emotion data of the visitors. Furthermore, store management decisions can be made based on objective data, which can result in expected improvements in sales and customer satisfaction.

[0940] A "customer" is someone who visits a physical store to receive a particular product or service.

[0941] "Video data" refers to video signals and image information obtained from surveillance cameras, smart devices, etc.

[0942] "Appearance information" refers to information such as a visitor's facial and physical features, age, gender, clothing, belongings, and type of shoes.

[0943] "Annual income" refers to the total income earned by visitors in a year.

[0944] "Emotional data" refers to information about the emotional state of a visitor as recognized from their facial expressions, actions, etc.

[0945] A "database" is a storage location for structured information that systematically stores collected information and makes it easy to search and analyze.

[0946] "Smart devices" refer to electronic devices such as mobile devices, smart glasses, and tablets that have internet connectivity and advanced processing capabilities.

[0947] "Store operations" refers to the daily operations of a store, such as store management, sales activities, customer service, and product management.

[0948] Decision making refers to choosing the best course of action or strategy to achieve a particular goal.

[0949] This invention provides a system that extracts appearance information and emotional data from video data of visitors in real time, estimates their annual income based on the extracted data, and is useful for store management and customer service support.

[0950] The system consists of the following modules:

[0951] 1. Video data collection module (terminal)

[0952] The device acquires video data in real time from surveillance cameras installed in the store. This video data records how customers enter the store and move around inside the store. For example, video of a customer entering the store at 10:00 AM is acquired.

[0953] 2. Video analysis module (server)

[0954] The server receives the video data sent from the device and uses an image recognition algorithm to extract information about the visitor's face and body appearance. This information includes the visitor's age, gender, clothing, belongings, and type of shoes. For example, the captured video may identify the visitor as a man around 35 years old and wearing a business suit.

[0955] 3. Annual income estimation module (server)

[0956] The server uses a machine learning model to estimate the visitor's annual income based on the extracted appearance information. This machine learning model is pre-trained and has learned the correlation between appearance information and annual income. For example, a visitor wearing a luxury watch is estimated to have an annual income of 8 million yen.

[0957] 4. Emotion engine (server)

[0958] The server is equipped with an emotion engine that recognizes visitors' emotions from video data. This emotion engine identifies the visitor's emotional state (e.g., happiness, surprise, irritation, etc.) through facial expression analysis. For example, if a visitor smiles while looking at a product, the emotion engine recognizes this as "happiness."

[0959] 5. Database storage module (server)

[0960] The server stores the estimated annual income data and emotion data in a database. This database systematically stores annual income data, emotion data, and other appearance information for each visitor. For example, data such as "male," "around 35 years old," "8 million yen," and "happiness" might be stored.

[0961] 6. Data analysis module (server)

[0962] The server statistically analyzes the accumulated income data and emotional data to determine which income brackets are most frequent visitors to a particular store and what emotional state they are in when looking at products. The results of this analysis are used to improve the store's product lineup and develop new store plans. For example, it may be possible to determine that a particular store is visited by many high-income earners, or that visitors feel "happy" when looking at a particular product.

[0963] 7. Product recommendation module (terminal)

[0964] The device generates a product list that recommends the most suitable products to the visitor based on the annual income data and emotional data sent from the server. Expensive products are listed for visitors with high annual incomes, while affordable products are listed for visitors with low annual incomes, and the device further recommends the most suitable products according to the visitor's emotional state. For example, if a visitor with an annual income of 8 million yen has a happy expression, a list of recommended luxury bags will be displayed first.

[0965] 8. Customer Service Support Module (User)

[0966] The user, a store clerk, checks the display on the device and suggests appropriate products to the customer. This creates an environment where customers can easily make a purchase. For example, if a customer smiles while looking at a luxury bag, the clerk can suggest a luxury bag and encourage them to try on the luxurious bag.

[0967] Specific examples

[0968] Suppose a woman in her 30s visits a store one afternoon. The device acquires video data from the surveillance camera and sends it to the server. The server analyzes the video and extracts information about the woman's appearance, such as her age, gender, clothing, and belongings. It also uses an emotion engine to recognize emotions such as "happiness" from the woman's facial expression. Based on this information, a machine learning model estimates her annual income, calculating it as 4 million yen, for example. The server stores this information in a database and analyzes it together with data from other visitors. The device then lists products suitable for a visitor with an annual income of 4 million yen who is emotionally happy, and displays this to the user, who is also a store clerk. The store clerk then uses this information to suggest products to the visitor.

[0969] Prompt Sentence Examples

[0970] We would like to implement a system that estimates annual income and recommends products based on visitors' appearance and emotional data.

[0971] Write a program to add a feature that displays real-time annual income estimates and emotions through an application installed on smart glasses when a customer enters a store.

[0972] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0973] Step 1:

[0974] Acquiring video data (device)

[0975] The terminal acquires video data in real time from surveillance cameras installed in the store. This video data records how customers enter the store and move around the store. For example, video of a customer entering the store at 10:00 AM is acquired. This video data becomes the input for the system.

[0976] Step 2:

[0977] Video data transmission (terminal)

[0978] The device transmits the acquired video data to the server. This data transmission process is performed in real time, and the video data arrives at the server without delay. The transmitted video data becomes the input for the next processing step.

[0979] Step 3:

[0980] Extraction of appearance information (server)

[0981] The server uses an image recognition algorithm to extract information about the visitor's face and body appearance based on the received video data. Specifically, it performs facial recognition on the video data and extracts facial features. The extracted appearance information includes the visitor's age, gender, clothing, belongings, and type of shoes. For example, it may recognize that the visitor is a man around 35 years old and wearing a business suit. This appearance information becomes the input for the next processing step.

[0982] Step 4:

[0983] Estimated annual income (server)

[0984] The server uses a pre-trained machine learning model to estimate the visitor's annual income based on the extracted appearance information. For example, a visitor wearing a luxury watch might be estimated to have an annual income of 8 million yen. This annual income data becomes the input for the next processing step.

[0985] Step 5:

[0986] Emotion Recognition (Server)

[0987] The server is equipped with an emotion engine that recognizes the emotions of visitors from video data. This emotion engine identifies the emotional state of the visitor (e.g., happiness, surprise, irritation, etc.) through facial expression analysis. For example, if a visitor smiles while looking at a product, the emotion engine recognizes this as "happiness." This emotion data becomes the input for the next processing step.

[0988] Step 6:

[0989] Data storage (server)

[0990] The server stores the estimated annual income data and emotion data in a database. This database systematically stores annual income data, emotion data, and other appearance information for each visitor. For example, data such as "male," "around 35 years old," "8 million yen," and "happy" is stored. This stored data becomes the input for the next processing step.

[0991] Step 7:

[0992] Data analysis (server)

[0993] The server statistically analyzes the accumulated income data and emotional data to analyze which income brackets are most frequent visitors to a particular store and what emotional state they are in when looking at products. The results of this analysis may reveal, for example, that a particular store is frequented by many high-income earners, or that visitors feel "happy" when looking at a particular product. These analysis results become the input for the next processing step.

[0994] Step 8:

[0995] Product recommendation generation (device)

[0996] The device generates a product list that recommends the most suitable products to the visitor based on the annual income data and emotional data sent from the server. Expensive products are listed for visitors with high annual incomes, and affordable products for visitors with low annual incomes, and the device further recommends the most suitable products according to the visitor's emotional state. For example, if a visitor with an annual income of 8 million yen has a happy expression, a list of recommended luxury bags will be displayed preferentially. This product list becomes the input for the next processing step.

[0997] Step 9:

[0998] Customer service support (user)

[0999] The user, a store clerk, checks the display on the terminal and suggests appropriate products to the customer. For example, if a customer smiles while looking at a luxury bag, the store clerk suggests a luxury bag and recommends that the customer try on a luxurious bag.

[1000] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1001] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1002] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1003] [Fourth embodiment]

[1004] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1005] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1006] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1007] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1008] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1009] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1010] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1011] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1012] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1013] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1014] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1015] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1016] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1017] This invention is a system for efficiently collecting annual income data of customers and utilizing it for decision-making in store management. This system is realized by analyzing video data acquired from video devices such as surveillance cameras, extracting information about the appearance of customers, and estimating their annual income.

[1018] System Overview

[1019] The system mainly consists of the following modules:

[1020] 1. Video data collection module (terminal)

[1021] 2. Video analysis module (server)

[1022] 3. Annual income estimation module (server)

[1023] 4. Database storage module (server)

[1024] 5. Data analysis module (server)

[1025] 6. Product recommendation module (terminal)

[1026] 7. Customer Service Support Module (User)

[1027] Program processing

[1028] Video data collection module (terminal)

[1029] The device acquires video data in real time from surveillance cameras installed in the store. This video data records how customers move around the store after entering. For example, video of a customer entering the store at 10:00 a.m. is acquired.

[1030] Video analysis module (server)

[1031] The server receives the video data sent from the device and uses an image recognition algorithm to extract information about the visitor's appearance, including their age, gender, clothing, belongings, and type of shoes. For example, the captured video may identify the visitor as a man around 35 years old wearing a business suit.

[1032] Annual income estimation module (server)

[1033] The server uses a machine learning model to estimate the visitor's annual income based on the extracted appearance information. This machine learning model is pre-trained and has learned the correlation between appearance information and annual income. For example, a visitor wearing a luxury watch is estimated to have an annual income of 8 million yen.

[1034] Database storage module (server)

[1035] The server stores the estimated income data in a database, which systematically stores income data and other appearance information for each visitor, for example, the income data of visitors who visited at a specific date and time.

[1036] Data analysis module (server)

[1037] The server statistically analyzes the accumulated annual income data to determine what income brackets are attracting customers to a particular store and what products are being purchased. The results of this analysis are used to improve the store's product lineup and formulate new store opening plans. For example, it can be determined that a particular store is frequented by customers with an annual income of 8 million yen or more.

[1038] Product recommendation module (terminal)

[1039] The device recommends products suitable for visitors based on the annual income data sent from the server. It lists expensive products for visitors with high annual incomes and affordable products for visitors with low annual incomes. For example, a visitor with an annual income of 5 million yen will be shown a list of mid-priced products.

[1040] Customer service support module (user)

[1041] The user, a store clerk, checks the display on the device and suggests appropriate products to the customer. This creates an environment where customers can easily make purchases. For example, a store clerk suggests a luxury bag to a customer wearing an expensive suit.

[1042] Specific examples

[1043] Suppose a woman in her 30s visits a store one afternoon. The device acquires video data from the surveillance camera and sends it to the server. The server analyzes the video and extracts information about the woman's appearance, such as her age, gender, clothing, and belongings. Based on this information, a machine learning model estimates her annual income, calculating it as 4 million yen, for example. The server stores this information in a database and analyzes it together with data from other visitors. The device then lists products suitable for a visitor with an annual income of 4 million yen and displays it to the user, who is also a store clerk. The clerk uses this information to suggest products to the visitor.

[1044] This system allows annual income data to be collected even if visitors do not purchase products, and can be used to make decisions about store management.

[1045] The processing flow will be explained below.

[1046] Step 1:

[1047] The device captures real-time video data from the store's surveillance cameras, recording customers entering and moving around the store.

[1048] Step 2:

[1049] The device temporarily stores the acquired video data and periodically transmits it to the server, for example, transmitting frame data every second.

[1050] Step 3:

[1051] The server receives the video data sent from the terminal, and the received video data is input to the image analysis module.

[1052] Step 4:

[1053] The server uses video analysis algorithms to extract information about the visitor's facial and physical appearance from the video data, including facial recognition, gender estimation, and age estimation. For example, facial recognition technology is used to identify the visitor's face and estimate their gender as "male" and their age as "around 35 years old."

[1054] Step 5:

[1055] The server extracts further appearance information such as the visitor's clothing, belongings, and type of shoes, including attributes such as color, brand, and fashion style. For example, it may be recognized that the visitor is wearing a business suit and a luxury watch.

[1056] Step 6:

[1057] The server inputs the extracted appearance information into a machine learning model. The machine learning model learns the correlation between appearance information and annual income based on past data, and estimates the visitor's annual income. For example, if a visitor is wearing a business suit and a luxury watch, it can estimate that the visitor's annual income is 8 million yen.

[1058] Step 7:

[1059] The server stores the estimated annual income data in a database. The stored data includes the visitor's age, gender, appearance information, and estimated annual income. For example, data such as "male," "around 35 years old," and "8 million yen" might be stored.

[1060] Step 8:

[1061] The server statistically analyzes the income data stored in the database to determine which income brackets are most frequent visitors to a particular store and what products are being purchased. For example, it may analyze whether a particular store is frequented by a large number of people in the high income bracket.

[1062] Step 9:

[1063] The device generates a product list that recommends the most suitable products to each visitor based on the estimated annual income data. For example, it would recommend luxury bags to a visitor with an annual income of 8 million yen.

[1064] Step 10:

[1065] The terminal displays the generated product list to the user, who is also a store clerk. The store clerk checks the displayed product list and suggests suitable products to the customer. For example, they may explain where to find luxury bags and recommend trying them on.

[1066] This allows the system to collect annual income data even if visitors do not purchase products, and use it to improve the store's product lineup and plan new stores.It can also increase purchasing motivation by suggesting products that are suitable for visitors.

[1067] Example 1

[1068] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1069] In current store operations, it is difficult to obtain data on the annual income of customers, limiting the effectiveness of marketing strategies and product recommendations. It has also been difficult to establish a system that estimates annual income based on information about customers' appearance and then uses that data to recommend specific products and make operational decisions.

[1070] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1071] In this invention, the server includes means for collecting video data of visitors from a video capture device, means for compressing the collected video data and transmitting it to a data server, means for analyzing appearance information of visitors from the video data in the data server, means for estimating the annual income of visitors from the appearance information using a machine learning model, means for storing the estimated annual income data in a database, means for statistically analyzing the stored annual income data and utilizing it in store management, and means for recommending products suitable for visitors based on the analysis results. This makes it possible to efficiently collect annual income data of visitors, improve the accuracy of marketing strategies and product recommendations, and support decision-making in store management.

[1072] The "image capture device" is a device for collecting image data of visitors.

[1073] The "means for compressing collected video data and transmitting it to a data server" refers to a means for effectively compressing collected video data and transmitting it to a data server using a secure communication method.

[1074] The "means for analyzing visitor appearance information from video data using a data server" refers to a means for analyzing video data and extracting visitor appearance information (age, gender, clothing, belongings, etc.).

[1075] The "means for estimating the annual income of a visitor from appearance information using a machine learning model" is a means for estimating the annual income of a visitor based on acquired appearance information using machine learning technology.

[1076] "Means for storing in a database" means a means for systematically storing all relevant information, including estimated annual income data.

[1077] "Means of statistically analyzing accumulated annual income data and utilizing it in store management" refers to a means of conducting statistical analysis based on accumulated annual income data and using the results to help with store management strategies and decision-making.

[1078] The "means for recommending products suitable for visitors based on the analysis results" refers to a means for selecting and recommending the most suitable products to each visitor using the results of statistical analysis.

[1079] The present invention provides a system for efficiently collecting annual income data of customers and utilizing it for decision-making in store management. The system includes an image capture device, a data server, a machine learning model, a database, a statistical analysis means, and a product recommendation means.

[1080] First, the terminal collects real-time video data of visitors using a video capture device installed in the store. The collected video data is compressed using video compression formats such as H.264 or H.265 and sent to a data server via HTTPS. The data server analyzes the received video data and extracts information about the visitor's appearance using image recognition algorithms such as OpenCV and TensorFlow. This information includes age, gender, clothing, and belongings.

[1081] The server then uses a machine learning model to estimate the visitor's annual income from the extracted appearance information. This machine learning model has been trained in advance on a large dataset and has learned the correlation between appearance information and annual income. For example, it can estimate the annual income of a visitor wearing a luxury watch to be 8 million yen.

[1082] The estimated annual income data is stored in a database on the server. This database systematically stores information such as the age, gender, and annual income of each visitor. The server analyzes the accumulated data using statistical analysis libraries such as Pandas, providing useful insights into store operations.

[1083] Furthermore, the server generates a list of recommended products suitable for the visitor based on the analysis results and sends this information to the terminal. The terminal displays the received product recommendation list, and the user (store clerk) suggests appropriate products to the visitor based on this list. Specifically, the server displays a list of expensive products for visitors with high annual incomes and affordable products for visitors with low annual incomes to the user.

[1084] Specific examples

[1085] Suppose a woman in her 30s visits a store one afternoon. The device acquires video data from the surveillance camera and sends it to the server. The server analyzes the video and extracts information about the woman's appearance, such as her age, gender, clothing, and belongings. Based on this information, a machine learning model estimates her annual income, calculating it as 4 million yen, for example. The server stores this information in a database and analyzes it together with data from other visitors. The device then lists products suitable for a visitor with an annual income of 4 million yen and displays it to the user, who is also a store clerk. The clerk uses this information to suggest products to the visitor.

[1086] Prompt Sentence Examples

[1087] Here are some example prompts to explain the system's operations to the generative AI model:

[1088] "Surveillance cameras installed in stores capture footage of customers, and the video data is analyzed to extract information about the customer's appearance (age, gender, clothing, belongings, etc.). Next, a machine learning model is used to estimate the customer's annual income from this appearance information and the information is stored in a database. Based on the stored data, appropriate products are recommended to the customer, and finally, a store associate suggests appropriate products to the customer. Please explain this process step by step."

[1089] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1090] Step 1:

[1091] The terminal collects video data of visitors in real time from a video capture device installed in the store. This input data records how customers move around the store after entering. Specifically, the surveillance camera captures video at about 60 frames per second and stores it on the terminal.

[1092] Step 2:

[1093] The terminal compresses the collected video data using video compression methods such as H.264 or H.265 and sends it to the data server using the HTTPS protocol. This reduces the amount of data traffic and ensures secure transmission. Specifically, the video data is compressed using batch processing and uploaded to the server at regular intervals.

[1094] Step 3:

[1095] The server receives the video data sent from the device and analyzes the visitor's appearance using image recognition algorithms such as OpenCV and TensorFlow. The input data is video frames, and the output extracts appearance information such as the visitor's age, gender, clothing, and belongings. Specifically, a deep learning model for video analysis processes the image data for each frame and extracts specific features.

[1096] Step 4:

[1097] The server uses a machine learning model (e.g., a sparse ridge regression model) to estimate the visitor's annual income from the appearance information. This model is trained in advance using a training dataset. Based on the appearance information extracted as input, the visitor's annual income is estimated as output. Specifically, the model performs a regression analysis using the appearance information variables to calculate an estimated annual income.

[1098] Step 5:

[1099] The server accumulates the estimated annual income data in a database. The input data is the estimated annual income data, and the output is saved in the database. Specifically, the server uses an SQL query to register the annual income data in the database, linking it with the visitor's ID.

[1100] Step 6:

[1101] The server statistically analyzes the accumulated annual income data. The input data is the annual income data stored in the database, and the output provides analysis reports and data trends. Specifically, it converts the data into a data frame using the Pandas library and calculates various statistics (such as the mean, median, and standard deviation).

[1102] Step 7:

[1103] The server generates a list of recommended products suitable for the visitor based on the analysis results. The input data is annual income data and the analysis results, and the output is a list of recommended products. Specifically, the server filters products from the product database according to annual income category and generates the list.

[1104] Step 8:

[1105] The terminal receives the product recommendation list sent from the server and displays it to the user, who is a store clerk. The input data is the product recommendation list, and the output is displayed on the terminal's display. Specifically, the terminal parses the received data and updates the interface displayed on the screen.

[1106] Step 9:

[1107] The user, a store clerk, refers to the product recommendation list displayed on the terminal and suggests appropriate products to the visitor. The input data is the information displayed on the terminal, and the output is specific product suggestions to the visitor. In concrete terms, the store clerk explains the features and benefits of the products to the visitor based on the displayed list, encouraging them to make a purchase.

[1108] (Application example 1)

[1109] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1110] In store operations, efficiently collecting annual income data of customers and using that data to recommend products and provide customer service support contributes to improving store profits and customer satisfaction. However, conventional methods have made it difficult to efficiently collect such data before customers make a purchase, and there has been a lack of concrete means to utilize the collected data in store management decision-making. Furthermore, there has been no system that can make product recommendations tailored to customers in real time.

[1111] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1112] In this invention, the server includes means for acquiring video data of visitors, means for extracting appearance information of the visitors from the acquired video data, means for estimating the annual income of the visitors based on the extracted appearance information, means for storing the estimated annual income data in a database, means for recommending products suitable for the visitors based on the stored annual income data, means for displaying information about the recommended products on a display device and supporting store staff in making suggestions to the visitors, and means for statistically analyzing the annual income data stored in the database and analyzing specific income brackets. This makes it possible to collect annual income data even if the visitors do not purchase products, and to utilize the data to recommend products and provide customer service support in real time.

[1113] "Visitor" refers to an individual who visits a store with the intent of using its services or products.

[1114] "Video data" refers to image and video information acquired from video devices such as surveillance cameras within a store.

[1115] "Appearance information" refers to visual characteristics of visitors, such as their age, gender, clothing, and belongings, extracted from video data.

[1116] "Annual income" refers to the estimated annual income of the visitor.

[1117] "Estimation" refers to predicting a visitor's annual income using a machine learning model based on acquired appearance information, etc.

[1118] "Database" refers to a system for storing and managing accumulated annual income data and other information.

[1119] "Product recommendation" refers to presenting the most suitable products to visitors based on accumulated annual income data.

[1120] "Display device" refers to a device such as a smartphone or tablet that displays product recommendation information and other notifications.

[1121] "Store staff" refers to employees who support sales activities in stores.

[1122] "Customer service support" refers to assistance provided to store staff to efficiently recommend products and provide services to customers.

[1123] "Statistical analysis" refers to a method for analyzing accumulated data and deriving specific patterns or trends.

[1124] This invention is a system for efficiently collecting annual income data of customers and using it to make decisions about store management, and includes the following components. Specifically, we will explain in detail the roles of the server and terminals.

[1125] Video data collection module (terminal)

[1126] The device captures real-time video data from high-resolution IP cameras installed in the store, recording customers as they enter and move around the store. The video data is then sent to a server via Wi-Fi.

[1127] Video analysis module (server)

[1128] The server receives the video data sent from the device and uses an image recognition algorithm to extract information about the visitor's appearance, including their age, gender, clothing, belongings, etc. Specifically, the server performs image analysis using OpenCV, TensorFlow, and the Google Cloud Vision API.

[1129] Annual income estimation module (server)

[1130] The server uses a machine learning model to estimate the visitor's annual income based on the extracted appearance information. This machine learning model is pre-trained and has learned the correlation between appearance information and annual income. Specific technologies used are scikit-learn and Python.

[1131] Database storage module (server)

[1132] The server stores the estimated annual income data in a database, which stores the annual income data and other appearance information of each visitor in an organized manner. The database is managed using MySQL or SQLite.

[1133] Data analysis module (server)

[1134] The server statistically analyzes the accumulated income data to determine what income brackets are attracting customers to specific stores and what products are being purchased. The analysis is performed using R and Python (Pandas, NumPy).

[1135] Product recommendation module (terminal)

[1136] The device recommends products suitable for visitors based on the annual income data sent from the server. It lists expensive products for visitors with high annual incomes and affordable products for visitors with low annual incomes. This is done using cloud-based data communication using Firebase.

[1137] Customer service support module (user)

[1138] Store staff, who are the users, can use their smartphones or tablets to check the displayed product recommendation information and help suggest appropriate products to customers. The built-in notification function and user interface enable efficient customer service.

[1139] Specific examples

[1140] For example, suppose a woman in her 30s visits a store one afternoon. Video data is acquired from a surveillance camera and sent to a server. The server analyzes the video and extracts information about the woman's appearance, such as her age, gender, clothing, and belongings. Based on this information, a machine learning model estimates her annual income, calculating it as, say, 4 million yen. The server stores this information in a database and analyzes it together with data from other visitors. The device then lists products suitable for a visitor with an annual income of 4 million yen and displays it to the store clerk. The clerk then uses this information to suggest products to the visitor.

[1141] Example prompts for generative AI models

[1142] "Estimate the annual income of women in their 30s and recommend products": Build a system that estimates the annual income of visitors from their appearance information (age, gender, clothing, belongings) and recommends appropriate products based on that income. Using the movie example, when a woman in her 30s enters a store, the system analyzes her age and clothing, estimates her annual income to be 4 million yen, and displays mid-priced products on a smartphone app. The store clerk uses this information to suggest products that suit her.

[1143] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1144] Step 1:

[1145] The terminal acquires video data in real time from high-resolution IP cameras installed in the store. Specifically, it captures images and videos of visitors through the camera lens and transmits the video data to a server via Wi-Fi connection. The input is the video data from the camera, and the output is the video data transmitted to the server via Wi-Fi.

[1146] Step 2:

[1147] The server receives the video data sent from the device. It uses an image recognition algorithm to extract the visitor's appearance information from the received video data. Specifically, it uses OpenCV, TensorFlow, and Google Cloud Vision API to analyze and extract characteristic information such as the visitor's age, gender, clothing, and belongings. The input is the video data, and the output is the extracted appearance information.

[1148] Step 3:

[1149] The server uses a machine learning model to estimate the visitor's annual income based on the extracted appearance information. Specifically, a pre-trained model using scikit-learn and Python is used to evaluate the correlation between appearance information and annual income and predict annual income. The input is appearance information, and the output is estimated annual income data.

[1150] Step 4:

[1151] The server stores the estimated annual income data and appearance information in a database. This database stores the annual income data and appearance information for each visitor in an organized manner. Specifically, data is managed efficiently using MySQL and SQLite. The input is annual income data and appearance information, and the output is data storage in the database.

[1152] Step 5:

[1153] The server statistically analyzes the income data stored in the database and analyzes the trends of visitors in specific income brackets. Specifically, it performs data analysis using R and Python (Pandas, NumPy). The input is the income data from the database, and the output is the analysis results.

[1154] Step 6:

[1155] The device recommends suitable products to visitors based on the analysis results sent from the server. Specifically, Firebase is used for cloud-based data communication and a list of recommended products is displayed on the smartphone or tablet. The input is the analysis results, and the output is the display data of the recommended products.

[1156] Step 7:

[1157] Store staff, who are users, check the product recommendation information displayed on their smartphones or tablets and suggest suitable products to visitors. Specifically, they check product information through the built-in notification function and user interface and provide appropriate customer service. The input is the display data of recommended products, and the output is the product suggestions to visitors.

[1158] By following the above steps, annual income data can be collected even if the visitor does not purchase a product, and this data can be used to recommend products and provide customer service support in real time.

[1159] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1160] This invention is a system for efficiently collecting annual income data and emotional data of customers and using it to help with store management decision-making and customer service. This system is realized by analyzing video data acquired from video devices such as surveillance cameras, extracting information on the appearance and emotional information of customers, and estimating their annual income.

[1161] System Overview

[1162] The system mainly consists of the following modules:

[1163] 1. Video data collection module (terminal)

[1164] 2. Video analysis module (server)

[1165] 3. Annual income estimation module (server)

[1166] 4. Emotion engine (server)

[1167] 5. Database storage module (server)

[1168] 6. Data analysis module (server)

[1169] 7. Product recommendation module (terminal)

[1170] 8. Customer Service Support Module (User)

[1171] Program processing

[1172] Video data collection module (terminal)

[1173] The device acquires video data in real time from surveillance cameras installed in the store. This video data records how customers move around the store after entering. For example, video of a customer entering the store at 10:00 a.m. is acquired.

[1174] Video analysis module (server)

[1175] The server receives the video data sent from the device and uses an image recognition algorithm to extract information about the visitor's face and body appearance. This information includes the visitor's age, gender, clothing, belongings, and type of shoes. For example, the captured video may identify the visitor as a man around 35 years old and wearing a business suit.

[1176] Annual income estimation module (server)

[1177] The server uses a machine learning model to estimate the visitor's annual income based on the extracted appearance information. This machine learning model is pre-trained and has learned the correlation between appearance information and annual income. For example, a visitor wearing a luxury watch is estimated to have an annual income of 8 million yen.

[1178] Emotion engine (server)

[1179] The server is equipped with an emotion engine that recognizes visitors' emotions from video data. This emotion engine identifies the visitor's emotional state (e.g., happiness, surprise, irritation, etc.) through facial expression analysis. For example, if a visitor smiles while looking at a product, the emotion engine recognizes this as "happiness."

[1180] Database storage module (server)

[1181] The server stores the estimated annual income data and emotion data in a database. This database systematically stores annual income data, emotion data, and other appearance information for each visitor. For example, data such as "male," "around 35 years old," "8 million yen," and "happiness" might be stored.

[1182] Data analysis module (server)

[1183] The server statistically analyzes the accumulated income data and emotional data to determine which income brackets are most frequent visitors to a particular store and what emotional state they are in when looking at products. The results of this analysis are used to improve the store's product lineup and develop new store plans. For example, it may be possible to determine that a particular store is visited by many high-income earners, or that visitors feel "happy" when looking at a particular product.

[1184] Product recommendation module (terminal)

[1185] The device generates a product list that recommends the most suitable products to the visitor based on the annual income data and emotional data sent from the server. Expensive products are listed for visitors with high annual incomes, while affordable products are listed for visitors with low annual incomes, and the device further recommends the most suitable products according to the visitor's emotional state. For example, if a visitor with an annual income of 8 million yen has a happy expression, a list of recommended luxury bags will be displayed first.

[1186] Customer service support module (user)

[1187] The user, a store clerk, checks the display on the device and suggests appropriate products to the customer. This creates an environment where customers can easily make a purchase. For example, if a customer smiles while looking at a luxury bag, the clerk can suggest a luxury bag and encourage them to try on the luxurious bag.

[1188] Specific examples

[1189] Suppose a woman in her 30s visits a store one afternoon. The device acquires video data from the surveillance camera and sends it to the server. The server analyzes the video and extracts information about the woman's appearance, such as her age, gender, clothing, and belongings. It also uses an emotion engine to recognize emotions such as "happiness" from the woman's facial expression. Based on this information, a machine learning model estimates her annual income, calculating it as 4 million yen, for example. The server stores this information in a database and analyzes it together with data from other visitors. The device then lists products suitable for a visitor with an annual income of 4 million yen who is emotionally happy, and displays this to the user, who is also a store clerk. The store clerk then uses this information to suggest products to the visitor.

[1190] This system can collect data on customers' annual income and emotions, even if they do not purchase products, and use this data to improve the store's product lineup and plan new stores. It can also provide more effective customer service by suggesting products based on the customer's emotional state.

[1191] The processing flow will be explained below.

[1192] Step 1:

[1193] The device captures real-time video data from the store's surveillance cameras, recording customers entering and moving around the store.

[1194] Step 2:

[1195] The device temporarily stores the acquired video data and periodically transmits it to the server, for example, transmitting frame data every second.

[1196] Step 3:

[1197] The server receives the video data sent from the terminal, and the received video data is input to the image analysis module.

[1198] Step 4:

[1199] The server uses video analysis algorithms to extract information about the visitor's facial and physical appearance from the video data, including facial recognition, gender estimation, and age estimation. For example, facial recognition technology is used to identify the visitor's face and estimate their gender as "male" and their age as "around 35 years old."

[1200] Step 5:

[1201] The server extracts further appearance information such as the visitor's clothing, belongings, and type of shoes, including attributes such as color, brand, and fashion style. For example, it may be recognized that the visitor is wearing a business suit and a luxury watch.

[1202] Step 6:

[1203] The server inputs the extracted appearance information into a machine learning model. The machine learning model learns the correlation between appearance information and annual income based on past data, and estimates the visitor's annual income. For example, if a visitor is wearing a business suit and a luxury watch, it can estimate that the visitor's annual income is 8 million yen.

[1204] Step 7:

[1205] The server runs an emotion engine that recognizes the visitor's emotions from the video data. The emotion engine identifies the visitor's emotional state (e.g., happiness, surprise, irritation, etc.) through facial expression analysis. For example, if a visitor is looking at a product and smiling, the emotion engine recognizes the visitor as "happy."

[1206] Step 8:

[1207] The server stores the estimated annual income data and emotional data in a database. The stored data includes the visitor's age, gender, appearance information, annual income, and emotional state. For example, the data stored might be "male," "around 35 years old," "8 million yen," and "happy."

[1208] Step 9:

[1209] The server statistically analyzes the income data and emotional data stored in the database to determine which income brackets are most frequent visitors to a particular store and what emotional state they are in when looking at products. The results of this analysis are used to improve the store's product lineup and develop new store plans. For example, it may be possible to determine that a particular store is visited by many high-income earners, or that visitors feel "happy" when looking at a particular product.

[1210] Step 10:

[1211] The device generates a product list that recommends the most suitable products to the visitor based on the annual income data and emotional data sent from the server. Expensive products are listed for visitors with high annual incomes, while affordable products are listed for visitors with low annual incomes, and the device further recommends the most suitable products according to the visitor's emotional state. For example, if a visitor with an annual income of 8 million yen has a happy expression, a list of recommended luxury bags will be displayed first.

[1212] Step 11:

[1213] The terminal displays the generated product list to the user, who is also a store clerk. The store clerk checks the displayed product list and suggests appropriate products to the visitor. For example, if a visitor smiles while looking at a luxury bag, the store clerk will suggest a luxury bag and recommend that the visitor try on a luxurious bag.

[1214] Example 2

[1215] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1216] In conventional store operations, the collection of information about customers was limited, making it difficult to grasp details such as the annual income and emotional state of customers. As a result, effective decision-making and product recommendations suited to customers were not fully implemented in store operations. In addition, it was difficult to provide individual support to stimulate customers' purchasing desire, resulting in a situation where sales could not be expected to increase.

[1217] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1218] In this invention, the server includes means for extracting appearance information of visitors from video data, means for estimating annual income based on the extracted appearance information, and means for analyzing the emotional state of visitors. This makes it possible to grasp annual income and emotions based on detailed information about visitors, support decision-making for store management, and suggest products suitable for visitors.

[1219] A "terminal" is a device that acquires video data of visitors and transmits it to a server.

[1220] A "server" is a computer system that analyzes video data, processes data such as appearance information, annual income, and emotional state, and stores and analyzes this data in a database.

[1221] "Video Data" refers to image and video data captured to record the appearance and actions of visitors.

[1222] "Appearance information" is data extracted from a visitor's age, gender, clothing, belongings, type of shoes, and other visual characteristics.

[1223] "Annual income" is the amount of income a visitor is estimated to earn per year.

[1224] "Emotional state" refers to the emotional state analyzed from the visitor's facial expression, and includes happiness, surprise, irritation, etc.

[1225] A "database" is a digital storage device for systematically storing information such as visitors' appearance, annual income, and emotional state.

[1226] "Data analysis" is the process of using data stored in a database to analyze trends and patterns related to customer visitation and store operations.

[1227] A "product list" is a list of products generated to recommend the most suitable products to visitors.

[1228] "User" refers to a store employee or salesperson who uses the system to suggest suitable products to visitors.

[1229] MODE FOR CARRYING OUT THE INVENTION

[1230] This invention is a system that efficiently collects annual income data and emotional data of customers and uses it to help with decision-making in store management and customer service. This system functions in cooperation with three entities: terminals, servers, and users.

[1231] Hardware and software used

[1232] The device includes a surveillance camera installed in the store that captures video data in real time. This video data is used to record how customers move around the store after entering. For example, a video of a customer entering the store at 10:00 AM is captured.

[1233] The server receives the video data sent from the device and uses an image recognition algorithm (such as TensorFlow or OpenCV) to analyze it. The server extracts the visitor's appearance information (age, gender, clothing, belongings, etc.). For example, the server recognizes from the captured video that the visitor is a man around 35 years old wearing a business suit.

[1234] Next, the server uses a machine learning model (for example, a model using scikit-learn or TensorFlow) to estimate the visitor's annual income based on the extracted appearance information. For example, a visitor wearing a luxury watch is estimated to have an annual income of 8 million yen.

[1235] The server is also equipped with an emotion engine that recognizes visitors' emotions from video data. This emotion engine identifies the visitor's emotional state (happiness, surprise, irritation, etc.) through facial expression analysis. For example, if a visitor smiles while looking at a product, the emotion engine will recognize this as "happiness."

[1236] The server stores the estimated annual income data and emotion data in a database. This database stores the annual income data, emotion data, and other appearance information for each visitor in an organized format. For example, data such as "male," "around 35 years old," "8 million yen," and "happy" might be stored.

[1237] The data stored in the database is statistically analyzed by a server. The analysis results provide information such as which income brackets are most frequent visitors to a particular store and the emotional state customers are in when looking at products. These analysis results are used to improve the store's product lineup and formulate new store opening plans. For example, it may be discovered that a particular store is visited by many high-income earners, or that visitors feel "happy" when looking at a particular product.

[1238] Specific examples

[1239] Let's say a woman in her 30s visits a store one afternoon. The device acquires video data from the surveillance camera and sends it to the server. The server analyzes the video and extracts information about the woman's appearance, such as her age, gender, clothing, and belongings. It also uses an emotion engine to recognize emotions such as "happiness" from the woman's facial expression. Based on this information, a machine learning model estimates her annual income, calculating it as 4 million yen, for example. The server stores this information in a database and analyzes it together with data from other visitors. The device then lists products suitable for a visitor with an annual income of 4 million yen who is emotionally happy, and displays this to the user, the store clerk. The store clerk then uses this information to suggest products that the visitor might like.

[1240] Prompt Sentence Examples

[1241] "Write a program that generates the best product listings for a visitor with a specific facial expression and annual income."

[1242] This system can collect data on customers' annual income and emotions, even if they do not purchase products, and use this data to improve the store's product lineup and plan new stores. It can also provide more effective customer service by suggesting products based on the customer's emotional state.

[1243] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1244] Step 1:

[1245] The terminal acquires video data in real time from surveillance cameras installed in the store. The input data is the video footage sent from the surveillance cameras, and the output data is the video footage itself. Specifically, the surveillance cameras capture the movement of customers within the store from the moment they enter. For example, the terminal acquires video of Visitor A, who entered the store at 10:00 AM.

[1246] Step 2:

[1247] The video data captured by the device is sent to the server in real time. The input data is the video captured by the device, and the output data is the video sent to the server. The video data is encrypted to protect privacy. For example, the video data of visitor A is transferred from the camera to the server within a few seconds.

[1248] Step 3:

[1249] The server analyzes the video data received from the device and extracts the visitor's appearance information using an image recognition algorithm. The input data is the transmitted video data, and the output data is the extracted appearance information. Specifically, the server uses a deep learning model to identify the visitor's age, gender, clothing, belongings, etc. For example, it recognizes that Visitor A is a man around 35 years old wearing a business suit.

[1250] Step 4:

[1251] The server estimates the visitor's annual income using a pre-trained machine learning model based on the extracted appearance information. The input data is the extracted appearance information, and the output data is the estimated annual income. Specifically, annual income is predicted based on personal belongings such as luxury watches and designer bags. For example, visitor A is wearing a luxury watch, so his annual income is estimated to be 8 million yen.

[1252] Step 5:

[1253] The server uses an emotion engine to analyze the emotional state of the visitor. The input data is the transmitted video data, and the output data is the analyzed emotional state. Specifically, facial expression analysis technology is used to recognize emotions such as happiness, surprise, and irritation. For example, if visitor A is looking at a product and smiling, the emotion engine will recognize this as "happiness."

[1254] Step 6:

[1255] The server stores the estimated annual income data and emotional state data in a database. The input data is the estimated annual income data and emotional state data, and the output data is the data accumulated in the database. The database stores detailed information about each visitor in an organized manner. For example, data such as "male," "around 35 years old," "8 million yen," and "happy" is stored.

[1256] Step 7:

[1257] The server performs statistical analysis based on the accumulated data. The input data is visitor information stored in the database, and the output data is the results of the statistical analysis. The analysis results reveal the visitor demographics and emotional trends at a particular store, and are useful for improving store operations. For example, it can reveal that visitors with high incomes tend to feel "happy" with certain products.

[1258] Step 8:

[1259] The terminal generates a product list that recommends the most suitable products to the visitor based on the annual income data and emotion data sent from the server. The input data are the annual income data and emotion data sent from the server, and the output data is the generated product list. Specifically, appropriate products are listed according to the visitor's annual income and emotion. For example, a recommended list of luxury bags is displayed to visitor A.

[1260] Step 9:

[1261] The user, a store clerk, checks the display on the terminal and suggests appropriate products to the visitor. The input data is the product list displayed on the terminal, and the output data is the product suggestions to the visitor. The store clerk refers to the product list and provides customer service according to the visitor's emotional state. For example, if visitor A has a happy expression, the store clerk will recommend a luxury bag and offer a special try-on service.

[1262] This series of processing flows allows detailed annual income data and emotional data to be collected from visitors even if they do not purchase any products, enabling improved store operations and more effective customer service support.

[1263] (Application example 2)

[1264] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1265] In current brick-and-mortar store operations, it is difficult to grasp the attributes and emotional state of visitors in real time and immediately make appropriate product recommendations. This is particularly true when targeting a large number of visitors, as it is difficult to provide highly accurate product recommendations and customer service support for each individual customer. Furthermore, there are limited means for making store management decisions based on objective data, which can hinder improvements in sales and customer satisfaction. Furthermore, because customer service does not take into account the emotions of visitors, opportunities to stimulate purchasing motivation are often missed.

[1266] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1267] In this invention, the server includes means for acquiring video data of visitors, means for extracting appearance information of the visitors from the acquired video data, means for estimating the annual income of the visitors based on the extracted appearance information, means for storing the estimated annual income data and emotion data in a database, means for analyzing the accumulated annual income data and emotion data to support decision-making related to store management, means for recognizing the emotional state of the visitors from the video data, and means for transmitting data for display on a smart device. This enables highly accurate product recommendations and customer service support based on the annual income and emotion data of the visitors. Furthermore, store management decisions can be made based on objective data, which can result in expected improvements in sales and customer satisfaction.

[1268] A "customer" is someone who visits a physical store to receive a particular product or service.

[1269] "Video data" refers to video signals and image information obtained from surveillance cameras, smart devices, etc.

[1270] "Appearance information" refers to information such as a visitor's facial and physical features, age, gender, clothing, belongings, and type of shoes.

[1271] "Annual income" refers to the total income earned by visitors in a year.

[1272] "Emotional data" refers to information about the emotional state of a visitor as recognized from their facial expressions, actions, etc.

[1273] A "database" is a storage location for structured information that systematically stores collected information and makes it easy to search and analyze.

[1274] "Smart devices" refer to electronic devices such as mobile devices, smart glasses, and tablets that have internet connectivity and advanced processing capabilities.

[1275] "Store operations" refers to the daily operations of a store, such as store management, sales activities, customer service, and product management.

[1276] Decision making refers to choosing the best course of action or strategy to achieve a particular goal.

[1277] This invention provides a system that extracts appearance information and emotional data from video data of visitors in real time, estimates their annual income based on the extracted data, and is useful for store management and customer service support.

[1278] The system consists of the following modules:

[1279] 1. Video data collection module (terminal)

[1280] The device acquires video data in real time from surveillance cameras installed in the store. This video data records how customers enter the store and move around inside the store. For example, video of a customer entering the store at 10:00 AM is acquired.

[1281] 2. Video analysis module (server)

[1282] The server receives the video data sent from the device and uses an image recognition algorithm to extract information about the visitor's face and body appearance. This information includes the visitor's age, gender, clothing, belongings, and type of shoes. For example, the captured video may identify the visitor as a man around 35 years old and wearing a business suit.

[1283] 3. Annual income estimation module (server)

[1284] The server uses a machine learning model to estimate the visitor's annual income based on the extracted appearance information. This machine learning model is pre-trained and has learned the correlation between appearance information and annual income. For example, a visitor wearing a luxury watch is estimated to have an annual income of 8 million yen.

[1285] 4. Emotion engine (server)

[1286] The server is equipped with an emotion engine that recognizes visitors' emotions from video data. This emotion engine identifies the visitor's emotional state (e.g., happiness, surprise, irritation, etc.) through facial expression analysis. For example, if a visitor smiles while looking at a product, the emotion engine recognizes this as "happiness."

[1287] 5. Database storage module (server)

[1288] The server stores the estimated annual income data and emotion data in a database. This database systematically stores annual income data, emotion data, and other appearance information for each visitor. For example, data such as "male," "around 35 years old," "8 million yen," and "happiness" might be stored.

[1289] 6. Data analysis module (server)

[1290] The server statistically analyzes the accumulated income data and emotional data to determine which income brackets are most frequent visitors to a particular store and what emotional state they are in when looking at products. The results of this analysis are used to improve the store's product lineup and develop new store plans. For example, it may be possible to determine that a particular store is visited by many high-income earners, or that visitors feel "happy" when looking at a particular product.

[1291] 7. Product recommendation module (terminal)

[1292] The device generates a product list that recommends the most suitable products to the visitor based on the annual income data and emotional data sent from the server. Expensive products are listed for visitors with high annual incomes, while affordable products are listed for visitors with low annual incomes, and the device further recommends the most suitable products according to the visitor's emotional state. For example, if a visitor with an annual income of 8 million yen has a happy expression, a list of recommended luxury bags will be displayed first.

[1293] 8. Customer Service Support Module (User)

[1294] The user, a store clerk, checks the display on the device and suggests appropriate products to the customer. This creates an environment where customers can easily make a purchase. For example, if a customer smiles while looking at a luxury bag, the clerk can suggest a luxury bag and encourage them to try on the luxurious bag.

[1295] Specific examples

[1296] Suppose a woman in her 30s visits a store one afternoon. The device acquires video data from the surveillance camera and sends it to the server. The server analyzes the video and extracts information about the woman's appearance, such as her age, gender, clothing, and belongings. It also uses an emotion engine to recognize emotions such as "happiness" from the woman's facial expression. Based on this information, a machine learning model estimates her annual income, calculating it as 4 million yen, for example. The server stores this information in a database and analyzes it together with data from other visitors. The device then lists products suitable for a visitor with an annual income of 4 million yen who is emotionally happy, and displays this to the user, who is also a store clerk. The store clerk then uses this information to suggest products to the visitor.

[1297] Prompt Sentence Examples

[1298] We would like to implement a system that estimates annual income and recommends products based on visitors' appearance and emotional data.

[1299] Write a program to add a feature that displays real-time annual income estimates and emotions through an application installed on smart glasses when a customer enters a store.

[1300] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1301] Step 1:

[1302] Acquiring video data (device)

[1303] The terminal acquires video data in real time from surveillance cameras installed in the store. This video data records how customers enter the store and move around the store. For example, video of a customer entering the store at 10:00 AM is acquired. This video data becomes the input for the system.

[1304] Step 2:

[1305] Video data transmission (terminal)

[1306] The device transmits the acquired video data to the server. This data transmission process is performed in real time, and the video data arrives at the server without delay. The transmitted video data becomes the input for the next processing step.

[1307] Step 3:

[1308] Extraction of appearance information (server)

[1309] The server uses an image recognition algorithm to extract information about the visitor's face and body appearance based on the received video data. Specifically, it performs facial recognition on the video data and extracts facial features. The extracted appearance information includes the visitor's age, gender, clothing, belongings, and type of shoes. For example, it may recognize that the visitor is a man around 35 years old and wearing a business suit. This appearance information becomes the input for the next processing step.

[1310] Step 4:

[1311] Estimated annual income (server)

[1312] The server uses a pre-trained machine learning model to estimate the visitor's annual income based on the extracted appearance information. For example, a visitor wearing a luxury watch might be estimated to have an annual income of 8 million yen. This annual income data becomes the input for the next processing step.

[1313] Step 5:

[1314] Emotion Recognition (Server)

[1315] The server is equipped with an emotion engine that recognizes the emotions of visitors from video data. This emotion engine identifies the emotional state of the visitor (e.g., happiness, surprise, irritation, etc.) through facial expression analysis. For example, if a visitor smiles while looking at a product, the emotion engine recognizes this as "happiness." This emotion data becomes the input for the next processing step.

[1316] Step 6:

[1317] Data storage (server)

[1318] The server stores the estimated annual income data and emotion data in a database. This database systematically stores annual income data, emotion data, and other appearance information for each visitor. For example, data such as "male," "around 35 years old," "8 million yen," and "happy" is stored. This stored data becomes the input for the next processing step.

[1319] Step 7:

[1320] Data analysis (server)

[1321] The server statistically analyzes the accumulated income data and emotional data to analyze which income brackets are most frequent visitors to a particular store and what emotional state they are in when looking at products. The results of this analysis may reveal, for example, that a particular store is frequented by many high-income earners, or that visitors feel "happy" when looking at a particular product. These analysis results become the input for the next processing step.

[1322] Step 8:

[1323] Product recommendation generation (device)

[1324] The device generates a product list that recommends the most suitable products to the visitor based on the annual income data and emotional data sent from the server. Expensive products are listed for visitors with high annual incomes, and affordable products for visitors with low annual incomes, and the device further recommends the most suitable products according to the visitor's emotional state. For example, if a visitor with an annual income of 8 million yen has a happy expression, a list of recommended luxury bags will be displayed preferentially. This product list becomes the input for the next processing step.

[1325] Step 9:

[1326] Customer service support (user)

[1327] The user, a store clerk, checks the display on the terminal and suggests appropriate products to the customer. For example, if a customer smiles while looking at a luxury bag, the store clerk suggests a luxury bag and recommends that the customer try on a luxurious bag.

[1328] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1329] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1330] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1331] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1332] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1333] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1334] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1335] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1336] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1337] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1338] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1339] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1340] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1341] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1342] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1343] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1344] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1345] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1346] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1347] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1348] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1349] The following is further disclosed regarding the above embodiment.

[1350] (Claim 1)

[1351] A means for acquiring video data of visitors;

[1352] means for extracting appearance information of visitors from the acquired video data;

[1353] A means for estimating the annual income of the visitor based on the extracted appearance information;

[1354] a means for storing the estimated annual income data in a database;

[1355] A system that includes a means of analyzing accumulated annual income data to support decision-making related to store management.

[1356] (Claim 2)

[1357] The system of claim 1, further comprising means for recommending suitable products to the visitor based on the annual income data.

[1358] (Claim 3)

[1359] 10. The system of claim 1, further comprising means for estimating the age and gender of the visitor from the video data.

[1360] "Example 1"

[1361] (Claim 1)

[1362] means for collecting visitor video data from a video capture device;

[1363] means for compressing the collected video data and transmitting it to a data server;

[1364] A means for analyzing the appearance information of visitors from the video data in a data server;

[1365] A method for estimating the annual income of visitors based on their appearance information using a machine learning model;

[1366] a means for storing the estimated annual income data in a database;

[1367] A method for statistically analyzing accumulated annual income data and utilizing it in store management,

[1368] The system includes a means for recommending suitable products to visitors based on the analysis results.

[1369] (Claim 2)

[1370] 10. The system of claim 1, further comprising means for estimating the age and gender of the visitor.

[1371] (Claim 3)

[1372] 10. The system of claim 1, further comprising means for determining whether the guest possesses a luxury item.

[1373] "Application Example 1"

[1374] (Claim 1)

[1375] A means for acquiring video data of visitors;

[1376] means for extracting appearance information of visitors from the acquired video data;

[1377] A means for estimating the annual income of the visitor based on the extracted appearance information;

[1378] a means for storing the estimated annual income data in a database;

[1379] A means of analyzing accumulated annual income data to support decision-making regarding store management, and

[1380] A means of recommending suitable products to visitors based on accumulated annual income data;

[1381] a means for displaying recommended product information on a display device and supporting store staff in making recommendations to customers;

[1382] A system including:

[1383] (Claim 2)

[1384] 2. The system according to claim 1, further comprising means for statistically analyzing the income data stored in the database and analyzing specific income brackets.

[1385] (Claim 3)

[1386] 10. The system of claim 1, further comprising means for estimating the age and gender of the visitor from the video data.

[1387] "Example 2: Combining Emotion Engines"

[1388] (Claim 1)

[1389] A means for the terminal to acquire video data of the visitor;

[1390] means for the server to extract appearance information of the visit...

Claims

1. A means for acquiring video data of visitors; means for extracting appearance information of visitors from the acquired video data; A means for estimating the annual income of the visitor based on the extracted appearance information; a means for storing the estimated annual income data in a database; A system that includes a means of analyzing accumulated annual income data to support decision-making related to store management.

2. The system of claim 1 , further comprising means for recommending suitable products to the visitor based on the annual income data.

3. The system of claim 1 further comprising means for estimating the age and gender of the visitor from the video data.

Images