System

Abstract

A system is provided.SOLUTION: A system comprising: means for acquiring location data; means for analyzing the acquired location data and calculating a crowdedness level; and means for providing the calculated crowdedness level to a user.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In large-scale gatherings such as traditional events, festivals, and fireworks displays, participants have few ways to understand the congestion situation in advance, and there is a lack of information to help them have a comfortable experience. This has led to problems such as participants concentrating in crowded areas, increasing the risk of unpleasant experiences and accidents. In addition, it has been difficult to quickly grasp changes in the congestion situation in real time, making it difficult to take effective measures. [Means for solving the problem]

[0005] The present invention provides a system that includes a means for acquiring location data, a means for analyzing the acquired location data and calculating the degree of congestion, and a means for providing the calculated degree of congestion to the user. Specifically, user location data is collected from a sensor and preprocessed based on this data. Next, the degree of congestion is calculated based on this data and provided to the user in a visually easy-to-understand format, such as a color-coded display. User feedback data is also collected and used to improve the accuracy of the analysis. This allows participants at events and festivals to understand the congestion situation in real time and obtain information to help them have a comfortable experience.

[0006] "Location data" is information indicating the current location and past movement routes of a user or a specific object.

[0007] "Crowding level" is an index that indicates the density of people and objects in a particular region or area, and is a numerical representation of the degree of congestion.

[0008] A "sensor" is a device that detects a physical, environmental, or chemical quantity and produces a corresponding output signal.

[0009] "Preprocessing" refers to a series of steps that convert collected raw data into a format that can be applied to analytics and machine learning models.

[0010] "Color-coded display" is a method of displaying information using different colors to make it easier to understand visually.

[0011] "Real-time" refers to very low latency in collecting and processing data and delivering results.

[0012] "Feedback data" refers to information such as reactions, opinions, and movement history provided by users, and is used to improve and optimize the system.

[0013] "Analysis accuracy" is an indicator of how accurately the results of data analysis reflect the actual situation.

[0014] "User" means any person or entity that uses the system or application to obtain services or information. [Brief explanation of the drawings]

[0015] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14]FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0016] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0017] First, the terms used in the following description will be explained.

[0018] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0019] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0020] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0021] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0022] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0023] [First embodiment]

[0024] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0025] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0026] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0027] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0028] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0029] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0030] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0031] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0032] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0033] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0034] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0035] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0036] The present invention relates to a system that allows users to grasp the congestion situation in real time at large gatherings such as events and festivals, and provides information to help them spend their time comfortably.

[0037] This system first has a means of collecting location data. Specifically, the server acquires data using sensors (e.g., cameras, Wi-Fi connections, BLE beacons, etc.) installed in each area of ​​the event venue. These sensors collect participants' location information and movement patterns in real time and send them to the server.

[0038] The collected data is preprocessed by the server, which formats the location data, fills in missing data, and removes outliers. Once the data is properly organized, the server uses a machine learning algorithm to calculate the congestion level of each area. This congestion level is calculated based on data such as the number of people in each area and their movement speed.

[0039] Next, the server generates congestion information based on the analysis results. This congestion information is converted into a form that is easy for users to understand visually (e.g., a color-coded map). Specifically, the congestion score is divided into categories for color-coding, for example, congested areas are displayed in red and uncrowded areas in green.

[0040] Users can check congestion information in real time through the app on their own devices (smartphones, tablets, etc.). The app has a map screen that displays the congestion status of each area in a color-coded format. This allows users to select an area where they can avoid crowds and spend their time comfortably.

[0041] The system also includes a means for collecting user feedback data. The system periodically sends the user's movement data and operation history within the app to a server, and uses this feedback data to improve the accuracy of the analysis. Specifically, the system tracks changes in the congestion level in the area selected by the user, and reflects this data in future congestion predictions.

[0042] For example, suppose a user is attending a fireworks festival and launches the app. The user can check the congestion status of each area on a map in a color-coded format. Since red areas indicate congestion, the user will choose a green, less crowded area and spend time there. After the event ends, the user's movement data is sent to the server, which will contribute to improving the analysis accuracy for the next event.

[0043] In this way, the present invention allows users to grasp the congestion situation in real time and enjoy the event comfortably, and also enables event organizers to quickly take measures to alleviate congestion.

[0044] The processing flow will be explained below.

[0045] Step 1: Data collection

[0046] The server collects location data from sensors installed in each area (e.g., cameras, Wi-Fi connections, BLE beacons, etc.). These sensors detect participants' location information and movement patterns in real time and send the data to the server. Specifically, the server periodically obtains data from the sensors using API requests and centrally manages the collected data.

[0047] Step 2: Preprocessing the data

[0048] The server converts the acquired raw data into an analyzable format, specifically by performing preprocessing such as filling in missing data, filtering out outliers, and standardizing the data format, so that the location data is stored in a consistent format and subsequent analysis can be performed accurately.

[0049] Step 3: Calculate congestion

[0050] The server uses a machine learning algorithm to calculate the congestion level of each area using the pre-processed data. Specifically, it calculates a congestion score based on the number of people in each area, their movement speed, past data, etc. This allows the congestion status of each area to be quantified in real time.

[0051] Step 4: Generate congestion information

[0052] Based on the calculated congestion score, the server generates congestion information in a form that is visually easy for users to understand. Specifically, the congestion score is converted into a color-coded display (e.g., red indicates congestion, green indicates vacant) and reflected in the map data. This color-coded map is intended to allow users to easily understand the congestion situation.

[0053] Step 5: Distributing information

[0054] The server distributes the generated congestion information to the user's device in real time. Specifically, it sends the information to the user's device using push notifications or API endpoints. This allows the user to always check the latest congestion status.

[0055] Step 6: User interaction

[0056] Users can launch the app on their own devices and check the congestion situation on the map screen. Specifically, they can look at each color-coded area on the map, select an area that is not crowded, and move to that area. This allows users to avoid crowds and enjoy the event in comfort.

[0057] Step 7: Gather feedback

[0058] The device periodically sends the user's movement data and operation history within the app to the server. Specifically, it periodically collects changes in congestion in the area selected by the user and movement history and sends them to the server. This allows the server to analyze the user's behavioral data and use it as feedback to improve the accuracy of future analysis.

[0059] Example 1

[0060] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0061] At large gatherings such as traditional events and festivals, participants have had limited means of understanding congestion levels in advance, making it difficult for them to enjoy a comfortable stay. Furthermore, because collected data is not analyzed or displayed in real time, it is not possible to provide appropriate information to participants in a timely manner. Furthermore, there is a lack of means to collect and utilize user feedback data, making it difficult to improve the accuracy of congestion predictions.

[0062] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0063] In this invention, the server includes a means for acquiring location data, a means for preprocessing the acquired location data to reshape the location information data, fill in missing data, and remove outliers, a means for calculating congestion levels using a machine learning algorithm based on the preprocessed data, a means for categorizing the calculated congestion level scores and generating a visually easy-to-understand color-coded map, a means for providing the generated congestion information to a user terminal, and a means for collecting user feedback data and using it to improve the accuracy of congestion level predictions. This allows users to understand congestion situations in real time and select appropriate areas for a comfortable stay. Furthermore, utilizing user feedback data can also contribute to improving the accuracy of future congestion predictions.

[0064] "Location data" refers to data that indicates the location of people or objects within a specific area, such as an event venue.

[0065] "Preprocessing" is the process of formatting raw location data, filling in missing data, removing outliers, etc., to make the data suitable for analysis.

[0066] A "machine learning algorithm" is an algorithm that analyzes data to find patterns and rules and uses them for future predictions, classification, etc.

[0067] "Crowding level" is a numerical representation of the density of people and objects in a specific area, and is an indicator of the congestion level of the area.

[0068] A "color-coded map" is a map that displays areas in different colors to visually indicate the degree of congestion.

[0069] A "user terminal" is an electronic device used by a user, such as a smartphone or tablet.

[0070] "Feedback data" is data collected to help improve the system, such as user behavior data and in-app operation history.

[0071] The present invention relates to a system that allows users to grasp the congestion situation in real time at large gatherings such as events and festivals, and provides information to help them spend their time comfortably. Specifically, the system includes the following means.

[0072] First, the server collects location data of participants using sensors installed at the event venue (e.g., cameras, Wi-Fi connection count, BLE beacons, etc.). This location data is sent to the server in real time. For example, a Wi-Fi connection count sensor can estimate the congestion level of the area based on the number of connected devices.

[0073] Next, the server preprocesses the collected raw location data. Preprocessing steps include formatting the location data, filling in missing data, and filtering out outliers. This organizes the data into a form suitable for analysis. For example, data formatting can be performed using the Python Pandas library.

[0074] After the preprocessing is complete, the server applies a machine learning algorithm (e.g., K-means clustering using the Scikit-learn library) to calculate the congestion level of each area, taking into account multiple data points such as the number of people in each area and their movement speed.

[0075] Once the congestion level is calculated, the server generates congestion information based on the analysis results. This congestion information is provided as a visually easy-to-understand color-coded map (using Mapbox or OpenStreetMap, for example). The congestion level score is divided into categories, for example, congested areas are displayed in red and vacant areas in green.

[0076] Users can check congestion information in real time through a dedicated app on their smartphones, tablets, or other devices. The app has a map screen that shows the congestion status of each area in a color-coded format. This allows users to avoid crowds and select an area where they can spend their time comfortably.

[0077] After the event ends, the server will collect user movement data and operation history within the app, and use this data as feedback to improve the accuracy of future congestion predictions. A Python data analysis library (e.g., Pandas) can be used to analyze this feedback data.

[0078] As a concrete example, consider the case where a user attends a fireworks display. The user can launch a dedicated app and check the congestion status of each area on a map in a color-coded format. Since red areas indicate congestion, the user will choose a green, less crowded area and spend time there. After the event ends, the user's movement data is sent to the server, which will contribute to improving the accuracy of analysis for the next event.

[0079] Examples of prompts for generative AI models include:

[0080] "Please create a program for a system that will allow users to grasp the congestion situation in real time at large gatherings such as events and festivals, and provide information to help them have a comfortable time. This program will include a process to preprocess location data collected by sensors, calculate the degree of congestion, and display it on a color-coded map. Please also include a function to utilize user feedback data to improve the accuracy of the analysis."

[0081] As described above, the present invention is a system that allows users to understand congestion situations in real time and enjoy events comfortably, and also contributes to improving the accuracy of congestion predictions by using collected feedback data.

[0082] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0083] Step 1:

[0084] The server collects location data using sensors (e.g., cameras, Wi-Fi connection count, BLE beacons, etc.) installed at the event venue. The data collected by the sensors is sent to the server in real time. This input data includes the location information and movement patterns of each participant. For example, a Wi-Fi connection count sensor can estimate the congestion level of the area from the number of connected devices. The output is raw location data.

[0085] Step 2:

[0086] The server pre-processes the collected raw location data. The raw location data is used as input data. The pre-processing includes the following specific operations:

[0087] Data Transformation: Convert location data into a unified format.

[0088] Missing data completion: If a sensor is temporarily out of service, the data from that time is completed using data from nearby sensors.

[0089] Outlier exclusion: If some data is detected as abnormal, it will be excluded.

[0090] The result of this preprocessing is clean location data suitable for analysis.

[0091] Step 3:

[0092] The server applies machine learning algorithms to the preprocessed data to calculate the congestion level of each area. The preprocessed location data is used as input data. The server analyzes people's density and movement patterns by using K-means clustering, for example, using Python's Scikit-learn library. This outputs the congestion level of each area as a numerical value.

[0093] Step 4:

[0094] The server generates congestion information based on the analysis results. Numerical data indicating the degree of congestion is used as input data. Specifically, the congestion scores are categorized and converted into a visually easy-to-understand color-coded map. For example, congested areas are displayed in red and vacant areas in green. Tools such as Mapbox and OpenStreetMap are used for this task. A visually displayable color-coded map is generated as output.

[0095] Step 5:

[0096] Users use devices such as smartphones and tablets to check congestion information in real time through a dedicated app. A color-coded map delivered from a server is used as input data. Specifically, the app is launched and the map screen is displayed, where the congestion status of each area is shown in color-coded format. Based on this, users can select areas to avoid congestion and spend time in those areas. The output is a congestion status map that the user can view.

[0097] Step 6:

[0098] The server collects user movement data and in-app operation history after the event ends or periodically. The user's location information and operation history are used as input data. Specifically, it tracks the user's movement patterns and the congestion status of the selected area and stores this in a database. This feedback data is used for analysis to improve the accuracy of future congestion predictions. For example, data analysis is performed using Python's Pandas library. The output makes it possible to improve the congestion prediction algorithm.

[0099] (Application example 1)

[0100] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0101] Large gatherings and events, especially in brick-and-mortar locations like shopping malls, present challenges for visitors, making it difficult to understand crowding conditions and ensure a comfortable experience. Real-time information on how to avoid crowded areas is lacking, often resulting in a poor visitor experience. Additionally, there is insufficient collection and utilization of feedback data to improve crowd prediction accuracy.

[0102] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0103] In this invention, the server includes a means for acquiring location data, a means for analyzing the acquired location data and calculating the congestion level, a means for providing the calculated congestion level to the user, a means for suggesting a route to avoid congestion based on the user's location information, and a means for collecting user feedback data and improving the accuracy of the analysis. This not only enables visitors to check the congestion situation in real time and select a comfortable area, but also makes it possible to suggest the optimal route to avoid congestion. Furthermore, by collecting and analyzing user feedback, the accuracy of congestion level prediction can be improved.

[0104] "Location data" is numerical data that indicates the current location of an individual user or group at a specific location, such as an event or shopping mall.

[0105] "Analysis" is the process of processing collected raw data and converting it into meaningful information.

[0106] "Crowding" is an indicator that indicates the density or number of people present in a particular area.

[0107] A "route suggestion method" is a system or algorithm that provides the user with the optimal route to avoid congestion based on the user's current location and congestion status.

[0108] "Feedback data" refers to information including data provided by users and logs of behavior when using the system, and is used to improve system performance and prediction accuracy.

[0109] A "server" is a computer system that centrally manages and processes the collection and analysis of location data, calculation of congestion levels, and collection of feedback data.

[0110] DETAILED DESCRIPTION OF THE INVENTION The following describes an embodiment of the present invention.

[0111] First, the server places various sensors in areas of the event venue or shopping mall to collect location data. The sensors include cameras, Wi-Fi connection counters, BLE beacons, etc. The location data collected by these sensors is sent to the server in real time.

[0112] The server then performs pre-processing on the collected raw data, which involves cleansing, formatting, imputing missing data, and filtering out outliers, using Python programs to analyze and format the data.

[0113] Once preprocessed, the data is analyzed on the server to calculate the congestion level. A machine learning algorithm is used for the analysis. Specifically, the congestion level is calculated based on data such as the number of people in each area and their movement speed. The resulting congestion level is categorized for color-coding, with red indicating congested areas and green indicating less crowded areas.

[0114] Furthermore, the server will suggest optimal routes in real time that allow users to avoid congestion based on the user's location information and congestion data. These suggestions are provided through an application installed on the user's smartphone. Users can easily check and select routes that avoid congestion by referring to the map screen.

[0115] The operations and movement data performed by the user through the application are sent to the server as feedback data, which is used to improve the accuracy of future congestion predictions.

[0116] As an example of this system, a user visiting a shopping mall can use the app to check the congestion status of each store in real time. This user can choose a less crowded area and enjoy shopping comfortably, resulting in a good user experience. In addition, feedback data from users is reflected in the next congestion status prediction, improving the accuracy of the entire system.

[0117] For example, here is a sample prompt:

[0118] "You have been asked to develop a 'Shopping Crowd Avoidance App' that uses data from BLE beacons and Wi-Fi connected counters to grasp the crowd situation in physical stores in real time and provide users with a pleasant shopping experience. This program uses Python and the Requests library to preprocess data and calculate crowd levels, generate a color-coded map, and send it to the server. Please provide the specific program flow and details of the hardware and software you will use."

[0119] This allows users to check congestion conditions in real time and choose an area to spend time in comfortably. The system can also use feedback data to improve the accuracy of its analysis.

[0120] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0121] Step 1:

[0122] The server collects real-time location data from various sensors (cameras, Wi-Fi connection counters, BLE beacons, etc.). The sensors are placed in various areas of the event venue or shopping mall to detect users' current locations and movement patterns. The input is raw data from the sensors, and the output is location data sent to the server.

[0123] Step 2:

[0124] The server performs preprocessing on the acquired location data. This preprocessing includes data cleansing, shaping, missing data completion, and outlier removal. The input is raw data acquired from the sensor, and the output is cleansed location data. A Python program is used for this preprocessing.

[0125] Step 3:

[0126] The server analyzes the preprocessed location data and calculates the congestion level of each area using a machine learning algorithm. The input is the preprocessed location data, and the output is a congestion score corresponding to each area. This analysis process also uses data such as the number of people and their movement speed.

[0127] Step 4:

[0128] The server generates a color-coded map based on the calculated congestion score, which is easy for users to understand visually. The congestion score is categorized, with red indicating congested areas and green indicating vacant areas. The input is the congestion score, and the output is a color-coded congestion map.

[0129] Step 5:

[0130] The user checks the color-coded map provided by the server through an application installed on their smartphone. The user can select a route to avoid congestion while referring to the map screen. The input is the color-coded congestion map, and the output is the route displayed on the user's application screen.

[0131] Step 6:

[0132] The user's location information and operation history within the application are sent to the server as feedback data. The server analyzes this feedback data and uses it to predict congestion levels from the next time onwards. The input is the user's feedback data, and the output is to improve the accuracy of the next analysis.

[0133] Step 7:

[0134] The server maintains the hardware and software environment for all data processing and analysis. Specifically, it uses a high-performance server, database, Python language, and the Requests library. The input is all of the aforementioned data, and the output is stable system operation and highly accurate congestion prediction.

[0135] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0136] The present invention relates to a system that allows users to grasp the congestion situation in real time at large gatherings such as events and festivals, and further recognizes the user's emotions to provide information to help them spend their time comfortably.

[0137] This system first has a means for collecting location data. Specifically, the server acquires location data from sensors (e.g., cameras, Wi-Fi connections, BLE beacons, etc.) installed in each area of ​​the event venue. These sensors detect participants' location information and movement patterns in real time and send the data to the server.

[0138] The collected data is preprocessed by the server, which formats the location data, fills in missing data, and removes outliers. Once the data is properly organized, the server uses a machine learning algorithm to calculate the congestion level of each area. This congestion level is calculated based on data such as the number of people in each area and their movement speed.

[0139] Next, the server generates congestion information based on the analysis results. This congestion information is converted into a format that is easy for users to understand visually (e.g., a color-coded map). Specifically, the congestion score is converted into a color-coded display (e.g., red indicates crowded, green indicates empty) and reflected in the map data. This color-coded map is intended to allow users to easily understand the congestion situation.

[0140] Furthermore, this system incorporates an emotion engine that recognizes the user's emotions. The emotion engine is equipped with a means for analyzing the user's facial expression data and voice data, and detects the user's emotional state based on this. Specifically, the device's camera is used to capture the user's facial expression, and the emotion engine analyzes the facial expression. The device's microphone is also used to collect the user's voice, which is then analyzed. This makes it possible to grasp the user's current emotions, such as stress and satisfaction, in real time.

[0141] Users launch the app on their own device and check the congestion situation and emotion information in real time on a map screen. For example, if a congested area is displayed in red and the user in that area is feeling stressed, the device will use this information to notify the user that the area should be avoided. This allows users to select an area where they can feel comfortable, taking into consideration their emotion and the degree of congestion.

[0142] The device also periodically sends the user's movement data, operation history within the app, and emotional feedback data to the server. Specifically, it periodically collects and sends to the server information on changes in congestion in the area selected by the user, movement history, and emotional changes. This allows the server to analyze the user's behavioral data and emotional data and use it as feedback to improve the accuracy of future analyses.

[0143] For example, suppose a user is attending a fireworks festival and launches the app. The user can check the congestion status of each area on the map and the emotional state of other users in that area in a color-coded format. If a red area indicates congestion and many users in that area are feeling stressed, the user can choose a green, less crowded area to avoid the crowds and spend time there comfortably.

[0144] In this way, the present invention allows users to grasp congestion conditions and emotional information in real time, and allows users to enjoy the event comfortably while taking their emotions into consideration. It also enables event organizers to quickly take measures to alleviate congestion.

[0145] The processing flow will be explained below.

[0146] Step 1: Data collection

[0147] The server collects location data from sensors installed in each area (e.g., cameras, Wi-Fi connections, BLE beacons, etc.). These sensors detect participants' location information and movement patterns in real time and send the data to the server. The server periodically retrieves data from the sensors using API requests and centrally manages it.

[0148] Step 2: Preprocessing the data

[0149] The server converts the acquired raw data into an analyzable format, specifically by performing preprocessing such as filling in missing data, filtering out outliers, and standardizing the data format, so that the location data is stored in a consistent format and subsequent analysis can be performed accurately.

[0150] Step 3: Calculate congestion

[0151] The server uses a machine learning algorithm to calculate the congestion level of each area using the pre-processed data. Specifically, it calculates a congestion score based on the number of people in each area, their movement speed, past data, etc. This allows the congestion status of each area to be quantified in real time.

[0152] Step 4: Generate congestion information

[0153] Based on the calculated congestion score, the server generates congestion information in a form that is visually easy for users to understand. Specifically, the congestion score is converted into a color-coded display (e.g., red indicates congestion, green indicates vacant) and reflected in the map data. This color-coded map is intended to allow users to easily understand the congestion situation.

[0154] Step 5: Collecting sentiment data

[0155] The device uses a camera and microphone to collect facial expression and voice data to recognize the user's emotions. The device's camera captures the user's facial expressions, and the microphone records the user's voice. This emotion data is processed within the device and sent to the emotion engine.

[0156] Step 6: Sentiment Analysis

[0157] The device's emotion engine analyzes the collected facial and voice data to detect the user's emotional state. Specifically, it uses facial and voice analysis algorithms to evaluate the user's stress, satisfaction, and other factors.

[0158] Step 7: Distributing information

[0159] The server delivers the generated congestion information and emotion information obtained from the device to the user's device in real time. Specifically, the information is sent to the user's device using push notifications or API endpoints and is reflected on the map screen. This allows the user to always check the latest congestion status and emotion information.

[0160] Step 8: User interaction

[0161] Users launch the app on their own device and check the congestion status and emotion information on a map screen. Specifically, they look at each color-coded area on the map, select a less congested or more comfortable area, and move to that area. Users can choose an area where they feel comfortable, taking into consideration emotions and congestion levels.

[0162] Step 9: Gather feedback

[0163] The device periodically sends the user's movement data, operation history within the app, and emotional feedback data to the server. Specifically, it periodically collects and sends to the server information on changes in congestion in the area selected by the user, movement history, and emotional changes. This allows the server to analyze the user's behavioral data and emotional data and use it as feedback to improve the accuracy of future analyses.

[0164] Example 2

[0165] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0166] At large-scale events and festivals, there is a need for a system that allows users to grasp the congestion situation in real time and provides appropriate information to help them have a comfortable experience, taking into account their emotional state. However, while conventional systems can grasp the congestion situation, they lack the functionality to analyze the user's emotional state and provide appropriate information based on that. This makes it difficult for users to select the optimal area to sit in, which can lead to a decrease in comfort.

[0167] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0168] In this invention, the server includes means for acquiring location data, means for preprocessing the collected location data, means for analyzing the preprocessed data to calculate the congestion level, means for visually displaying the calculated congestion level and providing it to the user, means for recognizing the user's emotions, means for notifying the user based on the emotion information and congestion information, and means for transmitting the user's movement data and emotion data to the server and providing feedback. This allows the user to obtain information that takes into account the congestion situation and their own emotions in real time and make the optimal choice.

[0169] "Location data" refers to data that indicates the location information and movement patterns of users at an event venue or in a specific area.

[0170] "Preprocessing means" refers to a method for converting collected location data into a normal format, filling in missing data, and excluding outliers.

[0171] The "congestion level" is an index that indicates the degree of congestion in a particular area, calculated based on information such as the density and movement speed of users in that area.

[0172] "Visual display means" refers to a method of reflecting the calculated congestion level in map data in the form of color coding or graphs, etc., so that the user can intuitively understand it.

[0173] "Means for recognizing emotions" refers to technology that analyzes a user's facial expression data and voice data to detect the user's emotional state (e.g., stress, satisfaction).

[0174] The "means for notifying" is a method for sending a notification to the terminal that suggests appropriate actions to the user based on the congestion information and emotion information.

[0175] "Movement data" is data relating to how a user moves through each area within an event venue.

[0176] "Emotion data" refers to data relating to the emotional state detected from the user's facial expression and voice.

[0177] The "means for providing feedback" is a method for periodically sending the user's movement data and emotion data to the server, and is used to improve the system's analysis accuracy.

[0178] The present invention provides a system that allows users to grasp the congestion situation in real time at large gatherings such as events and festivals, and also recognizes the user's emotions to provide information to help them spend their time comfortably. Hereinafter, an embodiment of the present invention will be described in detail.

[0179] System Configuration

[0180] The system consists of the following major hardware and software components:

[0181] Hardware

[0182] Sensors: Camera, Wi-Fi access point, BLE beacon

[0183] Device: The user's smartphone or tablet

[0184] Server: A central server that processes and analyzes data

[0185] software

[0186] Location Data Collection Module

[0187] Data Preprocessing Module

[0188] Machine Learning Algorithms

[0189] Emotion Recognition Engine

[0190] User Interface (UI) Applications

[0191] Processing Description

[0192] Location Data Collection

[0193] The server collects location data from sensors installed at the event venue, such as cameras, Wi-Fi access points, and BLE beacons. Cameras measure the density of people in an area through video analysis, and Wi-Fi access points count the number of connected devices. BLE beacons detect the locations of users in a specific area in real time. For example, if a Wi-Fi access point installed in a venue detects 100 smartphones, the area is deemed highly congested.

[0194] Data Preprocessing

[0195] The collected raw data is preprocessed by the server. This preprocessing includes normalizing location data, filling in missing data, and removing outliers. For example, if the number of Wi-Fi connections is abnormal (such as a sudden increase from 1 to 1,000), that portion is removed or replaced with imputed data.

[0196] Calculating congestion

[0197] Based on the pre-processed data, the server runs machine learning algorithms to calculate the congestion level of each area. The congestion level is calculated using data points such as the number of people, their speed of movement, and density. For example, if an area that normally holds around 30 people detects that 50 people are gathered there, the area is considered "highly congested."

[0198] Generation of congestion information

[0199] The server converts the results of the congestion calculation into a format that can be displayed visually. The congestion information is reflected in the map data as a color-coded map that is easy for users to understand. For example, on a map of a fireworks festival, areas with high congestion levels are displayed in red, and areas with low congestion levels are displayed in green.

[0200] Emotion recognition

[0201] The device uses a camera and microphone to collect facial and voice data, which is then analyzed by an emotion recognition engine. This allows the device to grasp the user's emotional state (e.g., stress, satisfaction) in real time. For example, the camera captures the user's facial expression and checks whether the user is smiling, and the microphone analyzes the user's tone of voice to determine satisfaction or stress.

[0202] Providing congestion information and emotion information

[0203] Users launch the app on their devices and check the congestion situation and emotional information on a map screen in real time. The app displays highly congested areas in red, and if the user's emotional state in that area is stressed, the app will notify them and suggest that they avoid the area. For example, when a user opens the app at a fireworks festival, congested areas will be displayed in red, indicating that the user in that area subjectively feels "congested."

[0204] Data Feedback

[0205] The device periodically sends the user's movement data, operation history within the app, and emotional feedback data to the server. This allows the server to analyze the user's movement patterns and emotional data, and uses them as feedback to improve the accuracy of future analysis. For example, changes in the congestion situation in the area selected by the user, as well as movement history and emotional changes, are collected every hour and sent to the server.

[0206] Examples of specific examples and prompts

[0207] Specific examples

[0208] If a user attends a fireworks festival and launches the app, they might see the following:

[0209] Users can check the congestion status of each area on the map and the emotional state of other users in that area in a color-coded format.

[0210] The red areas are congested, and it is clear that many users are feeling stressed.

[0211] To avoid crowds, users can choose an empty green area and move there.

[0212] Prompt Sentence Examples

[0213] "Please explain a system that uses location data collected from sensors installed in each area of ​​an event venue, and facial expression and voice data collected from cameras and microphones, to allow users to check the congestion situation and emotional information in real time and select an area where they can feel comfortable."

[0214] In this way, the system of the present invention allows users to grasp congestion conditions and emotional information in real time and supports them in selecting a comfortable area, thereby significantly improving the comfort of events.

[0215] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0216] Step 1:

[0217] The server acquires location data from sensors (cameras, Wi-Fi access points, BLE beacons, etc.) at the event venue. The sensors detect users' location information and movement patterns in each area in real time and send the data to the server. Specifically, the cameras analyze the footage to calculate the density of people, and the Wi-Fi access points count the number of connected devices. The input is raw data from the sensors, and the output is the collected location data.

[0218] Step 2:

[0219] The server preprocesses the location data acquired in the previous step. Preprocessing includes normalizing the location data, filling in missing data, and excluding outliers. Specifically, if the number of Wi-Fi connections is abnormal (e.g., a sudden increase from 1 to 1,000), that part is excluded or replaced with filled data. The input is the collected location data, and the output is the preprocessed data.

[0220] Step 3:

[0221] The server runs a machine learning algorithm on the preprocessed data to calculate the congestion level for each area. The congestion level is calculated based on data points such as the number of people in each area and their movement speed. The input is the preprocessed location data, and the output is a congestion score for each area. Specifically, if an area normally contains 30 people and 50 people are detected, the area is considered "highly congested."

[0222] Step 4:

[0223] The server generates congestion information based on the results of the congestion calculation. The generated congestion information is converted into a format that is easy for users to understand visually (e.g., a color-coded map). The input is the congestion score for each area, and the output is congestion information that can be displayed visually. Specifically, areas with high congestion levels are displayed in red on the map of the fireworks display, and areas with low congestion levels are displayed in green.

[0224] Step 5:

[0225] The device uses a camera and microphone to collect facial expression and voice data, which is then analyzed by an emotion recognition engine. The input is the user's facial expression and voice data, and the output is the user's emotional state (e.g., stress, satisfaction). Specifically, the camera captures the user's facial expression, and the microphone analyzes the user's tone of voice.

[0226] Step 6:

[0227] Users launch the app on their device and check the congestion situation and emotion information on a map screen in real time. Highly congested areas are displayed in red, and if the user in that area is feeling stressed, a notification is sent suggesting that the user avoid that area. The input is congestion information and emotion information, and the output is information provided to the user visually and in the form of a notification. Specifically, when a user opens the app at a fireworks festival, congested areas are displayed in red.

[0228] Step 7:

[0229] The device periodically sends the user's movement data, operation history within the app, and emotional feedback data to the server. The server analyzes the sent data and uses it as feedback to improve the accuracy of future analyses. The input is the user's movement data and emotional data, and the output is feedback data. Specifically, changes in congestion in the area selected by the user, movement history, and emotional changes are collected.

[0230] (Application example 2)

[0231] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0232] Physical stores are required to provide a comfortable shopping experience by grasping the congestion status in the store in real time and analyzing the emotional state of customers. Conventional systems can grasp the congestion status, but have difficulty providing information that takes into account the emotional state of customers. Furthermore, they lack the functionality to recommend comfortable areas and times to users by comprehensively considering the congestion level and emotional state. As a result, there are limitations to improving the shopping experience in physical stores. The present invention proposes a system that solves these problems and provides a more comfortable and satisfying shopping experience.

[0233] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for acquiring location data, means for analyzing the acquired location data and calculating a congestion level, means for analyzing and classifying the calculated congestion level and the user's emotional state, and means for recommending a comfortable area and time to the user based on the calculated congestion level and the analyzed emotional state. This makes it possible to recommend the most suitable shopping area and time to the user in real time, taking into consideration the congestion level and the user's emotional state comprehensively, thereby improving the shopping experience at physical stores.

[0234] "Location data" is data that indicates a user's current location and route of travel, and is information collected from sensors and devices.

[0235] "Crowding level" is a numerical value or index that indicates the density or degree of congestion of people in a particular area.

[0236] "Emotional state" is information that indicates the type and strength of the emotion the user is feeling, and is obtained by analyzing facial expressions and voice.

[0237] "Analyzing" means processing data using mathematical and statistical methods to extract meaningful information.

[0238] "Classifying" means separating analyzed data into specific categories or labels.

[0239] "Providing" means displaying the analysis results and recommendation information in a format that is easy for the user to view.

[0240] "Recommending" means suggesting the best options or actions for the user, with the aim of improving user convenience.

[0241] "Color-coded display" is a method of expressing data using colors to make it easier to visually understand congestion levels and emotional states.

[0242] "Preprocessing" refers to removing invalid parts from raw data and converting it into a format suitable for analysis.

[0243] The present invention provides a system for providing a more comfortable shopping experience by understanding the congestion situation and emotional state of customers in a physical store in real time. An embodiment of this system will be described in detail below.

[0244] System configuration

[0245] 1. How to obtain location data

[0246] Sensors (cameras, Bluetooth beacons, Wi-Fi access points, etc.) are used to collect location data for each area within the store. These sensors detect the customer's current location and movement route and send the data to a server.

[0247] 2. Method for calculating congestion level

[0248] The server uses the collected location data to calculate the congestion level of each area in real time, based on the density of customers and their movement speed. This calculation is done using machine learning algorithms.

[0249] 3. Means of analyzing emotional states

[0250] An application installed on the customer's device (smartphone) collects facial expression and voice data, which is then sent to a server, which uses an emotion analysis engine to analyze the customer's emotional state (stress, satisfaction, etc.).

[0251] Sentiment analysis uses machine learning models and image processing libraries such as TensorFlow and OpenCV.

[0252] 4. Information and recommendation means

[0253] The calculated congestion level and analyzed emotional state are provided to customers through an application. The method used is to display the congestion level and emotional state on a map in different colors. For example, areas with high congestion levels are displayed in red, and areas with good emotional states are displayed in green.

[0254] Furthermore, based on this data, the system can recommend areas and time periods that will allow customers to enjoy a comfortable shopping experience, for example, by recommending less crowded areas or areas where customers are in a good emotional state.

[0255] Specific Examples

[0256] When a user visits a shopping mall, they launch a dedicated application on their smartphone. The application receives real-time location data from sensors inside the store and also analyzes the user's facial expressions and voice to detect their emotional state. This data is sent to a server, which then comprehensively assesses the congestion situation and the user's emotional state and recommends an appropriate area for the user.

[0257] For example, if the clothing area is crowded and many customers are feeling stressed, the application will notify the user that the food area is empty and customers are happy, allowing the user to avoid the crowds and choose a place to enjoy shopping comfortably.

[0258] Prompt Sentence Examples

[0259] Below is an example of a prompt sentence to input to the generative AI model.

[0260] "I'm at the mall. Can you tell me how busy it is and what are some recommended places to visit?"

[0261] This system makes it possible to provide customers visiting physical stores with a comfortable shopping experience in real time.

[0262] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0263] Step 1:

[0264] When a user visits a shopping mall, they launch a dedicated application on their device (smartphone). The inputs include the user's location, facial expression, and voice data. This data is collected in real time using the device's camera and microphone. The output is the raw data acquired, which is sent to the server.

[0265] Step 2:

[0266] The server collects location data from sensors (cameras, Bluetooth beacons, Wi-Fi access points, etc.) installed in each area. It receives location information data from the sensors as input. Based on this, the server organizes the location data for each area and performs preprocessing such as filling in missing data and excluding outliers. The preprocessed location data is obtained as output.

[0267] Step 3:

[0268] The server uses machine learning algorithms to calculate congestion levels based on the preprocessed location data. Organized location data is required as input. The server calculates customer density and movement speed, and generates a congestion score for each area. The output is a congestion score for each area.

[0269] Step 4:

[0270] The facial expression and voice data acquired by the device is sent to a server, which analyzes it. Image and voice data are required as input. The server uses machine learning models such as TensorFlow and OpenCV to analyze and classify the emotional state (stress, satisfaction, etc.). The output is a label for the analyzed emotional state.

[0271] Step 5:

[0272] The server generates information based on the calculated congestion score and the analyzed emotional state data and provides it to the user's device. The congestion score and emotional state label are required as input. Based on this data, the server displays the congestion status and emotional state by color-coding them on the map data. As output, the color-coded map and information on recommended areas and time periods are displayed on the user's device.

[0273] Step 6:

[0274] The server periodically accumulates the user's movement history and emotion data to improve the accuracy of future analyses. As input, it receives the user's behavioral data and emotion data. The server analyzes this data and uses it to improve the accuracy of the congestion and emotion prediction model. As output, it obtains an improved congestion and emotion prediction model.

[0275] Specifically, when a user enters a prompt into the application such as "Please tell me the current congestion situation and recommended areas," the server analyzes the data in real time and provides the user with the most appropriate information.

[0276] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0277] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search<url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0278] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0279] [Second embodiment]

[0280] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0281] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0282] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0283] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0284] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0285] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0286] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0287] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0288] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0289] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0290] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0291] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0292] The present invention relates to a system that allows users to grasp the congestion situation in real time at large gatherings such as events and festivals, and provides information to help them spend their time comfortably.

[0293] This system first has a means of collecting location data. Specifically, the server acquires data using sensors (e.g., cameras, Wi-Fi connections, BLE beacons, etc.) installed in each area of ​​the event venue. These sensors collect participants' location information and movement patterns in real time and send them to the server.

[0294] The collected data is preprocessed by the server, which formats the location data, fills in missing data, and removes outliers. Once the data is properly organized, the server uses a machine learning algorithm to calculate the congestion level of each area. This congestion level is calculated based on data such as the number of people in each area and their movement speed.

[0295] Next, the server generates congestion information based on the analysis results. This congestion information is converted into a form that is easy for users to understand visually (e.g., a color-coded map). Specifically, the congestion score is divided into categories for color-coding, for example, congested areas are displayed in red and uncrowded areas in green.

[0296] Users can check congestion information in real time through the app on their own devices (smartphones, tablets, etc.). The app has a map screen that displays the congestion status of each area in a color-coded format. This allows users to select an area where they can avoid crowds and spend their time comfortably.

[0297] The system also includes a means for collecting user feedback data. The system periodically sends the user's movement data and operation history within the app to a server, and uses this feedback data to improve the accuracy of the analysis. Specifically, the system tracks changes in the congestion level in the area selected by the user, and reflects this data in future congestion predictions.

[0298] For example, suppose a user is attending a fireworks festival and launches the app. The user can check the congestion status of each area on a map in a color-coded format. Since red areas indicate congestion, the user will choose a green, less crowded area and spend time there. After the event ends, the user's movement data is sent to the server, which will contribute to improving the analysis accuracy for the next event.

[0299] In this way, the present invention allows users to grasp the congestion situation in real time and enjoy the event comfortably, and also enables event organizers to quickly take measures to alleviate congestion.

[0300] The processing flow will be explained below.

[0301] Step 1: Data collection

[0302] The server collects location data from sensors installed in each area (e.g., cameras, Wi-Fi connections, BLE beacons, etc.). These sensors detect participants' location information and movement patterns in real time and send the data to the server. Specifically, the server periodically obtains data from the sensors using API requests and centrally manages the collected data.

[0303] Step 2: Preprocessing the data

[0304] The server converts the acquired raw data into an analyzable format, specifically by performing preprocessing such as filling in missing data, filtering out outliers, and standardizing the data format, so that the location data is stored in a consistent format and subsequent analysis can be performed accurately.

[0305] Step 3: Calculate congestion

[0306] The server uses a machine learning algorithm to calculate the congestion level of each area using the pre-processed data. Specifically, it calculates a congestion score based on the number of people in each area, their movement speed, past data, etc. This allows the congestion status of each area to be quantified in real time.

[0307] Step 4: Generate congestion information

[0308] Based on the calculated congestion score, the server generates congestion information in a form that is visually easy for users to understand. Specifically, the congestion score is converted into a color-coded display (e.g., red indicates congestion, green indicates vacant) and reflected in the map data. This color-coded map is intended to allow users to easily understand the congestion situation.

[0309] Step 5: Distributing information

[0310] The server distributes the generated congestion information to the user's device in real time. Specifically, it sends the information to the user's device using push notifications or API endpoints. This allows the user to always check the latest congestion status.

[0311] Step 6: User interaction

[0312] Users can launch the app on their own devices and check the congestion situation on the map screen. Specifically, they can look at each color-coded area on the map, select an area that is not crowded, and move to that area. This allows users to avoid crowds and enjoy the event in comfort.

[0313] Step 7: Gather feedback

[0314] The device periodically sends the user's movement data and operation history within the app to the server. Specifically, it periodically collects changes in congestion in the area selected by the user and movement history and sends them to the server. This allows the server to analyze the user's behavioral data and use it as feedback to improve the accuracy of future analysis.

[0315] Example 1

[0316] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0317] At large gatherings such as traditional events and festivals, participants have had limited means of understanding congestion levels in advance, making it difficult for them to enjoy a comfortable stay. Furthermore, because collected data is not analyzed or displayed in real time, it is not possible to provide appropriate information to participants in a timely manner. Furthermore, there is a lack of means to collect and utilize user feedback data, making it difficult to improve the accuracy of congestion predictions.

[0318] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0319] In this invention, the server includes a means for acquiring location data, a means for preprocessing the acquired location data to reshape the location information data, fill in missing data, and remove outliers, a means for calculating congestion levels using a machine learning algorithm based on the preprocessed data, a means for categorizing the calculated congestion level scores and generating a visually easy-to-understand color-coded map, a means for providing the generated congestion information to a user terminal, and a means for collecting user feedback data and using it to improve the accuracy of congestion level predictions. This allows users to understand congestion situations in real time and select appropriate areas for a comfortable stay. Furthermore, utilizing user feedback data can also contribute to improving the accuracy of future congestion predictions.

[0320] "Location data" refers to data that indicates the location of people or objects within a specific area, such as an event venue.

[0321] "Preprocessing" is the process of formatting raw location data, filling in missing data, removing outliers, etc., to make the data suitable for analysis.

[0322] A "machine learning algorithm" is an algorithm that analyzes data to find patterns and rules and uses them for future predictions, classification, etc.

[0323] "Crowding level" is a numerical representation of the density of people and objects in a specific area, and is an indicator of the congestion level of the area.

[0324] A "color-coded map" is a map that displays areas in different colors to visually indicate the degree of congestion.

[0325] A "user terminal" is an electronic device used by a user, such as a smartphone or tablet.

[0326] "Feedback data" is data collected to help improve the system, such as user behavior data and in-app operation history.

[0327] The present invention relates to a system that allows users to grasp the congestion situation in real time at large gatherings such as events and festivals, and provides information to help them spend their time comfortably. Specifically, the system includes the following means.

[0328] First, the server collects location data of participants using sensors installed at the event venue (e.g., cameras, Wi-Fi connection count, BLE beacons, etc.). This location data is sent to the server in real time. For example, a Wi-Fi connection count sensor can estimate the congestion level of the area based on the number of connected devices.

[0329] Next, the server preprocesses the collected raw location data. Preprocessing steps include formatting the location data, filling in missing data, and filtering out outliers. This organizes the data into a form suitable for analysis. For example, data formatting can be performed using the Python Pandas library.

[0330] After the preprocessing is complete, the server applies a machine learning algorithm (e.g., K-means clustering using the Scikit-learn library) to calculate the congestion level of each area, taking into account multiple data points such as the number of people in each area and their movement speed.

[0331] Once the congestion level is calculated, the server generates congestion information based on the analysis results. This congestion information is provided as a visually easy-to-understand color-coded map (using Mapbox or OpenStreetMap, for example). The congestion level score is divided into categories, for example, congested areas are displayed in red and vacant areas in green.

[0332] Users can check congestion information in real time through a dedicated app on their smartphones, tablets, or other devices. The app has a map screen that shows the congestion status of each area in a color-coded format. This allows users to avoid crowds and select an area where they can spend their time comfortably.

[0333] After the event ends, the server will collect user movement data and operation history within the app, and use this data as feedback to improve the accuracy of future congestion predictions. A Python data analysis library (e.g., Pandas) can be used to analyze this feedback data.

[0334] As a concrete example, consider the case where a user attends a fireworks display. The user can launch a dedicated app and check the congestion status of each area on a map in a color-coded format. Since red areas indicate congestion, the user will choose a green, less crowded area and spend time there. After the event ends, the user's movement data is sent to the server, which will contribute to improving the accuracy of analysis for the next event.

[0335] Examples of prompts for generative AI models include:

[0336] "Please create a program for a system that will allow users to grasp the congestion situation in real time at large gatherings such as events and festivals, and provide information to help them have a comfortable time. This program will include a process to preprocess location data collected by sensors, calculate the degree of congestion, and display it on a color-coded map. Please also include a function to utilize user feedback data to improve the accuracy of the analysis."

[0337] As described above, the present invention is a system that allows users to understand congestion situations in real time and enjoy events comfortably, and also contributes to improving the accuracy of congestion predictions by using collected feedback data.

[0338] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0339] Step 1:

[0340] The server collects location data using sensors (e.g., cameras, Wi-Fi connection count, BLE beacons, etc.) installed at the event venue. The data collected by the sensors is sent to the server in real time. This input data includes the location information and movement patterns of each participant. For example, a Wi-Fi connection count sensor can estimate the congestion level of the area from the number of connected devices. The output is raw location data.

[0341] Step 2:

[0342] The server pre-processes the collected raw location data. The raw location data is used as input data. The pre-processing includes the following specific operations:

[0343] Data Transformation: Convert location data into a unified format.

[0344] Missing data completion: If a sensor is temporarily out of service, the data from that time is completed using data from nearby sensors.

[0345] Outlier exclusion: If some data is detected as abnormal, it will be excluded.

[0346] The result of this preprocessing is clean location data suitable for analysis.

[0347] Step 3:

[0348] The server applies machine learning algorithms to the preprocessed data to calculate the congestion level of each area. The preprocessed location data is used as input data. The server analyzes people's density and movement patterns by using K-means clustering, for example, using Python's Scikit-learn library. This outputs the congestion level of each area as a numerical value.

[0349] Step 4:

[0350] The server generates congestion information based on the analysis results. Numerical data indicating the degree of congestion is used as input data. Specifically, the congestion scores are categorized and converted into a visually easy-to-understand color-coded map. For example, congested areas are displayed in red and vacant areas in green. Tools such as Mapbox and OpenStreetMap are used for this task. A visually displayable color-coded map is generated as output.

[0351] Step 5:

[0352] Users use devices such as smartphones and tablets to check congestion information in real time through a dedicated app. A color-coded map delivered from a server is used as input data. Specifically, the app is launched and the map screen is displayed, where the congestion status of each area is shown in color-coded format. Based on this, users can select areas to avoid congestion and spend time in those areas. The output is a congestion status map that the user can view.

[0353] Step 6:

[0354] The server collects user movement data and in-app operation history after the event ends or periodically. The user's location information and operation history are used as input data. Specifically, it tracks the user's movement patterns and the congestion status of the selected area and stores this in a database. This feedback data is used for analysis to improve the accuracy of future congestion predictions. For example, data analysis is performed using Python's Pandas library. The output makes it possible to improve the congestion prediction algorithm.

[0355] (Application example 1)

[0356] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0357] Large gatherings and events, especially in brick-and-mortar locations like shopping malls, present challenges for visitors, making it difficult to understand crowding conditions and ensure a comfortable experience. Real-time information on how to avoid crowded areas is lacking, often resulting in a poor visitor experience. Additionally, there is insufficient collection and utilization of feedback data to improve crowd prediction accuracy.

[0358] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0359] In this invention, the server includes a means for acquiring location data, a means for analyzing the acquired location data and calculating the congestion level, a means for providing the calculated congestion level to the user, a means for suggesting a route to avoid congestion based on the user's location information, and a means for collecting user feedback data and improving the accuracy of the analysis. This not only enables visitors to check the congestion situation in real time and select a comfortable area, but also makes it possible to suggest the optimal route to avoid congestion. Furthermore, by collecting and analyzing user feedback, the accuracy of congestion level prediction can be improved.

[0360] "Location data" is numerical data that indicates the current location of an individual user or group at a specific location, such as an event or shopping mall.

[0361] "Analysis" is the process of processing collected raw data and converting it into meaningful information.

[0362] "Crowding" is an indicator that indicates the density or number of people present in a particular area.

[0363] A "route suggestion method" is a system or algorithm that provides the user with the optimal route to avoid congestion based on the user's current location and congestion status.

[0364] "Feedback data" refers to information including data provided by users and logs of behavior when using the system, and is used to improve system performance and prediction accuracy.

[0365] A "server" is a computer system that centrally manages and processes the collection and analysis of location data, calculation of congestion levels, and collection of feedback data.

[0366] DETAILED DESCRIPTION OF THE INVENTION The following describes an embodiment of the present invention.

[0367] First, the server places various sensors in areas of the event venue or shopping mall to collect location data. The sensors include cameras, Wi-Fi connection counters, BLE beacons, etc. The location data collected by these sensors is sent to the server in real time.

[0368] The server then performs pre-processing on the collected raw data, which involves cleansing, formatting, imputing missing data, and filtering out outliers, using Python programs to analyze and format the data.

[0369] Once preprocessed, the data is analyzed on the server to calculate the congestion level. A machine learning algorithm is used for the analysis. Specifically, the congestion level is calculated based on data such as the number of people in each area and their movement speed. The resulting congestion level is categorized for color-coding, with red indicating congested areas and green indicating less crowded areas.

[0370] Furthermore, the server will suggest optimal routes in real time that allow users to avoid congestion based on the user's location information and congestion data. These suggestions are provided through an application installed on the user's smartphone. Users can easily check and select routes that avoid congestion by referring to the map screen.

[0371] The operations and movement data performed by the user through the application are sent to the server as feedback data, which is used to improve the accuracy of future congestion predictions.

[0372] As an example of this system, a user visiting a shopping mall can use the app to check the congestion status of each store in real time. This user can choose a less crowded area and enjoy shopping comfortably, resulting in a good user experience. In addition, feedback data from users is reflected in the next congestion status prediction, improving the accuracy of the entire system.

[0373] For example, here is a sample prompt:

[0374] "You have been asked to develop a 'Shopping Crowd Avoidance App' that uses data from BLE beacons and Wi-Fi connected counters to grasp the crowd situation in physical stores in real time and provide users with a pleasant shopping experience. This program uses Python and the Requests library to preprocess data and calculate crowd levels, generate a color-coded map, and send it to the server. Please provide the specific program flow and details of the hardware and software you will use."

[0375] This allows users to check congestion conditions in real time and choose an area to spend time in comfortably. The system can also use feedback data to improve the accuracy of its analysis.

[0376] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0377] Step 1:

[0378] The server collects real-time location data from various sensors (cameras, Wi-Fi connection counters, BLE beacons, etc.). The sensors are placed in various areas of the event venue or shopping mall to detect users' current locations and movement patterns. The input is raw data from the sensors, and the output is location data sent to the server.

[0379] Step 2:

[0380] The server performs preprocessing on the acquired location data. This preprocessing includes data cleansing, shaping, missing data completion, and outlier removal. The input is raw data acquired from the sensor, and the output is cleansed location data. A Python program is used for this preprocessing.

[0381] Step 3:

[0382] The server analyzes the preprocessed location data and calculates the congestion level of each area using a machine learning algorithm. The input is the preprocessed location data, and the output is a congestion score corresponding to each area. This analysis process also uses data such as the number of people and their movement speed.

[0383] Step 4:

[0384] The server generates a color-coded map based on the calculated congestion score, which is easy for users to understand visually. The congestion score is categorized, with red indicating congested areas and green indicating vacant areas. The input is the congestion score, and the output is a color-coded congestion map.

[0385] Step 5:

[0386] The user checks the color-coded map provided by the server through an application installed on their smartphone. The user can select a route to avoid congestion while referring to the map screen. The input is the color-coded congestion map, and the output is the route displayed on the user's application screen.

[0387] Step 6:

[0388] The user's location information and operation history within the application are sent to the server as feedback data. The server analyzes this feedback data and uses it to predict congestion levels from the next time onwards. The input is the user's feedback data, and the output is to improve the accuracy of the next analysis.

[0389] Step 7:

[0390] The server maintains the hardware and software environment for all data processing and analysis. Specifically, it uses a high-performance server, database, Python language, and the Requests library. The input is all of the aforementioned data, and the output is stable system operation and highly accurate congestion prediction.

[0391] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0392] The present invention relates to a system that allows users to grasp the congestion situation in real time at large gatherings such as events and festivals, and further recognizes the user's emotions to provide information to help them spend their time comfortably.

[0393] This system first has a means for collecting location data. Specifically, the server acquires location data from sensors (e.g., cameras, Wi-Fi connections, BLE beacons, etc.) installed in each area of ​​the event venue. These sensors detect participants' location information and movement patterns in real time and send the data to the server.

[0394] The collected data is preprocessed by the server, which formats the location data, fills in missing data, and removes outliers. Once the data is properly organized, the server uses a machine learning algorithm to calculate the congestion level of each area. This congestion level is calculated based on data such as the number of people in each area and their movement speed.

[0395] Next, the server generates congestion information based on the analysis results. This congestion information is converted into a format that is easy for users to understand visually (e.g., a color-coded map). Specifically, the congestion score is converted into a color-coded display (e.g., red indicates crowded, green indicates empty) and reflected in the map data. This color-coded map is intended to allow users to easily understand the congestion situation.

[0396] Furthermore, this system incorporates an emotion engine that recognizes the user's emotions. The emotion engine is equipped with a means for analyzing the user's facial expression data and voice data, and detects the user's emotional state based on this. Specifically, the device's camera is used to capture the user's facial expression, and the emotion engine analyzes the facial expression. The device's microphone is also used to collect the user's voice, which is then analyzed. This makes it possible to grasp the user's current emotions, such as stress and satisfaction, in real time.

[0397] Users launch the app on their own device and check the congestion situation and emotion information in real time on a map screen. For example, if a congested area is displayed in red and the user in that area is feeling stressed, the device will use this information to notify the user that the area should be avoided. This allows users to select an area where they can feel comfortable, taking into consideration their emotion and the degree of congestion.

[0398] The device also periodically sends the user's movement data, operation history within the app, and emotional feedback data to the server. Specifically, it periodically collects and sends to the server information on changes in congestion in the area selected by the user, movement history, and emotional changes. This allows the server to analyze the user's behavioral data and emotional data and use it as feedback to improve the accuracy of future analyses.

[0399] For example, suppose a user is attending a fireworks festival and launches the app. The user can check the congestion status of each area on the map and the emotional state of other users in that area in a color-coded format. If a red area indicates congestion and many users in that area are feeling stressed, the user can choose a green, less crowded area to avoid the crowds and spend time there comfortably.

[0400] In this way, the present invention allows users to grasp congestion conditions and emotional information in real time, and allows users to enjoy the event comfortably while taking their emotions into consideration. It also enables event organizers to quickly take measures to alleviate congestion.

[0401] The processing flow will be explained below.

[0402] Step 1: Data collection

[0403] The server collects location data from sensors installed in each area (e.g., cameras, Wi-Fi connections, BLE beacons, etc.). These sensors detect participants' location information and movement patterns in real time and send the data to the server. The server periodically retrieves data from the sensors using API requests and centrally manages it.

[0404] Step 2: Preprocessing the data

[0405] The server converts the acquired raw data into an analyzable format, specifically by performing preprocessing such as filling in missing data, filtering out outliers, and standardizing the data format, so that the location data is stored in a consistent format and subsequent analysis can be performed accurately.

[0406] Step 3: Calculate congestion

[0407] The server uses a machine learning algorithm to calculate the congestion level of each area using the pre-processed data. Specifically, it calculates a congestion score based on the number of people in each area, their movement speed, past data, etc. This allows the congestion status of each area to be quantified in real time.

[0408] Step 4: Generate congestion information

[0409] Based on the calculated congestion score, the server generates congestion information in a form that is visually easy for users to understand. Specifically, the congestion score is converted into a color-coded display (e.g., red indicates congestion, green indicates vacant) and reflected in the map data. This color-coded map is intended to allow users to easily understand the congestion situation.

[0410] Step 5: Collecting sentiment data

[0411] The device uses a camera and microphone to collect facial expression and voice data to recognize the user's emotions. The device's camera captures the user's facial expressions, and the microphone records the user's voice. This emotion data is processed within the device and sent to the emotion engine.

[0412] Step 6: Sentiment Analysis

[0413] The device's emotion engine analyzes the collected facial and voice data to detect the user's emotional state. Specifically, it uses facial and voice analysis algorithms to evaluate the user's stress, satisfaction, and other factors.

[0414] Step 7: Distributing information

[0415] The server delivers the generated congestion information and emotion information obtained from the device to the user's device in real time. Specifically, the information is sent to the user's device using push notifications or API endpoints and is reflected on the map screen. This allows the user to always check the latest congestion status and emotion information.

[0416] Step 8: User interaction

[0417] Users launch the app on their own device and check the congestion status and emotion information on a map screen. Specifically, they look at each color-coded area on the map, select a less congested or more comfortable area, and move to that area. Users can choose an area where they feel comfortable, taking into consideration emotions and congestion levels.

[0418] Step 9: Gather feedback

[0419] The device periodically sends the user's movement data, operation history within the app, and emotional feedback data to the server. Specifically, it periodically collects and sends to the server information on changes in congestion in the area selected by the user, movement history, and emotional changes. This allows the server to analyze the user's behavioral data and emotional data and use it as feedback to improve the accuracy of future analyses.

[0420] Example 2

[0421] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0422] At large-scale events and festivals, there is a need for a system that allows users to grasp the congestion situation in real time and provides appropriate information to help them have a comfortable experience, taking into account their emotional state. However, while conventional systems can grasp the congestion situation, they lack the functionality to analyze the user's emotional state and provide appropriate information based on that. This makes it difficult for users to select the optimal area to sit in, which can lead to a decrease in comfort.

[0423] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0424] In this invention, the server includes means for acquiring location data, means for preprocessing the collected location data, means for analyzing the preprocessed data to calculate the congestion level, means for visually displaying the calculated congestion level and providing it to the user, means for recognizing the user's emotions, means for notifying the user based on the emotion information and congestion information, and means for transmitting the user's movement data and emotion data to the server and providing feedback. This allows the user to obtain information that takes into account the congestion situation and their own emotions in real time and make the optimal choice.

[0425] "Location data" refers to data that indicates the location information and movement patterns of users at an event venue or in a specific area.

[0426] "Preprocessing means" refers to a method for converting collected location data into a normal format, filling in missing data, and excluding outliers.

[0427] The "congestion level" is an index that indicates the degree of congestion in a particular area, calculated based on information such as the density and movement speed of users in that area.

[0428] "Visual display means" refers to a method of reflecting the calculated congestion level in map data in the form of color coding or graphs, etc., so that the user can intuitively understand it.

[0429] "Means for recognizing emotions" refers to technology that analyzes a user's facial expression data and voice data to detect the user's emotional state (e.g., stress, satisfaction).

[0430] The "means for notifying" is a method for sending a notification to the terminal that suggests appropriate actions to the user based on the congestion information and emotion information.

[0431] "Movement data" is data relating to how a user moves through each area within an event venue.

[0432] "Emotion data" refers to data relating to the emotional state detected from the user's facial expression and voice.

[0433] The "means for providing feedback" is a method for periodically sending the user's movement data and emotion data to the server, and is used to improve the system's analysis accuracy.

[0434] The present invention provides a system that allows users to grasp the congestion situation in real time at large gatherings such as events and festivals, and also recognizes the user's emotions to provide information to help them spend their time comfortably. Hereinafter, an embodiment of the present invention will be described in detail.

[0435] System Configuration

[0436] The system consists of the following major hardware and software components:

[0437] Hardware

[0438] Sensors: Camera, Wi-Fi access point, BLE beacon

[0439] Device: The user's smartphone or tablet

[0440] Server: A central server that processes and analyzes data

[0441] software

[0442] Location Data Collection Module

[0443] Data Preprocessing Module

[0444] Machine Learning Algorithms

[0445] Emotion Recognition Engine

[0446] User Interface (UI) Applications

[0447] Processing Description

[0448] Location Data Collection

[0449] The server collects location data from sensors installed at the event venue, such as cameras, Wi-Fi access points, and BLE beacons. Cameras measure the density of people in an area through video analysis, and Wi-Fi access points count the number of connected devices. BLE beacons detect the locations of users in a specific area in real time. For example, if a Wi-Fi access point installed in a venue detects 100 smartphones, the area is deemed highly congested.

[0450] Data Preprocessing

[0451] The collected raw data is preprocessed by the server. This preprocessing includes normalizing location data, filling in missing data, and removing outliers. For example, if the number of Wi-Fi connections is abnormal (such as a sudden increase from 1 to 1,000), that portion is removed or replaced with imputed data.

[0452] Calculating congestion

[0453] Based on the pre-processed data, the server runs machine learning algorithms to calculate the congestion level of each area. The congestion level is calculated using data points such as the number of people, their speed of movement, and density. For example, if an area that normally holds around 30 people detects that 50 people are gathered there, the area is considered "highly congested."

[0454] Generation of congestion information

[0455] The server converts the results of the congestion calculation into a format that can be displayed visually. The congestion information is reflected in the map data as a color-coded map that is easy for users to understand. For example, on a map of a fireworks festival, areas with high congestion levels are displayed in red, and areas with low congestion levels are displayed in green.

[0456] Emotion recognition

[0457] The device uses a camera and microphone to collect facial and voice data, which is then analyzed by an emotion recognition engine. This allows the device to grasp the user's emotional state (e.g., stress, satisfaction) in real time. For example, the camera captures the user's facial expression and checks whether the user is smiling, and the microphone analyzes the user's tone of voice to determine satisfaction or stress.

[0458] Providing congestion information and emotion information

[0459] Users launch the app on their devices and check the congestion situation and emotional information on a map screen in real time. The app displays highly congested areas in red, and if the user's emotional state in that area is stressed, the app will notify them and suggest that they avoid the area. For example, when a user opens the app at a fireworks festival, congested areas will be displayed in red, indicating that the user in that area subjectively feels "congested."

[0460] Data Feedback

[0461] The device periodically sends the user's movement data, operation history within the app, and emotional feedback data to the server. This allows the server to analyze the user's movement patterns and emotional data, and uses them as feedback to improve the accuracy of future analysis. For example, changes in the congestion situation in the area selected by the user, as well as movement history and emotional changes, are collected every hour and sent to the server.

[0462] Examples of specific examples and prompts

[0463] Specific examples

[0464] If a user attends a fireworks festival and launches the app, they might see the following:

[0465] Users can check the congestion status of each area on the map and the emotional state of other users in that area in a color-coded format.

[0466] The red areas are congested, and it is clear that many users are feeling stressed.

[0467] To avoid crowds, users can choose an empty green area and move there.

[0468] Prompt Sentence Examples

[0469] "Please explain a system that uses location data collected from sensors installed in each area of ​​an event venue, and facial expression and voice data collected from cameras and microphones, to allow users to check the congestion situation and emotional information in real time and select an area where they can feel comfortable."

[0470] In this way, the system of the present invention allows users to grasp congestion conditions and emotional information in real time and supports them in selecting a comfortable area, thereby significantly improving the comfort of events.

[0471] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0472] Step 1:

[0473] The server acquires location data from sensors (cameras, Wi-Fi access points, BLE beacons, etc.) at the event venue. The sensors detect users' location information and movement patterns in each area in real time and send the data to the server. Specifically, the cameras analyze the footage to calculate the density of people, and the Wi-Fi access points count the number of connected devices. The input is raw data from the sensors, and the output is the collected location data.

[0474] Step 2:

[0475] The server preprocesses the location data acquired in the previous step. Preprocessing includes normalizing the location data, filling in missing data, and excluding outliers. Specifically, if the number of Wi-Fi connections is abnormal (e.g., a sudden increase from 1 to 1,000), that part is excluded or replaced with filled data. The input is the collected location data, and the output is the preprocessed data.

[0476] Step 3:

[0477] The server runs a machine learning algorithm on the preprocessed data to calculate the congestion level for each area. The congestion level is calculated based on data points such as the number of people in each area and their movement speed. The input is the preprocessed location data, and the output is a congestion score for each area. Specifically, if an area normally contains 30 people and 50 people are detected, the area is considered "highly congested."

[0478] Step 4:

[0479] The server generates congestion information based on the results of the congestion calculation. The generated congestion information is converted into a format that is easy for users to understand visually (e.g., a color-coded map). The input is the congestion score for each area, and the output is congestion information that can be displayed visually. Specifically, areas with high congestion levels are displayed in red on the map of the fireworks display, and areas with low congestion levels are displayed in green.

[0480] Step 5:

[0481] The device uses a camera and microphone to collect facial expression and voice data, which is then analyzed by an emotion recognition engine. The input is the user's facial expression and voice data, and the output is the user's emotional state (e.g., stress, satisfaction). Specifically, the camera captures the user's facial expression, and the microphone analyzes the user's tone of voice.

[0482] Step 6:

[0483] Users launch the app on their device and check the congestion situation and emotion information on a map screen in real time. Highly congested areas are displayed in red, and if the user in that area is feeling stressed, a notification is sent suggesting that the user avoid that area. The input is congestion information and emotion information, and the output is information provided to the user visually and in the form of a notification. Specifically, when a user opens the app at a fireworks festival, congested areas are displayed in red.

[0484] Step 7:

[0485] The device periodically sends the user's movement data, operation history within the app, and emotional feedback data to the server. The server analyzes the sent data and uses it as feedback to improve the accuracy of future analyses. The input is the user's movement data and emotional data, and the output is feedback data. Specifically, changes in congestion in the area selected by the user, movement history, and emotional changes are collected.

[0486] (Application example 2)

[0487] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0488] Physical stores are required to provide a comfortable shopping experience by grasping the congestion status in the store in real time and analyzing the emotional state of customers. Conventional systems can grasp the congestion status, but have difficulty providing information that takes into account the emotional state of customers. Furthermore, they lack the functionality to recommend comfortable areas and times to users by comprehensively considering the congestion level and emotional state. As a result, there are limitations to improving the shopping experience in physical stores. The present invention proposes a system that solves these problems and provides a more comfortable and satisfying shopping experience.

[0489] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for acquiring location data, means for analyzing the acquired location data and calculating a congestion level, means for analyzing and classifying the calculated congestion level and the user's emotional state, and means for recommending a comfortable area and time to the user based on the calculated congestion level and the analyzed emotional state. This makes it possible to recommend the most suitable shopping area and time to the user in real time, taking into consideration the congestion level and the user's emotional state comprehensively, thereby improving the shopping experience at physical stores.

[0490] "Location data" is data that indicates a user's current location and route of travel, and is information collected from sensors and devices.

[0491] "Crowding level" is a numerical value or index that indicates the density or degree of congestion of people in a particular area.

[0492] "Emotional state" is information that indicates the type and strength of the emotion the user is feeling, and is obtained by analyzing facial expressions and voice.

[0493] "Analyzing" means processing data using mathematical and statistical methods to extract meaningful information.

[0494] "Classifying" means separating analyzed data into specific categories or labels.

[0495] "Providing" means displaying the analysis results and recommendation information in a format that is easy for the user to view.

[0496] "Recommending" means suggesting the best options or actions for the user, with the aim of improving user convenience.

[0497] "Color-coded display" is a method of expressing data using colors to make it easier to visually understand congestion levels and emotional states.

[0498] "Preprocessing" refers to removing invalid parts from raw data and converting it into a format suitable for analysis.

[0499] The present invention provides a system for providing a more comfortable shopping experience by understanding the congestion situation and emotional state of customers in a physical store in real time. An embodiment of this system will be described in detail below.

[0500] System configuration

[0501] 1. How to obtain location data

[0502] Sensors (cameras, Bluetooth beacons, Wi-Fi access points, etc.) are used to collect location data for each area within the store. These sensors detect the customer's current location and movement route and send the data to a server.

[0503] 2. Method for calculating congestion level

[0504] The server uses the collected location data to calculate the congestion level of each area in real time, based on the density of customers and their movement speed. This calculation is done using machine learning algorithms.

[0505] 3. Means of analyzing emotional states

[0506] An application installed on the customer's device (smartphone) collects facial expression and voice data, which is then sent to a server, which uses an emotion analysis engine to analyze the customer's emotional state (stress, satisfaction, etc.).

[0507] Sentiment analysis uses machine learning models and image processing libraries such as TensorFlow and OpenCV.

[0508] 4. Information and recommendation means

[0509] The calculated congestion level and analyzed emotional state are provided to customers through an application. The method used is to display the congestion level and emotional state on a map in different colors. For example, areas with high congestion levels are displayed in red, and areas with good emotional states are displayed in green.

[0510] Furthermore, based on this data, the system can recommend areas and time periods that will allow customers to enjoy a comfortable shopping experience, for example, by recommending less crowded areas or areas where customers are in a good emotional state.

[0511] Specific Examples

[0512] When a user visits a shopping mall, they launch a dedicated application on their smartphone. The application receives real-time location data from sensors inside the store and also analyzes the user's facial expressions and voice to detect their emotional state. This data is sent to a server, which then comprehensively assesses the congestion situation and the user's emotional state and recommends an appropriate area for the user.

[0513] For example, if the clothing area is crowded and many customers are feeling stressed, the application will notify the user that the food area is empty and customers are happy, allowing the user to avoid the crowds and choose a place to enjoy shopping comfortably.

[0514] Prompt Sentence Examples

[0515] Below is an example of a prompt sentence to input to the generative AI model.

[0516] "I'm at the mall. Can you tell me how busy it is and what are some recommended places to visit?"

[0517] This system makes it possible to provide customers visiting physical stores with a comfortable shopping experience in real time.

[0518] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0519] Step 1:

[0520] When a user visits a shopping mall, they launch a dedicated application on their device (smartphone). The inputs include the user's location, facial expression, and voice data. This data is collected in real time using the device's camera and microphone. The output is the raw data acquired, which is sent to the server.

[0521] Step 2:

[0522] The server collects location data from sensors (cameras, Bluetooth beacons, Wi-Fi access points, etc.) installed in each area. It receives location information data from the sensors as input. Based on this, the server organizes the location data for each area and performs preprocessing such as filling in missing data and excluding outliers. The preprocessed location data is obtained as output.

[0523] Step 3:

[0524] The server uses machine learning algorithms to calculate congestion levels based on the preprocessed location data. Organized location data is required as input. The server calculates customer density and movement speed, and generates a congestion score for each area. The output is a congestion score for each area.

[0525] Step 4:

[0526] The facial expression and voice data acquired by the device is sent to a server, which analyzes it. Image and voice data are required as input. The server uses machine learning models such as TensorFlow and OpenCV to analyze and classify the emotional state (stress, satisfaction, etc.). The output is a label for the analyzed emotional state.

[0527] Step 5:

[0528] The server generates information based on the calculated congestion score and the analyzed emotional state data and provides it to the user's device. The congestion score and emotional state label are required as input. Based on this data, the server displays the congestion status and emotional state by color-coding them on the map data. As output, the color-coded map and information on recommended areas and time periods are displayed on the user's device.

[0529] Step 6:

[0530] The server periodically accumulates the user's movement history and emotion data to improve the accuracy of future analyses. As input, it receives the user's behavioral data and emotion data. The server analyzes this data and uses it to improve the accuracy of the congestion and emotion prediction model. As output, it obtains an improved congestion and emotion prediction model.

[0531] Specifically, when a user enters a prompt into the application such as "Please tell me the current congestion situation and recommended areas," the server analyzes the data in real time and provides the user with the most appropriate information.

[0532] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0533] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0534] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0535] [Third embodiment]

[0536] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0537] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0538] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0539] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0540] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0541] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0542] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0543] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0544] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0545] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0546] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0547] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0548] The present invention relates to a system that allows users to grasp the congestion situation in real time at large gatherings such as events and festivals, and provides information to help them spend their time comfortably.

[0549] This system first has a means of collecting location data. Specifically, the server acquires data using sensors (e.g., cameras, Wi-Fi connections, BLE beacons, etc.) installed in each area of ​​the event venue. These sensors collect participants' location information and movement patterns in real time and send them to the server.

[0550] The collected data is preprocessed by the server, which formats the location data, fills in missing data, and removes outliers. Once the data is properly organized, the server uses a machine learning algorithm to calculate the congestion level of each area. This congestion level is calculated based on data such as the number of people in each area and their movement speed.

[0551] Next, the server generates congestion information based on the analysis results. This congestion information is converted into a form that is easy for users to understand visually (e.g., a color-coded map). Specifically, the congestion score is divided into categories for color-coding, for example, congested areas are displayed in red and uncrowded areas in green.

[0552] Users can check congestion information in real time through the app on their own devices (smartphones, tablets, etc.). The app has a map screen that displays the congestion status of each area in a color-coded format. This allows users to select an area where they can avoid crowds and spend their time comfortably.

[0553] The system also includes a means for collecting user feedback data. The system periodically sends the user's movement data and operation history within the app to a server, and uses this feedback data to improve the accuracy of the analysis. Specifically, the system tracks changes in the congestion level in the area selected by the user, and reflects this data in future congestion predictions.

[0554] For example, suppose a user is attending a fireworks festival and launches the app. The user can check the congestion status of each area on a map in a color-coded format. Since red areas indicate congestion, the user will choose a green, less crowded area and spend time there. After the event ends, the user's movement data is sent to the server, which will contribute to improving the analysis accuracy for the next event.

[0555] In this way, the present invention allows users to grasp the congestion situation in real time and enjoy the event comfortably, and also enables event organizers to quickly take measures to alleviate congestion.

[0556] The processing flow will be explained below.

[0557] Step 1: Data collection

[0558] The server collects location data from sensors installed in each area (e.g., cameras, Wi-Fi connections, BLE beacons, etc.). These sensors detect participants' location information and movement patterns in real time and send the data to the server. Specifically, the server periodically obtains data from the sensors using API requests and centrally manages the collected data.

[0559] Step 2: Preprocessing the data

[0560] The server converts the acquired raw data into an analyzable format, specifically by performing preprocessing such as filling in missing data, filtering out outliers, and standardizing the data format, so that the location data is stored in a consistent format and subsequent analysis can be performed accurately.

[0561] Step 3: Calculate congestion

[0562] The server uses a machine learning algorithm to calculate the congestion level of each area using the pre-processed data. Specifically, it calculates a congestion score based on the number of people in each area, their movement speed, past data, etc. This allows the congestion status of each area to be quantified in real time.

[0563] Step 4: Generate congestion information

[0564] Based on the calculated congestion score, the server generates congestion information in a form that is visually easy for users to understand. Specifically, the congestion score is converted into a color-coded display (e.g., red indicates congestion, green indicates vacant) and reflected in the map data. This color-coded map is intended to allow users to easily understand the congestion situation.

[0565] Step 5: Distributing information

[0566] The server distributes the generated congestion information to the user's device in real time. Specifically, it sends the information to the user's device using push notifications or API endpoints. This allows the user to always check the latest congestion status.

[0567] Step 6: User interaction

[0568] Users can launch the app on their own devices and check the congestion situation on the map screen. Specifically, they can look at each color-coded area on the map, select an area that is not crowded, and move to that area. This allows users to avoid crowds and enjoy the event in comfort.

[0569] Step 7: Gather feedback

[0570] The device periodically sends the user's movement data and operation history within the app to the server. Specifically, it periodically collects changes in congestion in the area selected by the user and movement history and sends them to the server. This allows the server to analyze the user's behavioral data and use it as feedback to improve the accuracy of future analysis.

[0571] Example 1

[0572] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0573] At large gatherings such as traditional events and festivals, participants have had limited means of understanding congestion levels in advance, making it difficult for them to enjoy a comfortable stay. Furthermore, because collected data is not analyzed or displayed in real time, it is not possible to provide appropriate information to participants in a timely manner. Furthermore, there is a lack of means to collect and utilize user feedback data, making it difficult to improve the accuracy of congestion predictions.

[0574] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0575] In this invention, the server includes a means for acquiring location data, a means for preprocessing the acquired location data to reshape the location information data, fill in missing data, and remove outliers, a means for calculating congestion levels using a machine learning algorithm based on the preprocessed data, a means for categorizing the calculated congestion level scores and generating a visually easy-to-understand color-coded map, a means for providing the generated congestion information to a user terminal, and a means for collecting user feedback data and using it to improve the accuracy of congestion level predictions. This allows users to understand congestion situations in real time and select appropriate areas for a comfortable stay. Furthermore, utilizing user feedback data can also contribute to improving the accuracy of future congestion predictions.

[0576] "Location data" refers to data that indicates the location of people or objects within a specific area, such as an event venue.

[0577] "Preprocessing" is the process of formatting raw location data, filling in missing data, removing outliers, etc., to make the data suitable for analysis.

[0578] A "machine learning algorithm" is an algorithm that analyzes data to find patterns and rules and uses them for future predictions, classification, etc.

[0579] "Crowding level" is a numerical representation of the density of people and objects in a specific area, and is an indicator of the congestion level of the area.

[0580] A "color-coded map" is a map that displays areas in different colors to visually indicate the degree of congestion.

[0581] A "user terminal" is an electronic device used by a user, such as a smartphone or tablet.

[0582] "Feedback data" is data collected to help improve the system, such as user behavior data and in-app operation history.

[0583] The present invention relates to a system that allows users to grasp the congestion situation in real time at large gatherings such as events and festivals, and provides information to help them spend their time comfortably. Specifically, the system includes the following means.

[0584] First, the server collects location data of participants using sensors installed at the event venue (e.g., cameras, Wi-Fi connection count, BLE beacons, etc.). This location data is sent to the server in real time. For example, a Wi-Fi connection count sensor can estimate the congestion level of the area based on the number of connected devices.

[0585] Next, the server preprocesses the collected raw location data. Preprocessing steps include formatting the location data, filling in missing data, and filtering out outliers. This organizes the data into a form suitable for analysis. For example, data formatting can be performed using the Python Pandas library.

[0586] After the preprocessing is complete, the server applies a machine learning algorithm (e.g., K-means clustering using the Scikit-learn library) to calculate the congestion level of each area, taking into account multiple data points such as the number of people in each area and their movement speed.

[0587] Once the congestion level is calculated, the server generates congestion information based on the analysis results. This congestion information is provided as a visually easy-to-understand color-coded map (using Mapbox or OpenStreetMap, for example). The congestion level score is divided into categories, for example, congested areas are displayed in red and vacant areas in green.

[0588] Users can check congestion information in real time through a dedicated app on their smartphones, tablets, or other devices. The app has a map screen that shows the congestion status of each area in a color-coded format. This allows users to avoid crowds and select an area where they can spend their time comfortably.

[0589] After the event ends, the server will collect user movement data and operation history within the app, and use this data as feedback to improve the accuracy of future congestion predictions. A Python data analysis library (e.g., Pandas) can be used to analyze this feedback data.

[0590] As a concrete example, consider the case where a user attends a fireworks display. The user can launch a dedicated app and check the congestion status of each area on a map in a color-coded format. Since red areas indicate congestion, the user will choose a green, less crowded area and spend time there. After the event ends, the user's movement data is sent to the server, which will contribute to improving the accuracy of analysis for the next event.

[0591] Examples of prompts for generative AI models include:

[0592] "Please create a program for a system that will allow users to grasp the congestion situation in real time at large gatherings such as events and festivals, and provide information to help them have a comfortable time. This program will include a process to preprocess location data collected by sensors, calculate the degree of congestion, and display it on a color-coded map. Please also include a function to utilize user feedback data to improve the accuracy of the analysis."

[0593] As described above, the present invention is a system that allows users to understand congestion situations in real time and enjoy events comfortably, and also contributes to improving the accuracy of congestion predictions by using collected feedback data.

[0594] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0595] Step 1:

[0596] The server collects location data using sensors (e.g., cameras, Wi-Fi connection count, BLE beacons, etc.) installed at the event venue. The data collected by the sensors is sent to the server in real time. This input data includes the location information and movement patterns of each participant. For example, a Wi-Fi connection count sensor can estimate the congestion level of the area from the number of connected devices. The output is raw location data.

[0597] Step 2:

[0598] The server pre-processes the collected raw location data. The raw location data is used as input data. The pre-processing includes the following specific operations:

[0599] Data Transformation: Convert location data into a unified format.

[0600] Missing data completion: If a sensor is temporarily out of service, the data from that time is completed using data from nearby sensors.

[0601] Outlier exclusion: If some data is detected as abnormal, it will be excluded.

[0602] The result of this preprocessing is clean location data suitable for analysis.

[0603] Step 3:

[0604] The server applies machine learning algorithms to the preprocessed data to calculate the congestion level of each area. The preprocessed location data is used as input data. The server analyzes people's density and movement patterns by using K-means clustering, for example, using Python's Scikit-learn library. This outputs the congestion level of each area as a numerical value.

[0605] Step 4:

[0606] The server generates congestion information based on the analysis results. Numerical data indicating the degree of congestion is used as input data. Specifically, the congestion scores are categorized and converted into a visually easy-to-understand color-coded map. For example, congested areas are displayed in red and vacant areas in green. Tools such as Mapbox and OpenStreetMap are used for this task. A visually displayable color-coded map is generated as output.

[0607] Step 5:

[0608] Users use devices such as smartphones and tablets to check congestion information in real time through a dedicated app. A color-coded map delivered from a server is used as input data. Specifically, the app is launched and the map screen is displayed, where the congestion status of each area is shown in color-coded format. Based on this, users can select areas to avoid congestion and spend time in those areas. The output is a congestion status map that the user can view.

[0609] Step 6:

[0610] The server collects user movement data and in-app operation history after the event ends or periodically. The user's location information and operation history are used as input data. Specifically, it tracks the user's movement patterns and the congestion status of the selected area and stores this in a database. This feedback data is used for analysis to improve the accuracy of future congestion predictions. For example, data analysis is performed using Python's Pandas library. The output makes it possible to improve the congestion prediction algorithm.

[0611] (Application example 1)

[0612] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0613] Large gatherings and events, especially in brick-and-mortar locations like shopping malls, present challenges for visitors, making it difficult to understand crowding conditions and ensure a comfortable experience. Real-time information on how to avoid crowded areas is lacking, often resulting in a poor visitor experience. Additionally, there is insufficient collection and utilization of feedback data to improve crowd prediction accuracy.

[0614] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0615] In this invention, the server includes a means for acquiring location data, a means for analyzing the acquired location data and calculating the congestion level, a means for providing the calculated congestion level to the user, a means for suggesting a route to avoid congestion based on the user's location information, and a means for collecting user feedback data and improving the accuracy of the analysis. This not only enables visitors to check the congestion situation in real time and select a comfortable area, but also makes it possible to suggest the optimal route to avoid congestion. Furthermore, by collecting and analyzing user feedback, the accuracy of congestion level prediction can be improved.

[0616] "Location data" is numerical data that indicates the current location of an individual user or group at a specific location, such as an event or shopping mall.

[0617] "Analysis" is the process of processing collected raw data and converting it into meaningful information.

[0618] "Crowding" is an indicator that indicates the density or number of people present in a particular area.

[0619] A "route suggestion method" is a system or algorithm that provides the user with the optimal route to avoid congestion based on the user's current location and congestion status.

[0620] "Feedback data" refers to information including data provided by users and logs of behavior when using the system, and is used to improve system performance and prediction accuracy.

[0621] A "server" is a computer system that centrally manages and processes the collection and analysis of location data, calculation of congestion levels, and collection of feedback data.

[0622] DETAILED DESCRIPTION OF THE INVENTION The following describes an embodiment of the present invention.

[0623] First, the server places various sensors in areas of the event venue or shopping mall to collect location data. The sensors include cameras, Wi-Fi connection counters, BLE beacons, etc. The location data collected by these sensors is sent to the server in real time.

[0624] The server then performs pre-processing on the collected raw data, which involves cleansing, formatting, imputing missing data, and filtering out outliers, using Python programs to analyze and format the data.

[0625] Once preprocessed, the data is analyzed on the server to calculate the congestion level. A machine learning algorithm is used for the analysis. Specifically, the congestion level is calculated based on data such as the number of people in each area and their movement speed. The resulting congestion level is categorized for color-coding, with red indicating congested areas and green indicating less crowded areas.

[0626] Furthermore, the server will suggest optimal routes in real time that allow users to avoid congestion based on the user's location information and congestion data. These suggestions are provided through an application installed on the user's smartphone. Users can easily check and select routes that avoid congestion by referring to the map screen.

[0627] The operations and movement data performed by the user through the application are sent to the server as feedback data, which is used to improve the accuracy of future congestion predictions.

[0628] As an example of this system, a user visiting a shopping mall can use the app to check the congestion status of each store in real time. This user can choose a less crowded area and enjoy shopping comfortably, resulting in a good user experience. In addition, feedback data from users is reflected in the next congestion status prediction, improving the accuracy of the entire system.

[0629] For example, here is a sample prompt:

[0630] "You have been asked to develop a 'Shopping Crowd Avoidance App' that uses data from BLE beacons and Wi-Fi connected counters to grasp the crowd situation in physical stores in real time and provide users with a pleasant shopping experience. This program uses Python and the Requests library to preprocess data and calculate crowd levels, generate a color-coded map, and send it to the server. Please provide the specific program flow and details of the hardware and software you will use."

[0631] This allows users to check congestion conditions in real time and choose an area to spend time in comfortably. The system can also use feedback data to improve the accuracy of its analysis.

[0632] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0633] Step 1:

[0634] The server collects real-time location data from various sensors (cameras, Wi-Fi connection counters, BLE beacons, etc.). The sensors are placed in various areas of the event venue or shopping mall to detect users' current locations and movement patterns. The input is raw data from the sensors, and the output is location data sent to the server.

[0635] Step 2:

[0636] The server performs preprocessing on the acquired location data. This preprocessing includes data cleansing, shaping, missing data completion, and outlier removal. The input is raw data acquired from the sensor, and the output is cleansed location data. A Python program is used for this preprocessing.

[0637] Step 3:

[0638] The server analyzes the preprocessed location data and calculates the congestion level of each area using a machine learning algorithm. The input is the preprocessed location data, and the output is a congestion score corresponding to each area. This analysis process also uses data such as the number of people and their movement speed.

[0639] Step 4:

[0640] The server generates a color-coded map based on the calculated congestion score, which is easy for users to understand visually. The congestion score is categorized, with red indicating congested areas and green indicating vacant areas. The input is the congestion score, and the output is a color-coded congestion map.

[0641] Step 5:

[0642] The user checks the color-coded map provided by the server through an application installed on their smartphone. The user can select a route to avoid congestion while referring to the map screen. The input is the color-coded congestion map, and the output is the route displayed on the user's application screen.

[0643] Step 6:

[0644] The user's location information and operation history within the application are sent to the server as feedback data. The server analyzes this feedback data and uses it to predict congestion levels from the next time onwards. The input is the user's feedback data, and the output is to improve the accuracy of the next analysis.

[0645] Step 7:

[0646] The server maintains the hardware and software environment for all data processing and analysis. Specifically, it uses a high-performance server, database, Python language, and the Requests library. The input is all of the aforementioned data, and the output is stable system operation and highly accurate congestion prediction.

[0647] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0648] The present invention relates to a system that allows users to grasp the congestion situation in real time at large gatherings such as events and festivals, and further recognizes the user's emotions to provide information to help them spend their time comfortably.

[0649] This system first has a means for collecting location data. Specifically, the server acquires location data from sensors (e.g., cameras, Wi-Fi connections, BLE beacons, etc.) installed in each area of ​​the event venue. These sensors detect participants' location information and movement patterns in real time and send the data to the server.

[0650] The collected data is preprocessed by the server, which formats the location data, fills in missing data, and removes outliers. Once the data is properly organized, the server uses a machine learning algorithm to calculate the congestion level of each area. This congestion level is calculated based on data such as the number of people in each area and their movement speed.

[0651] Next, the server generates congestion information based on the analysis results. This congestion information is converted into a format that is easy for users to understand visually (e.g., a color-coded map). Specifically, the congestion score is converted into a color-coded display (e.g., red indicates crowded, green indicates empty) and reflected in the map data. This color-coded map is intended to allow users to easily understand the congestion situation.

[0652] Furthermore, this system incorporates an emotion engine that recognizes the user's emotions. The emotion engine is equipped with a means for analyzing the user's facial expression data and voice data, and detects the user's emotional state based on this. Specifically, the device's camera is used to capture the user's facial expression, and the emotion engine analyzes the facial expression. The device's microphone is also used to collect the user's voice, which is then analyzed. This makes it possible to grasp the user's current emotions, such as stress and satisfaction, in real time.

[0653] Users launch the app on their own device and check the congestion situation and emotion information in real time on a map screen. For example, if a congested area is displayed in red and the user in that area is feeling stressed, the device will use this information to notify the user that the area should be avoided. This allows users to select an area where they can feel comfortable, taking into consideration their emotion and the degree of congestion.

[0654] The device also periodically sends the user's movement data, operation history within the app, and emotional feedback data to the server. Specifically, it periodically collects and sends to the server information on changes in congestion in the area selected by the user, movement history, and emotional changes. This allows the server to analyze the user's behavioral data and emotional data and use it as feedback to improve the accuracy of future analyses.

[0655] For example, suppose a user is attending a fireworks festival and launches the app. The user can check the congestion status of each area on the map and the emotional state of other users in that area in a color-coded format. If a red area indicates congestion and many users in that area are feeling stressed, the user can choose a green, less crowded area to avoid the crowds and spend time there comfortably.

[0656] In this way, the present invention allows users to grasp congestion conditions and emotional information in real time, and allows users to enjoy the event comfortably while taking their emotions into consideration. It also enables event organizers to quickly take measures to alleviate congestion.

[0657] The processing flow will be explained below.

[0658] Step 1: Data collection

[0659] The server collects location data from sensors installed in each area (e.g., cameras, Wi-Fi connections, BLE beacons, etc.). These sensors detect participants' location information and movement patterns in real time and send the data to the server. The server periodically retrieves data from the sensors using API requests and centrally manages it.

[0660] Step 2: Preprocessing the data

[0661] The server converts the acquired raw data into an analyzable format, specifically by performing preprocessing such as filling in missing data, filtering out outliers, and standardizing the data format, so that the location data is stored in a consistent format and subsequent analysis can be performed accurately.

[0662] Step 3: Calculate congestion

[0663] The server uses a machine learning algorithm to calculate the congestion level of each area using the pre-processed data. Specifically, it calculates a congestion score based on the number of people in each area, their movement speed, past data, etc. This allows the congestion status of each area to be quantified in real time.

[0664] Step 4: Generate congestion information

[0665] Based on the calculated congestion score, the server generates congestion information in a form that is visually easy for users to understand. Specifically, the congestion score is converted into a color-coded display (e.g., red indicates congestion, green indicates vacant) and reflected in the map data. This color-coded map is intended to allow users to easily understand the congestion situation.

[0666] Step 5: Collecting sentiment data

[0667] The device uses a camera and microphone to collect facial expression and voice data to recognize the user's emotions. The device's camera captures the user's facial expressions, and the microphone records the user's voice. This emotion data is processed within the device and sent to the emotion engine.

[0668] Step 6: Sentiment Analysis

[0669] The device's emotion engine analyzes the collected facial and voice data to detect the user's emotional state. Specifically, it uses facial and voice analysis algorithms to evaluate the user's stress, satisfaction, and other factors.

[0670] Step 7: Distributing information

[0671] The server delivers the generated congestion information and emotion information obtained from the device to the user's device in real time. Specifically, the information is sent to the user's device using push notifications or API endpoints and is reflected on the map screen. This allows the user to always check the latest congestion status and emotion information.

[0672] Step 8: User interaction

[0673] Users launch the app on their own device and check the congestion status and emotion information on a map screen. Specifically, they look at each color-coded area on the map, select a less congested or more comfortable area, and move to that area. Users can choose an area where they feel comfortable, taking into consideration emotions and congestion levels.

[0674] Step 9: Gather feedback

[0675] The device periodically sends the user's movement data, operation history within the app, and emotional feedback data to the server. Specifically, it periodically collects and sends to the server information on changes in congestion in the area selected by the user, movement history, and emotional changes. This allows the server to analyze the user's behavioral data and emotional data and use it as feedback to improve the accuracy of future analyses.

[0676] Example 2

[0677] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0678] At large-scale events and festivals, there is a need for a system that allows users to grasp the congestion situation in real time and provides appropriate information to help them have a comfortable experience, taking into account their emotional state. However, while conventional systems can grasp the congestion situation, they lack the functionality to analyze the user's emotional state and provide appropriate information based on that. This makes it difficult for users to select the optimal area to sit in, which can lead to a decrease in comfort.

[0679] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0680] In this invention, the server includes means for acquiring location data, means for preprocessing the collected location data, means for analyzing the preprocessed data to calculate the congestion level, means for visually displaying the calculated congestion level and providing it to the user, means for recognizing the user's emotions, means for notifying the user based on the emotion information and congestion information, and means for transmitting the user's movement data and emotion data to the server and providing feedback. This allows the user to obtain information that takes into account the congestion situation and their own emotions in real time and make the optimal choice.

[0681] "Location data" refers to data that indicates the location information and movement patterns of users at an event venue or in a specific area.

[0682] "Preprocessing means" refers to a method for converting collected location data into a normal format, filling in missing data, and excluding outliers.

[0683] The "congestion level" is an index that indicates the degree of congestion in a particular area, calculated based on information such as the density and movement speed of users in that area.

[0684] "Visual display means" refers to a method of reflecting the calculated congestion level in map data in the form of color coding or graphs, etc., so that the user can intuitively understand it.

[0685] "Means for recognizing emotions" refers to technology that analyzes a user's facial expression data and voice data to detect the user's emotional state (e.g., stress, satisfaction).

[0686] The "means for notifying" is a method for sending a notification to the terminal that suggests appropriate actions to the user based on the congestion information and emotion information.

[0687] "Movement data" is data relating to how a user moves through each area within an event venue.

[0688] "Emotion data" refers to data relating to the emotional state detected from the user's facial expression and voice.

[0689] The "means for providing feedback" is a method for periodically sending the user's movement data and emotion data to the server, and is used to improve the system's analysis accuracy.

[0690] The present invention provides a system that allows users to grasp the congestion situation in real time at large gatherings such as events and festivals, and also recognizes the user's emotions to provide information to help them spend their time comfortably. Hereinafter, an embodiment of the present invention will be described in detail.

[0691] System Configuration

[0692] The system consists of the following major hardware and software components:

[0693] Hardware

[0694] Sensors: Camera, Wi-Fi access point, BLE beacon

[0695] Device: The user's smartphone or tablet

[0696] Server: A central server that processes and analyzes data

[0697] software

[0698] Location Data Collection Module

[0699] Data Preprocessing Module

[0700] Machine Learning Algorithms

[0701] Emotion Recognition Engine

[0702] User Interface (UI) Applications

[0703] Processing Description

[0704] Location Data Collection

[0705] The server collects location data from sensors installed at the event venue, such as cameras, Wi-Fi access points, and BLE beacons. Cameras measure the density of people in an area through video analysis, and Wi-Fi access points count the number of connected devices. BLE beacons detect the locations of users in a specific area in real time. For example, if a Wi-Fi access point installed in a venue detects 100 smartphones, the area is deemed highly congested.

[0706] Data Preprocessing

[0707] The collected raw data is preprocessed by the server. This preprocessing includes normalizing location data, filling in missing data, and removing outliers. For example, if the number of Wi-Fi connections is abnormal (such as a sudden increase from 1 to 1,000), that portion is removed or replaced with imputed data.

[0708] Calculating congestion

[0709] Based on the pre-processed data, the server runs machine learning algorithms to calculate the congestion level of each area. The congestion level is calculated using data points such as the number of people, their speed of movement, and density. For example, if an area that normally holds around 30 people detects that 50 people are gathered there, the area is considered "highly congested."

[0710] Generation of congestion information

[0711] The server converts the results of the congestion calculation into a format that can be displayed visually. The congestion information is reflected in the map data as a color-coded map that is easy for users to understand. For example, on a map of a fireworks festival, areas with high congestion levels are displayed in red, and areas with low congestion levels are displayed in green.

[0712] Emotion recognition

[0713] The device uses a camera and microphone to collect facial and voice data, which is then analyzed by an emotion recognition engine. This allows the device to grasp the user's emotional state (e.g., stress, satisfaction) in real time. For example, the camera captures the user's facial expression and checks whether the user is smiling, and the microphone analyzes the user's tone of voice to determine satisfaction or stress.

[0714] Providing congestion information and emotion information

[0715] Users launch the app on their devices and check the congestion situation and emotional information on a map screen in real time. The app displays highly congested areas in red, and if the user's emotional state in that area is stressed, the app will notify them and suggest that they avoid the area. For example, when a user opens the app at a fireworks festival, congested areas will be displayed in red, indicating that the user in that area subjectively feels "congested."

[0716] Data Feedback

[0717] The device periodically sends the user's movement data, operation history within the app, and emotional feedback data to the server. This allows the server to analyze the user's movement patterns and emotional data, and uses them as feedback to improve the accuracy of future analysis. For example, changes in the congestion situation in the area selected by the user, as well as movement history and emotional changes, are collected every hour and sent to the server.

[0718] Examples of specific examples and prompts

[0719] Specific examples

[0720] If a user attends a fireworks festival and launches the app, they might see the following:

[0721] Users can check the congestion status of each area on the map and the emotional state of other users in that area in a color-coded format.

[0722] The red areas are congested, and it is clear that many users are feeling stressed.

[0723] To avoid crowds, users can choose an empty green area and move there.

[0724] Prompt Sentence Examples

[0725] "Please explain a system that uses location data collected from sensors installed in each area of ​​an event venue, and facial expression and voice data collected from cameras and microphones, to allow users to check the congestion situation and emotional information in real time and select an area where they can feel comfortable."

[0726] In this way, the system of the present invention allows users to grasp congestion conditions and emotional information in real time and supports them in selecting a comfortable area, thereby significantly improving the comfort of events.

[0727] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0728] Step 1:

[0729] The server acquires location data from sensors (cameras, Wi-Fi access points, BLE beacons, etc.) at the event venue. The sensors detect users' location information and movement patterns in each area in real time and send the data to the server. Specifically, the cameras analyze the footage to calculate the density of people, and the Wi-Fi access points count the number of connected devices. The input is raw data from the sensors, and the output is the collected location data.

[0730] Step 2:

[0731] The server preprocesses the location data acquired in the previous step. Preprocessing includes normalizing the location data, filling in missing data, and excluding outliers. Specifically, if the number of Wi-Fi connections is abnormal (e.g., a sudden increase from 1 to 1,000), that part is excluded or replaced with filled data. The input is the collected location data, and the output is the preprocessed data.

[0732] Step 3:

[0733] The server runs a machine learning algorithm on the preprocessed data to calculate the congestion level for each area. The congestion level is calculated based on data points such as the number of people in each area and their movement speed. The input is the preprocessed location data, and the output is a congestion score for each area. Specifically, if an area normally contains 30 people and 50 people are detected, the area is considered "highly congested."

[0734] Step 4:

[0735] The server generates congestion information based on the results of the congestion calculation. The generated congestion information is converted into a format that is easy for users to understand visually (e.g., a color-coded map). The input is the congestion score for each area, and the output is congestion information that can be displayed visually. Specifically, areas with high congestion levels are displayed in red on the map of the fireworks display, and areas with low congestion levels are displayed in green.

[0736] Step 5:

[0737] The device uses a camera and microphone to collect facial expression and voice data, which is then analyzed by an emotion recognition engine. The input is the user's facial expression and voice data, and the output is the user's emotional state (e.g., stress, satisfaction). Specifically, the camera captures the user's facial expression, and the microphone analyzes the user's tone of voice.

[0738] Step 6:

[0739] Users launch the app on their device and check the congestion situation and emotion information on a map screen in real time. Highly congested areas are displayed in red, and if the user in that area is feeling stressed, a notification is sent suggesting that the user avoid that area. The input is congestion information and emotion information, and the output is information provided to the user visually and in the form of a notification. Specifically, when a user opens the app at a fireworks festival, congested areas are displayed in red.

[0740] Step 7:

[0741] The device periodically sends the user's movement data, operation history within the app, and emotional feedback data to the server. The server analyzes the sent data and uses it as feedback to improve the accuracy of future analyses. The input is the user's movement data and emotional data, and the output is feedback data. Specifically, changes in congestion in the area selected by the user, movement history, and emotional changes are collected.

[0742] (Application example 2)

[0743] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0744] Physical stores are required to provide a comfortable shopping experience by grasping the congestion status in the store in real time and analyzing the emotional state of customers. Conventional systems can grasp the congestion status, but have difficulty providing information that takes into account the emotional state of customers. Furthermore, they lack the functionality to recommend comfortable areas and times to users by comprehensively considering the congestion level and emotional state. As a result, there are limitations to improving the shopping experience in physical stores. The present invention proposes a system that solves these problems and provides a more comfortable and satisfying shopping experience.

[0745] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for acquiring location data, means for analyzing the acquired location data and calculating a congestion level, means for analyzing and classifying the calculated congestion level and the user's emotional state, and means for recommending a comfortable area and time to the user based on the calculated congestion level and the analyzed emotional state. This makes it possible to recommend the most suitable shopping area and time to the user in real time, taking into consideration the congestion level and the user's emotional state comprehensively, thereby improving the shopping experience at physical stores.

[0746] "Location data" is data that indicates a user's current location and route of travel, and is information collected from sensors and devices.

[0747] "Crowding level" is a numerical value or index that indicates the density or degree of congestion of people in a particular area.

[0748] "Emotional state" is information that indicates the type and strength of the emotion the user is feeling, and is obtained by analyzing facial expressions and voice.

[0749] "Analyzing" means processing data using mathematical and statistical methods to extract meaningful information.

[0750] "Classifying" means separating analyzed data into specific categories or labels.

[0751] "Providing" means displaying the analysis results and recommendation information in a format that is easy for the user to view.

[0752] "Recommending" means suggesting the best options or actions for the user, with the aim of improving user convenience.

[0753] "Color-coded display" is a method of expressing data using colors to make it easier to visually understand congestion levels and emotional states.

[0754] "Preprocessing" refers to removing invalid parts from raw data and converting it into a format suitable for analysis.

[0755] The present invention provides a system for providing a more comfortable shopping experience by understanding the congestion situation and emotional state of customers in a physical store in real time. An embodiment of this system will be described in detail below.

[0756] System configuration

[0757] 1. How to obtain location data

[0758] Sensors (cameras, Bluetooth beacons, Wi-Fi access points, etc.) are used to collect location data for each area within the store. These sensors detect the customer's current location and movement route and send the data to a server.

[0759] 2. Method for calculating congestion level

[0760] The server uses the collected location data to calculate the congestion level of each area in real time, based on the density of customers and their movement speed. This calculation is done using machine learning algorithms.

[0761] 3. Means of analyzing emotional states

[0762] An application installed on the customer's device (smartphone) collects facial expression and voice data, which is then sent to a server, which uses an emotion analysis engine to analyze the customer's emotional state (stress, satisfaction, etc.).

[0763] Sentiment analysis uses machine learning models and image processing libraries such as TensorFlow and OpenCV.

[0764] 4. Information and recommendation means

[0765] The calculated congestion level and analyzed emotional state are provided to customers through an application. The method used is to display the congestion level and emotional state on a map in different colors. For example, areas with high congestion levels are displayed in red, and areas with good emotional states are displayed in green.

[0766] Furthermore, based on this data, the system can recommend areas and time periods that will allow customers to enjoy a comfortable shopping experience, for example, by recommending less crowded areas or areas where customers are in a good emotional state.

[0767] Specific Examples

[0768] When a user visits a shopping mall, they launch a dedicated application on their smartphone. The application receives real-time location data from sensors inside the store and also analyzes the user's facial expressions and voice to detect their emotional state. This data is sent to a server, which then comprehensively assesses the congestion situation and the user's emotional state and recommends an appropriate area for the user.

[0769] For example, if the clothing area is crowded and many customers are feeling stressed, the application will notify the user that the food area is empty and customers are happy, allowing the user to avoid the crowds and choose a place to enjoy shopping comfortably.

[0770] Prompt Sentence Examples

[0771] Below is an example of a prompt sentence to input to the generative AI model.

[0772] "I'm at the mall. Can you tell me how busy it is and what are some recommended places to visit?"

[0773] This system makes it possible to provide customers visiting physical stores with a comfortable shopping experience in real time.

[0774] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0775] Step 1:

[0776] When a user visits a shopping mall, they launch a dedicated application on their device (smartphone). The inputs include the user's location, facial expression, and voice data. This data is collected in real time using the device's camera and microphone. The output is the raw data acquired, which is sent to the server.

[0777] Step 2:

[0778] The server collects location data from sensors (cameras, Bluetooth beacons, Wi-Fi access points, etc.) installed in each area. It receives location information data from the sensors as input. Based on this, the server organizes the location data for each area and performs preprocessing such as filling in missing data and excluding outliers. The preprocessed location data is obtained as output.

[0779] Step 3:

[0780] The server uses machine learning algorithms to calculate congestion levels based on the preprocessed location data. Organized location data is required as input. The server calculates customer density and movement speed, and generates a congestion score for each area. The output is a congestion score for each area.

[0781] Step 4:

[0782] The facial expression and voice data acquired by the device is sent to a server, which analyzes it. Image and voice data are required as input. The server uses machine learning models such as TensorFlow and OpenCV to analyze and classify the emotional state (stress, satisfaction, etc.). The output is a label for the analyzed emotional state.

[0783] Step 5:

[0784] The server generates information based on the calculated congestion score and the analyzed emotional state data and provides it to the user's device. The congestion score and emotional state label are required as input. Based on this data, the server displays the congestion status and emotional state by color-coding them on the map data. As output, the color-coded map and information on recommended areas and time periods are displayed on the user's device.

[0785] Step 6:

[0786] The server periodically accumulates the user's movement history and emotion data to improve the accuracy of future analyses. As input, it receives the user's behavioral data and emotion data. The server analyzes this data and uses it to improve the accuracy of the congestion and emotion prediction model. As output, it obtains an improved congestion and emotion prediction model.

[0787] Specifically, when a user enters a prompt into the application such as "Please tell me the current congestion situation and recommended areas," the server analyzes the data in real time and provides the user with the most appropriate information.

[0788] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0789] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0790] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[0791] [Fourth embodiment]

[0792] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[0793] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0794] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0795] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[0796] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0797] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0798] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0799] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[0800] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0801] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0802] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0803] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0804] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[0805] The present invention relates to a system that allows users to grasp the congestion situation in real time at large gatherings such as events and festivals, and provides information to help them spend their time comfortably.

[0806] This system first has a means of collecting location data. Specifically, the server acquires data using sensors (e.g., cameras, Wi-Fi connections, BLE beacons, etc.) installed in each area of ​​the event venue. These sensors collect participants' location information and movement patterns in real time and send them to the server.

[0807] The collected data is preprocessed by the server, which formats the location data, fills in missing data, and removes outliers. Once the data is properly organized, the server uses a machine learning algorithm to calculate the congestion level of each area. This congestion level is calculated based on data such as the number of people in each area and their movement speed.

[0808] Next, the server generates congestion information based on the analysis results. This congestion information is converted into a form that is easy for users to understand visually (e.g., a color-coded map). Specifically, the congestion score is divided into categories for color-coding, for example, congested areas are displayed in red and uncrowded areas in green.

[0809] Users can check congestion information in real time through the app on their own devices (smartphones, tablets, etc.). The app has a map screen that displays the congestion status of each area in a color-coded format. This allows users to select an area where they can avoid crowds and spend their time comfortably.

[0810] The system also includes a means for collecting user feedback data. The system periodically sends the user's movement data and operation history within the app to a server, and uses this feedback data to improve the accuracy of the analysis. Specifically, the system tracks changes in the congestion level in the area selected by the user, and reflects this data in future congestion predictions.

[0811] For example, suppose a user is attending a fireworks festival and launches the app. The user can check the congestion status of each area on a map in a color-coded format. Since red areas indicate congestion, the user will choose a green, less crowded area and spend time there. After the event ends, the user's movement data is sent to the server, which will contribute to improving the analysis accuracy for the next event.

[0812] In this way, the present invention allows users to grasp the congestion situation in real time and enjoy the event comfortably, and also enables event organizers to quickly take measures to alleviate congestion.

[0813] The processing flow will be explained below.

[0814] Step 1: Data collection

[0815] The server collects location data from sensors installed in each area (e.g., cameras, Wi-Fi connections, BLE beacons, etc.). These sensors detect participants' location information and movement patterns in real time and send the data to the server. Specifically, the server periodically obtains data from the sensors using API requests and centrally manages the collected data.

[0816] Step 2: Preprocessing the data

[0817] The server converts the acquired raw data into an analyzable format, specifically by performing preprocessing such as filling in missing data, filtering out outliers, and standardizing the data format, so that the location data is stored in a consistent format and subsequent analysis can be performed accurately.

[0818] Step 3: Calculate congestion

[0819] The server uses a machine learning algorithm to calculate the congestion level of each area using the pre-processed data. Specifically, it calculates a congestion score based on the number of people in each area, their movement speed, past data, etc. This allows the congestion status of each area to be quantified in real time.

[0820] Step 4: Generate congestion information

[0821] Based on the calculated congestion score, the server generates congestion information in a form that is visually easy for users to understand. Specifically, the congestion score is converted into a color-coded display (e.g., red indicates congestion, green indicates vacant) and reflected in the map data. This color-coded map is intended to allow users to easily understand the congestion situation.

[0822] Step 5: Distributing information

[0823] The server distributes the generated congestion information to the user's device in real time. Specifically, it sends the information to the user's device using push notifications or API endpoints. This allows the user to always check the latest congestion status.

[0824] Step 6: User interaction

[0825] Users can launch the app on their own devices and check the congestion situation on the map screen. Specifically, they can look at each color-coded area on the map, select an area that is not crowded, and move to that area. This allows users to avoid crowds and enjoy the event in comfort.

[0826] Step 7: Gather feedback

[0827] The device periodically sends the user's movement data and operation history within the app to the server. Specifically, it periodically collects changes in congestion in the area selected by the user and movement history and sends them to the server. This allows the server to analyze the user's behavioral data and use it as feedback to improve the accuracy of future analysis.

[0828] Example 1

[0829] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[0830] At large gatherings such as traditional events and festivals, participants have had limited means of understanding congestion levels in advance, making it difficult for them to enjoy a comfortable stay. Furthermore, because collected data is not analyzed or displayed in real time, it is not possible to provide appropriate information to participants in a timely manner. Furthermore, there is a lack of means to collect and utilize user feedback data, making it difficult to improve the accuracy of congestion predictions.

[0831] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0832] In this invention, the server includes a means for acquiring location data, a means for preprocessing the acquired location data to reshape the location information data, fill in missing data, and remove outliers, a means for calculating congestion levels using a machine learning algorithm based on the preprocessed data, a means for categorizing the calculated congestion level scores and generating a visually easy-to-understand color-coded map, a means for providing the generated congestion information to a user terminal, and a means for collecting user feedback data and using it to improve the accuracy of congestion level predictions. This allows users to understand congestion situations in real time and select appropriate areas for a comfortable stay. Furthermore, utilizing user feedback data can also contribute to improving the accuracy of future congestion predictions.

[0833] "Location data" refers to data that indicates the location of people or objects within a specific area, such as an event venue.

[0834] "Preprocessing" is the process of formatting raw location data, filling in missing data, removing outliers, etc., to make the data suitable for analysis.

[0835] A "machine learning algorithm" is an algorithm that analyzes data to find patterns and rules and uses them for future predictions, classification, etc.

[0836] "Crowding level" is a numerical representation of the density of people and objects in a specific area, and is an indicator of the congestion level of the area.

[0837] A "color-coded map" is a map that displays areas in different colors to visually indicate the degree of congestion.

[0838] A "user terminal" is an electronic device used by a user, such as a smartphone or tablet.

[0839] "Feedback data" is data collected to help improve the system, such as user behavior data and in-app operation history.

[0840] The present invention relates to a system that allows users to grasp the congestion situation in real time at large gatherings such as events and festivals, and provides information to help them spend their time comfortably. Specifically, the system includes the following means.

[0841] First, the server collects location data of participants using sensors installed at the event venue (e.g., cameras, Wi-Fi connection count, BLE beacons, etc.). This location data is sent to the server in real time. For example, a Wi-Fi connection count sensor can estimate the congestion level of the area based on the number of connected devices.

[0842] Next, the server preprocesses the collected raw location data. Preprocessing steps include formatting the location data, filling in missing data, and filtering out outliers. This organizes the data into a form suitable for analysis. For example, data formatting can be performed using the Python Pandas library.

[0843] After the preprocessing is complete, the server applies a machine learning algorithm (e.g., K-means clustering using the Scikit-learn library) to calculate the congestion level of each area, taking into account multiple data points such as the number of people in each area and their movement speed.

[0844] Once the congestion level is calculated, the server generates congestion information based on the analysis results. This congestion information is provided as a visually easy-to-understand color-coded map (using Mapbox or OpenStreetMap, for example). The congestion level score is divided into categories, for example, congested areas are displayed in red and vacant areas in green.

[0845] Users can check congestion information in real time through a dedicated app on their smartphones, tablets, or other devices. The app has a map screen that shows the congestion status of each area in a color-coded format. This allows users to avoid crowds and select an area where they can spend their time comfortably.

[0846] After the event ends, the server will collect user movement data and operation history within the app, and use this data as feedback to improve the accuracy of future congestion predictions. A Python data analysis library (e.g., Pandas) can be used to analyze this feedback data.

[0847] As a concrete example, consider the case where a user attends a fireworks display. The user can launch a dedicated app and check the congestion status of each area on a map in a color-coded format. Since red areas indicate congestion, the user will choose a green, less crowded area and spend time there. After the event ends, the user's movement data is sent to the server, which will contribute to improving the accuracy of analysis for the next event.

[0848] Examples of prompts for generative AI models include:

[0849] "Please create a program for a system that will allow users to grasp the congestion situation in real time at large gatherings such as events and festivals, and provide information to help them have a comfortable time. This program will include a process to preprocess location data collected by sensors, calculate the degree of congestion, and display it on a color-coded map. Please also include a function to utilize user feedback data to improve the accuracy of the analysis."

[0850] As described above, the present invention is a system that allows users to understand congestion situations in real time and enjoy events comfortably, and also contributes to improving the accuracy of congestion predictions by using collected feedback data.

[0851] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0852] Step 1:

[0853] The server collects location data using sensors (e.g., cameras, Wi-Fi connection count, BLE beacons, etc.) installed at the event venue. The data collected by the sensors is sent to the server in real time. This input data includes the location information and movement patterns of each participant. For example, a Wi-Fi connection count sensor can estimate the congestion level of the area from the number of connected devices. The output is raw location data.

[0854] Step 2:

[0855] The server pre-processes the collected raw location data. The raw location data is used as input data. The pre-processing includes the following specific operations:

[0856] Data Transformation: Convert location data into a unified format.

[0857] Missing data completion: If a sensor is temporarily out of service, the data from that time is completed using data from nearby sensors.

[0858] Outlier exclusion: If some data is detected as abnormal, it will be excluded.

[0859] The result of this preprocessing is clean location data suitable for analysis.

[0860] Step 3:

[0861] The server applies machine learning algorithms to the preprocessed data to calculate the congestion level of each area. The preprocessed location data is used as input data. The server analyzes people's density and movement patterns by using K-means clustering, for example, using Python's Scikit-learn library. This outputs the congestion level of each area as a numerical value.

[0862] Step 4:

[0863] The server generates congestion information based on the analysis results. Numerical data indicating the degree of congestion is used as input data. Specifically, the congestion scores are categorized and converted into a visually easy-to-understand color-coded map. For example, congested areas are displayed in red and vacant areas in green. Tools such as Mapbox and OpenStreetMap are used for this task. A visually displayable color-coded map is generated as output.

[0864] Step 5:

[0865] Users use devices such as smartphones and tablets to check congestion information in real time through a dedicated app. A color-coded map delivered from a server is used as input data. Specifically, the app is launched and the map screen is displayed, where the congestion status of each area is shown in color-coded format. Based on this, users can select areas to avoid congestion and spend time in those areas. The output is a congestion status map that the user can view.

[0866] Step 6:

[0867] The server collects user movement data and in-app operation history after the event ends or periodically. The user's location information and operation history are used as input data. Specifically, it tracks the user's movement patterns and the congestion status of the selected area and stores this in a database. This feedback data is used for analysis to improve the accuracy of future congestion predictions. For example, data analysis is performed using Python's Pandas library. The output makes it possible to improve the congestion prediction algorithm.

[0868] (Application example 1)

[0869] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[0870] Large gatherings and events, especially in brick-and-mortar locations like shopping malls, present challenges for visitors, making it difficult to understand crowding conditions and ensure a comfortable experience. Real-time information on how to avoid crowded areas is lacking, often resulting in a poor visitor experience. Additionally, there is insufficient collection and utilization of feedback data to improve crowd prediction accuracy.

[0871] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0872] In this invention, the server includes a means for acquiring location data, a means for analyzing the acquired location data and calculating the congestion level, a means for providing the calculated congestion level to the user, a means for suggesting a route to avoid congestion based on the user's location information, and a means for collecting user feedback data and improving the accuracy of the analysis. This not only enables visitors to check the congestion situation in real time and select a comfortable area, but also makes it possible to suggest the optimal route to avoid congestion. Furthermore, by collecting and analyzing user feedback, the accuracy of congestion level prediction can be improved.

[0873] "Location data" is numerical data that indicates the current location of an individual user or group at a specific location, such as an event or shopping mall.

[0874] "Analysis" is the process of processing collected raw data and converting it into meaningful information.

[0875] "Crowding" is an indicator that indicates the density or number of people present in a particular area.

[0876] A "route suggestion method" is a system or algorithm that provides the user with the optimal route to avoid congestion based on the user's current location and congestion status.

[0877] "Feedback data" refers to information including data provided by users and logs of behavior when using the system, and is used to improve system performance and prediction accuracy.

[0878] A "server" is a computer system that centrally manages and processes the collection and analysis of location data, calculation of congestion levels, and collection of feedback data.

[0879] DETAILED DESCRIPTION OF THE INVENTION The following describes an embodiment of the present invention.

[0880] First, the server places various sensors in areas of the event venue or shopping mall to collect location data. The sensors include cameras, Wi-Fi connection counters, BLE beacons, etc. The location data collected by these sensors is sent to the server in real time.

[0881] The server then performs pre-processing on the collected raw data, which involves cleansing, formatting, imputing missing data, and filtering out outliers, using Python programs to analyze and format the data.

[0882] Once preprocessed, the data is analyzed on the server to calculate the congestion level. A machine learning algorithm is used for the analysis. Specifically, the congestion level is calculated based on data such as the number of people in each area and their movement speed. The resulting congestion level is categorized for color-coding, with red indicating congested areas and green indicating less crowded areas.

[0883] Furthermore, the server will suggest optimal routes in real time that allow users to avoid congestion based on the user's location information and congestion data. These suggestions are provided through an application installed on the user's smartphone. Users can easily check and select routes that avoid congestion by referring to the map screen.

[0884] The operations and movement data performed by the user through the application are sent to the server as feedback data, which is used to improve the accuracy of future congestion predictions.

[0885] As an example of this system, a user visiting a shopping mall can use the app to check the congestion status of each store in real time. This user can choose a less crowded area and enjoy shopping comfortably, resulting in a good user experience. In addition, feedback data from users is reflected in the next congestion status prediction, improving the accuracy of the entire system.

[0886] For example, here is a sample prompt:

[0887] "You have been asked to develop a 'Shopping Crowd Avoidance App' that uses data from BLE beacons and Wi-Fi connected counters to grasp the crowd situation in physical stores in real time and provide users with a pleasant shopping experience. This program uses Python and the Requests library to preprocess data and calculate crowd levels, generate a color-coded map, and send it to the server. Please provide the specific program flow and details of the hardware and software you will use."

[0888] This allows users to check congestion conditions in real time and choose an area to spend time in comfortably. The system can also use feedback data to improve the accuracy of its analysis.

[0889] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0890] Step 1:

[0891] The server collects real-time location data from various sensors (cameras, Wi-Fi connection counters, BLE beacons, etc.). The sensors are placed in various areas of the event venue or shopping mall to detect users' current locations and movement patterns. The input is raw data from the sensors, and the output is location data sent to the server.

[0892] Step 2:

[0893] The server performs preprocessing on the acquired location data. This preprocessing includes data cleansing, shaping, missing data completion, and outlier removal. The input is raw data acquired from the sensor, and the output is cleansed location data. A Python program is used for this preprocessing.

[0894] Step 3:

[0895] The server analyzes the preprocessed location data and calculates the congestion level of each area using a machine learning algorithm. The input is the preprocessed location data, and the output is a congestion score corresponding to each area. This analysis process also uses data such as the number of people and their movement speed.

[0896] Step 4:

[0897] The server generates a color-coded map based on the calculated congestion score, which is easy for users to understand visually. The congestion score is categorized, with red indicating congested areas and green indicating vacant areas. The input is the congestion score, and the output is a color-coded congestion map.

[0898] Step 5:

[0899] The user checks the color-coded map provided by the server through an application installed on their smartphone. The user can select a route to avoid congestion while referring to the map screen. The input is the color-coded congestion map, and the output is the route displayed on the user's application screen.

[0900] Step 6:

[0901] The user's location information and operation history within the application are sent to the server as feedback data. The server analyzes this feedback data and uses it to predict congestion levels from the next time onwards. The input is the user's feedback data, and the output is to improve the accuracy of the next analysis.

[0902] Step 7:

[0903] The server maintains the hardware and software environment for all data processing and analysis. Specifically, it uses a high-performance server, database, Python language, and the Requests library. The input is all of the aforementioned data, and the output is stable system operation and highly accurate congestion prediction.

[0904] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0905] The present invention relates to a system that allows users to grasp the congestion situation in real time at large gatherings such as events and festivals, and further recognizes the user's emotions to provide information to help them spend their time comfortably.

[0906] This system first has a means for collecting location data. Specifically, the server acquires location data from sensors (e.g., cameras, Wi-Fi connections, BLE beacons, etc.) installed in each area of ​​the event venue. These sensors detect participants' location information and movement patterns in real time and send the data to the server.

[0907] The collected data is preprocessed by the server, which formats the location data, fills in missing data, and removes outliers. Once the data is properly organized, the server uses a machine learning algorithm to calculate the congestion level of each area. This congestion level is calculated based on data such as the number of people in each area and their movement speed.

[0908] Next, the server generates congestion information based on the analysis results. This congestion information is converted into a format that is easy for users to understand visually (e.g., a color-coded map). Specifically, the congestion score is converted into a color-coded display (e.g., red indicates crowded, green indicates empty) and reflected in the map data. This color-coded map is intended to allow users to easily understand the congestion situation.

[0909] Furthermore, this system incorporates an emotion engine that recognizes the user's emotions. The emotion engine is equipped with a means for analyzing the user's facial expression data and voice data, and detects the user's emotional state based on this. Specifically, the device's camera is used to capture the user's facial expression, and the emotion engine analyzes the facial expression. The device's microphone is also used to collect the user's voice, which is then analyzed. This makes it possible to grasp the user's current emotions, such as stress and satisfaction, in real time.

[0910] Users launch the app on their own device and check the congestion situation and emotion information in real time on a map screen. For example, if a congested area is displayed in red and the user in that area is feeling stressed, the device will use this information to notify the user that the area should be avoided. This allows users to select an area where they can feel comfortable, taking into consideration their emotion and the degree of congestion.

[0911] The device also periodically sends the user's movement data, operation history within the app, and emotional feedback data to the server. Specifically, it periodically collects and sends to the server information on changes in congestion in the area selected by the user, movement history, and emotional changes. This allows the server to analyze the user's behavioral data and emotional data and use it as feedback to improve the accuracy of future analyses.

[0912] For example, suppose a user is attending a fireworks festival and launches the app. The user can check the congestion status of each area on the map and the emotional state of other users in that area in a color-coded format. If a red area indicates congestion and many users in that area are feeling stressed, the user can choose a green, less crowded area to avoid the crowds and spend time there comfortably.

[0913] In this way, the present invention allows users to grasp congestion conditions and emotional information in real time, and allows users to enjoy the event comfortably while taking their emotions into consideration. It also enables event organizers to quickly take measures to alleviate congestion.

[0914] The processing flow will be explained below.

[0915] Step 1: Data collection

[0916] The server collects location data from sensors installed in each area (e.g., cameras, Wi-Fi connections, BLE beacons, etc.). These sensors detect participants' location information and movement patterns in real time and send the data to the server. The server periodically retrieves data from the sensors using API requests and centrally manages it.

[0917] Step 2: Preprocessing the data

[0918] The server converts the acquired raw data into an analyzable format, specifically by performing preprocessing such as filling in missing data, filtering out outliers, and standardizing the data format, so that the location data is stored in a consistent format and subsequent analysis can be performed accurately.

[0919] Step 3: Calculate congestion

[0920] The server uses a machine learning algorithm to calculate the congestion level of each area using the pre-processed data. Specifically, it calculates a congestion score based on the number of people in each area, their movement speed, past data, etc. This allows the congestion status of each area to be quantified in real time.

[0921] Step 4: Generate congestion information

[0922] Based on the calculated congestion score, the server generates congestion information in a form that is visually easy for users to understand. Specifically, the congestion score is converted into a color-coded display (e.g., red indicates congestion, green indicates vacant) and reflected in the map data. This color-coded map is intended to allow users to easily understand the congestion situation.

[0923] Step 5: Collecting sentiment data

[0924] The device uses a camera and microphone to collect facial expression and voice data to recognize the user's emotions. The device's camera captures the user's facial expressions, and the microphone records the user's voice. This emotion data is processed within the device and sent to the emotion engine.

[0925] Step 6: Sentiment Analysis

[0926] The device's emotion engine analyzes the collected facial and voice data to detect the user's emotional state. Specifically, it uses facial and voice analysis algorithms to evaluate the user's stress, satisfaction, and other factors.

[0927] Step 7: Distributing information

[0928] The server delivers the generated congestion information and emotion information obtained from the device to the user's device in real time. Specifically, the information is sent to the user's device using push notifications or API endpoints and is reflected on the map screen. This allows the user to always check the latest congestion status and emotion information.

[0929] Step 8: User interaction

[0930] Users launch the app on their own device and check the congestion status and emotion information on a map screen. Specifically, they look at each color-coded area on the map, select a less congested or more comfortable area, and move to that area. Users can choose an area where they feel comfortable, taking into consideration emotions and congestion levels.

[0931] Step 9: Gather feedback

[0932] The device periodically sends the user's movement data, operation history within the app, and emotional feedback data to the server. Specifically, it periodically collects and sends to the server information on changes in congestion in the area selected by the user, movement history, and emotional changes. This allows the server to analyze the user's behavioral data and emotional data and use it as feedback to improve the accuracy of future analyses.

[0933] Example 2

[0934] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[0935] At large-scale events and festivals, there is a need for a system that allows users to grasp the congestion situation in real time and provides appropriate information to help them have a comfortable experience, taking into account their emotional state. However, while conventional systems can grasp the congestion situation, they lack the functionality to analyze the user's emotional state and provide appropriate information based on that. This makes it difficult for users to select the optimal area to sit in, which can lead to a decrease in comfort.

[0936] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0937] In this invention, the server includes means for acquiring location data, means for preprocessing the collected location data, means for analyzing the preprocessed data to calculate the congestion level, means for visually displaying the calculated congestion level and providing it to the user, means for recognizing the user's emotions, means for notifying the user based on the emotion information and congestion information, and means for transmitting the user's movement data and emotion data to the server and providing feedback. This allows the user to obtain information that takes into account the congestion situation and their own emotions in real time and make the optimal choice.

[0938] "Location data" refers to data that indicates the location information and movement patterns of users at an event venue or in a specific area.

[0939] "Preprocessing means" refers to a method for converting collected location data into a normal format, filling in missing data, and excluding outliers.

[0940] The "congestion level" is an index that indicates the degree of congestion in a particular area, calculated based on information such as the density and movement speed of users in that area.

[0941] "Visual display means" refers to a method of reflecting the calculated congestion level in map data in the form of color coding or graphs, etc., so that the user can intuitively understand it.

[0942] "Means for recognizing emotions" refers to technology that analyzes a user's facial expression data and voice data to detect the user's emotional state (e.g., stress, satisfaction).

[0943] The "means for notifying" is a method for sending a notification to the terminal that suggests appropriate actions to the user based on the congestion information and emotion information.

[0944] "Movement data" is data relating to how a user moves through each area within an event venue.

[0945] "Emotion data" refers to data relating to the emotional state detected from the user's facial expression and voice.

[0946] The "means for providing feedback" is a method for periodically sending the user's movement data and emotion data to the server, and is used to improve the system's analysis accuracy.

[0947] The present invention provides a system that allows users to grasp the congestion situation in real time at large gatherings such as events and festivals, and also recognizes the user's emotions to provide information to help them spend their time comfortably. Hereinafter, an embodiment of the present invention will be described in detail.

[0948] System Configuration

[0949] The system consists of the following major hardware and software components:

[0950] Hardware

[0951] Sensors: Camera, Wi-Fi access point, BLE beacon

[0952] Device: The user's smartphone or tablet

[0953] Server: A central server that processes and analyzes data

[0954] software

[0955] Location Data Collection Module

[0956] Data Preprocessing Module

[0957] Machine Learning Algorithms

[0958] Emotion Recognition Engine

[0959] User Interface (UI) Applications

[0960] Processing Description

[0961] Location Data Collection

[0962] The server collects location data from sensors installed at the event venue, such as cameras, Wi-Fi access points, and BLE beacons. Cameras measure the density of people in an area through video analysis, and Wi-Fi access points count the number of connected devices. BLE beacons detect the locations of users in a specific area in real time. For example, if a Wi-Fi access point installed in a venue detects 100 smartphones, the area is deemed highly congested.

[0963] Data Preprocessing

[0964] The collected raw data is preprocessed by the server. This preprocessing includes normalizing location data, filling in missing data, and removing outliers. For example, if the number of Wi-Fi connections is abnormal (such as a sudden increase from 1 to 1,000), that portion is removed or replaced with imputed data.

[0965] Calculating congestion

[0966] Based on the pre-processed data, the server runs machine learning algorithms to calculate the congestion level of each area. The congestion level is calculated using data points such as the number of people, their speed of movement, and density. For example, if an area that normally holds around 30 people detects that 50 people are gathered there, the area is considered "highly congested."

[0967] Generation of congestion information

[0968] The server converts the results of the congestion calculation into a format that can be displayed visually. The congestion information is reflected in the map data as a color-coded map that is easy for users to understand. For example, on a map of a fireworks festival, areas with high congestion levels are displayed in red, and areas with low congestion levels are displayed in green.

[0969] Emotion recognition

[0970] The device uses a camera and microphone to collect facial and voice data, which is then analyzed by an emotion recognition engine. This allows the device to grasp the user's emotional state (e.g., stress, satisfaction) in real time. For example, the camera captures the user's facial expression and checks whether the user is smiling, and the microphone analyzes the user's tone of voice to determine satisfaction or stress.

[0971] Providing congestion information and emotion information

[0972] Users launch the app on their devices and check the congestion situation and emotional information on a map screen in real time. The app displays highly congested areas in red, and if the user's emotional state in that area is stressed, the app will notify them and suggest that they avoid the area. For example, when a user opens the app at a fireworks festival, congested areas will be displayed in red, indicating that the user in that area subjectively feels "congested."

[0973] Data Feedback

[0974] The device periodically sends the user's movement data, operation history within the app, and emotional feedback data to the server. This allows the server to analyze the user's movement patterns and emotional data, and uses them as feedback to improve the accuracy of future analysis. For example, changes in the congestion situation in the area selected by the user, as well as movement history and emotional changes, are collected every hour and sent to the server.

[0975] Examples of specific examples and prompts

[0976] Specific examples

[0977] If a user attends a fireworks festival and launches the app, they might see the following:

[0978] Users can check the congestion status of each area on the map and the emotional state of other users in that area in a color-coded format.

[0979] The red areas are congested, and it is clear that many users are feeling stressed.

[0980] To avoid crowds, users can choose an empty green area and move there.

[0981] Prompt Sentence Examples

[0982] "Please explain a system that uses location data collected from sensors installed in each area of ​​an event venue, and facial expression and voice data collected from cameras and microphones, to allow users to check the congestion situation and emotional information in real time and select an area where they can feel comfortable."

[0983] In this way, the system of the present invention allows users to grasp congestion conditions and emotional information in real time and supports them in selecting a comfortable area, thereby significantly improving the comfort of events.

[0984] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0985] Step 1:

[0986] The server acquires location data from sensors (cameras, Wi-Fi access points, BLE beacons, etc.) at the event venue. The sensors detect users' location information and movement patterns in each area in real time and send the data to the server. Specifically, the cameras analyze the footage to calculate the density of people, and the Wi-Fi access points count the number of connected devices. The input is raw data from the sensors, and the output is the collected location data.

[0987] Step 2:

[0988] The server preprocesses the location data acquired in the previous step. Preprocessing includes normalizing the location data, filling in missing data, and excluding outliers. Specifically, if the number of Wi-Fi connections is abnormal (e.g., a sudden increase from 1 to 1,000), that part is excluded or replaced with filled data. The input is the collected location data, and the output is the preprocessed data.

[0989] Step 3:

[0990] The server runs a machine learning algorithm on the preprocessed data to calculate the congestion level for each area. The congestion level is calculated based on data points such as the number of people in each area and their movement speed. The input is the preprocessed location data, and the output is a congestion score for each area. Specifically, if an area normally contains 30 people and 50 people are detected, the area is considered "highly congested."

[0991] Step 4:

[0992] The server generates congestion information based on the results of the congestion calculation. The generated congestion information is converted into a format that is easy for users to understand visually (e.g., a color-coded map). The input is the congestion score for each area, and the output is congestion information that can be displayed visually. Specifically, areas with high congestion levels are displayed in red on the map of the fireworks display, and areas with low congestion levels are displayed in green.

[0993] Step 5:

[0994] The device uses a camera and microphone to collect facial expression and voice data, which is then analyzed by an emotion recognition engine. The input is the user's facial expression and voice data, and the output is the user's emotional state (e.g., stress, satisfaction). Specifically, the camera captures the user's facial expression, and the microphone analyzes the user's tone of voice.

[0995] Step 6:

[0996] Users launch the app on their device and check the congestion situation and emotion information on a map screen in real time. Highly congested areas are displayed in red, and if the user in that area is feeling stressed, a notification is sent suggesting that the user avoid that area. The input is congestion information and emotion information, and the output is information provided to the user visually and in the form of a notification. Specifically, when a user opens the app at a fireworks festival, congested areas are displayed in red.

[0997] Step 7:

[0998] The device periodically sends the user's movement data, operation history within the app, and emotional feedback data to the server. The server analyzes the sent data and uses it as feedback to improve the accuracy of future analyses. The input is the user's movement data and emotional data, and the output is feedback data. Specifically, changes in congestion in the area selected by the user, movement history, and emotional changes are collected.

[0999] (Application example 2)

[1000] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1001] Physical stores are required to provide a comfortable shopping experience by grasping the congestion status in the store in real time and analyzing the emotional state of customers. Conventional systems can grasp the congestion status, but have difficulty providing information that takes into account the emotional state of customers. Furthermore, they lack the functionality to recommend comfortable areas and times to users by comprehensively considering the congestion level and emotional state. As a result, there are limitations to improving the shopping experience in physical stores. The present invention proposes a system that solves these problems and provides a more comfortable and satisfying shopping experience.

[1002] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for acquiring location data, means for analyzing the acquired location data and calculating a congestion level, means for analyzing and classifying the calculated congestion level and the user's emotional state, and means for recommending a comfortable area and time to the user based on the calculated congestion level and the analyzed emotional state. This makes it possible to recommend the most suitable shopping area and time to the user in real time, taking into consideration the congestion level and the user's emotional state comprehensively, thereby improving the shopping experience at physical stores.

[1003] "Location data" is data that indicates a user's current location and route of travel, and is information collected from sensors and devices.

[1004] "Crowding level" is a numerical value or index that indicates the density or degree of congestion of people in a particular area.

[1005] "Emotional state" is information that indicates the type and strength of the emotion the user is feeling, and is obtained by analyzing facial expressions and voice.

[1006] "Analyzing" means processing data using mathematical and statistical methods to extract meaningful information.

[1007] "Classifying" means separating analyzed data into specific categories or labels.

[1008] "Providing" means displaying the analysis results and recommendation information in a format that is easy for the user to view.

[1009] "Recommending" means suggesting the best options or actions for the user, with the aim of improving user convenience.

[1010] "Color-coded display" is a method of expressing data using colors to make it easier to visually understand congestion levels and emotional states.

[1011] "Preprocessing" refers to removing invalid parts from raw data and converting it into a format suitable for analysis.

[1012] The present invention provides a system for providing a more comfortable shopping experience by understanding the congestion situation and emotional state of customers in a physical store in real time. An embodiment of this system will be described in detail below.

[1013] System configuration

[1014] 1. How to obtain location data

[1015] Sensors (cameras, Bluetooth beacons, Wi-Fi access points, etc.) are used to collect location data for each area within the store. These sensors detect the customer's current location and movement route and send the data to a server.

[1016] 2. Method for calculating congestion level

[1017] The server uses the collected location data to calculate the congestion level of each area in real time, based on the density of customers and their movement speed. This calculation is done using machine learning algorithms.

[1018] 3. Means of analyzing emotional states

[1019] An application installed on the customer's device (smartphone) collects facial expression and voice data, which is then sent to a server, which uses an emotion analysis engine to analyze the customer's emotional state (stress, satisfaction, etc.).

[1020] Sentiment analysis uses machine learning models and image processing libraries such as TensorFlow and OpenCV.

[1021] 4. Information and recommendation means

[1022] The calculated congestion level and analyzed emotional state are provided to customers through an application. The method used is to display the congestion level and emotional state on a map in different colors. For example, areas with high congestion levels are displayed in red, and areas with good emotional states are displayed in green.

[1023] Furthermore, based on this data, the system can recommend areas and time periods that will allow customers to enjoy a comfortable shopping experience, for example, by recommending less crowded areas or areas where customers are in a good emotional state.

[1024] Specific Examples

[1025] When a user visits a shopping mall, they launch a dedicated application on their smartphone. The application receives real-time location data from sensors inside the store and also analyzes the user's facial expressions and voice to detect their emotional state. This data is sent to a server, which then comprehensively assesses the congestion situation and the user's emotional state and recommends an appropriate area for the user.

[1026] For example, if the clothing area is crowded and many customers are feeling stressed, the application will notify the user that the food area is empty and customers are happy, allowing the user to avoid the crowds and choose a place to enjoy shopping comfortably.

[1027] Prompt Sentence Examples

[1028] Below is an example of a prompt sentence to input to the generative AI model.

[1029] "I'm at the mall. Can you tell me how busy it is and what are some recommended places to visit?"

[1030] This system makes it possible to provide customers visiting physical stores with a comfortable shopping experience in real time.

[1031] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1032] Step 1:

[1033] When a user visits a shopping mall, they launch a dedicated application on their device (smartphone). The inputs include the user's location, facial expression, and voice data. This data is collected in real time using the device's camera and microphone. The output is the raw data acquired, which is sent to the server.

[1034] Step 2:

[1035] The server collects location data from sensors (cameras, Bluetooth beacons, Wi-Fi access points, etc.) installed in each area. It receives location information data from the sensors as input. Based on this, the server organizes the location data for each area and performs preprocessing such as filling in missing data and excluding outliers. The preprocessed location data is obtained as output.

[1036] Step 3:

[1037] The server uses machine learning algorithms to calculate congestion levels based on the preprocessed location data. Organized location data is required as input. The server calculates customer density and movement speed, and generates a congestion score for each area. The output is a congestion score for each area.

[1038] Step 4:

[1039] The facial expression and voice data acquired by the device is sent to a server, which analyzes it. Image and voice data are required as input. The server uses machine learning models such as TensorFlow and OpenCV to analyze and classify the emotional state (stress, satisfaction, etc.). The output is a label for the analyzed emotional state.

[1040] Step 5:

[1041] The server generates information based on the calculated congestion score and the analyzed emotional state data and provides it to the user's device. The congestion score and emotional state label are required as input. Based on this data, the server displays the congestion status and emotional state by color-coding them on the map data. As output, the color-coded map and information on recommended areas and time periods are displayed on the user's device.

[1042] Step 6:

[1043] The server periodically accumulates the user's movement history and emotion data to improve the accuracy of future analyses. As input, it receives the user's behavioral data and emotion data. The server analyzes this data and uses it to improve the accuracy of the congestion and emotion prediction model. As output, it obtains an improved congestion and emotion prediction model.

[1044] Specifically, when a user enters a prompt into the application such as "Please tell me the current congestion situation and recommended areas," the server analyzes the data in real time and provides the user with the most appropriate information.

[1045] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1046] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1047] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1048] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1049] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1050] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1051] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1052] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1053] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1054] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1055] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1056] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1057] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1058] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1059] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1060] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1061] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1062] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1063] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1064] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1065] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1066] The following is further disclosed regarding the above embodiment.

[1067] (Claim 1)

[1068] means for obtaining location data;

[1069] A means for analyzing the acquired location data and calculating a congestion degree;

[1070] means for providing the calculated congestion degree to a user;

[1071] A system including:

[1072] (Claim 2)

[1073] The system according to claim 1, further comprising means for displaying the congestion degree in different colors.

[1074] (Claim 3)

[1075] 10. The system of claim 1, further comprising means for pre-processing the collected data.

[1076] (Claim 4)

[1077] 10. The system of claim 1, further comprising means for collecting user feedback data.

[1078] (Claim 5)

[1079] 5. The system of claim 4, further comprising means for improving analysis accuracy using said user feedback data.

[1080] (Claim 6)

[1081] 10. The system of claim 1, further comprising means for updating the analysis results in real time.

[1082] (Claim 7)

[1083] The system of claim 1 , further comprising means for collecting said location data from a sensor.

[1084] "Example 1"

[1085] (Claim 1)

[1086] means for obtaining location data;

[1087] A means for preprocessing the acquired location data, formatting the location information data, filling in missing data, and excluding outliers;

[1088] A means for calculating congestion levels using a machine learning algorithm based on preprocessed data;

[1089] A means for dividing the calculated congestion scores into categories and generating a visually easy-to-understand color-coded map;

[1090] means for providing the generated congestion information to a user terminal;

[1091] A system including:

[1092] (Claim 2)

[1093] The system according to claim 1, further comprising means for displaying the congestion degree in different colors.

[1094] (Claim 3)

[1095] 2. The system according to claim 1, further comprising means for collecting user feedback data and using the data to improve the accuracy of congestion prediction.

[1096] "Application Example 1"

[1097] (Claim 1)

[1098] means for obtaining location data;

[1099] A means for analyzing the acquired location data and calculating a congestion degree;

[1100] means for providing the calculated congestion degree to a user;

[1101] A means for suggesting routes to avoid congestion based on the user's location information;

[1102] A means for collecting user feedback data to improve the accuracy of the analysis;

[1103] A system including:

[1104] (Claim 2)

[1105] 2. The system according to claim 1, further comprising means for displaying the degree of congestion in different colors.

[1106] (Claim 3)

[1107] 10. The system of claim 1, further comprising means for pre-processing the collected data.

[1108] "Example 2: Combining Emotion Engines"

[1109] (Claim 1)

[1110] means for obtaining location data;

[1111] means for pre-processing the collected location data;

[1112] A means for analyzing the preprocessed data to calculate a congestion degree;

[1113] a means for visually displaying the calculated congestion degree and providing it to a user;

[1114] means for recognizing a user's emotion;

[1115] a means for notifying a user based on emotion information and congestion information;

[1116] means for transmitting user movement data and emotion data to a server and providing feedback;

[1117] A system including:

[1118] (Claim 2)

[1119] 2. The system according to claim 1, further comprising means for displaying the degree of congestion in different colors.

[1120] (Claim 3)

[1121] 10. The system of claim 1, further comprising means for performing emotion recognition using a camera and a microphone.

[1122] "Application example 2 when combining emotion engines"

[1123] (Claim 1)

[1124] means for obtaining location data;

[1125] A means for analyzing the acquired location data and calculating a congestion degree;

[1126] A means for analyzing and classifying the calculated congestion level and the user's emotional state;

[1127] A means for recommending a comfortable area and time to the user based on the calculated congestion level and the analyzed emotional state;

[1128] A system including:

[1129] (Claim 2)

[1130] 2. The system of claim 1, further comprising means for displaying the congestion level and emotional state in a color-coded manner.

[1131] (Claim 3)

[1132] 10. The system of claim 1, further comprising means for pre-processing the collected location data and emotion data. [Explanation of symbols]

[1133] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. means for obtaining location data; A means for analyzing the acquired location data and calculating a congestion degree; means for providing the calculated congestion degree to a user; A system including:

2. 2. The system according to claim 1, further comprising means for displaying the congestion level in different colors.

3. 10. The system of claim 1, further comprising means for pre-processing the collected data.

4. 10. The system of claim 1, further comprising means for collecting user feedback data.

5. 5. The system of claim 4, further comprising means for improving analysis accuracy using said user feedback data.

6. 10. The system of claim 1, further comprising means for updating the analysis results in real time.

7. The system of claim 1 further comprising means for collecting said location data from a sensor.

Images