System

Abstract

A system is provided.SOLUTION: This system includes a means for collecting position information from a portable terminal, a means for collecting image data from a monitoring device, a means for analyzing the position information and the image data, and for calculating the congestion degree of each vehicle, and a means for calculating an optimal getting-on position based on the analysis result, and for notifying the portable terminal of the result.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Trains are extremely crowded during rush hour, with uneven occupancy rates in some cars. This not only causes discomfort for many passengers, but also hinders efficient commuting. Furthermore, there is no way for passengers to determine the optimal boarding location in real time, and their normal riding behavior exacerbates the congestion. To solve this problem, a system is needed that can grasp the congestion status of each car in real time and suggest the optimal boarding location to passengers. [Means for solving the problem]

[0005] The present invention provides a system including a means for collecting location information from a mobile device, a means for collecting image data from a monitoring device, a means for analyzing the location information and the image data to calculate the congestion level of each vehicle, and a means for calculating the optimal boarding position based on the analysis results and notifying the mobile device.Furthermore, by including a means for predicting future congestion conditions using past data and a means for obtaining feedback from users to improve the accuracy of the analysis and prediction, more effective and accurate boarding position suggestions can be realized.

[0006] A "mobile device" refers to a personal electronic device that can collect location information using GPS or beacons and has communication capabilities.

[0007] "Location information" refers to data such as latitude and longitude collected from a mobile terminal, and is information for identifying the user's current location.

[0008] "Monitoring equipment" refers to equipment used to acquire image data, such as cameras and sensors.

[0009] "Image data" refers to visual information acquired by a surveillance device, typically provided in the form of still images or video.

[0010] "Analysis" refers to the mathematical and computational means of using collected data to interpret information and arrive at a specific purpose or result.

[0011] "Crowding" refers to an indicator that shows the number of passengers and space occupancy in a particular area or vehicle.

[0012] The "optimal boarding position" refers to the position of the vehicle and its boarding door that is estimated to provide the most comfortable boarding experience for passengers.

[0013] "Notification" refers to the act of sending information about the calculated optimal boarding location to a mobile device and informing the user.

[0014] "Historical data" refers to previously collected and recorded information used to train and improve predictive algorithms.

[0015] "Predicting future congestion conditions" refers to the act of predicting the degree of congestion at a specific time in the future based on past data and current data.

[0016] "Feedback" refers to opinions and ratings collected from users regarding application usage and congestion, which are used to improve the accuracy of the system and the user experience. [Brief explanation of the drawings]

[0017] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11]FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0018] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0019] First, the terms used in the following description will be explained.

[0020] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0021] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0022] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0023] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0024] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0025] [First embodiment]

[0026] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0027] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0028] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0029] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0030] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0031] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0032] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0033] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0034] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0035] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0036] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0037] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0038] This invention is a system that uses location information from a mobile device and image data collected from monitoring devices in and around stations to appropriately guide passengers to the vehicle they are boarding. Specific embodiments based on the system program are described below.

[0039] Data Collection Phase

[0040] Collection of mobile phone location information

[0041] Device: Each user's mobile device periodically (for example, every 5 seconds) transmits its location information to the server. This information is sent using GPS or beacons.

[0042] User: Before arriving at the station, you need to enable location services on your mobile device.

[0043] Data collection from monitoring devices

[0044] Server: Collects image data in real time from monitoring devices (cameras and sensors) inside and around stations. This data is later used to analyze congestion levels.

[0045] Data analysis phase

[0046] Understanding congestion levels through image analysis

[0047] Server: Analyzes image data collected from the monitoring devices and runs algorithms to calculate the congestion level of each vehicle, including the number of passengers and the space occupancy rate.

[0048] Server: Based on the results of image analysis, classify each vehicle into categories such as "empty," "normal," or "crowded."

[0049] Integrated analysis of location information

[0050] Server: Aggregates location information from each user's mobile device and analyzes the location and destination information of all users, thereby estimating the current number of passengers in each vehicle.

[0051] Optimization Phase

[0052] Optimization calculation of boarding position

[0053] Server: Based on the collected location information and image data, the server calculates the optimal boarding location for the user. For example, an algorithm is designed to prioritize suggesting vehicles with low congestion levels.

[0054] Server: Refers to past congestion data and learns congestion patterns by time of day and day of the week to predict future congestion conditions.

[0055] Guide Phase

[0056] Notification of optimal pickup location

[0057] Server: Based on the calculation results, the server notifies each user of the optimal boarding location (vehicle and door location). Notifications are sent to the device in real time.

[0058] Device: Displays received notification information to the user and provides specific instructions in the form of text and maps.

[0059] Feedback Phase

[0060] Collecting post-ride feedback

[0061] User: After the ride, the user provides feedback through the app about the congestion situation during the ride, which is then sent to the server.

[0062] Terminal: Provides an interface for collecting feedback and accepts input from the user.

[0063] Processing Feedback

[0064] Server: Analyzes the received feedback and uses it to improve the accuracy of future congestion predictions and boarding location suggestions.

[0065] Specific examples

[0066] Morning rush hour scenario

[0067] 1. Device: User A leaves home and activates the location information on his mobile device on his way to the nearest station.

[0068] 2. Server: Obtains user A's location information and sends it to the server. At the same time, it uses augmented reality (AR) functions to confirm the user's location.

[0069] 3. Server: Acquires image data from station monitoring devices and analyzes it.

[0070] 4. Server: Based on the analysis results, it determines that car 3 is currently the vacant car, and notifies user A, "Car 3 is the vacant car. Please wait in front of car 3."

[0071] 5. Terminal: Display a message to user A, prompting him to move in front of car 3.

[0072] 6. User: Follow the instructions to stand in front of car 3, get in, and provide feedback through the app.

[0073] 7. Server: Receives User A's feedback and improves the accuracy of the next prediction based on the analysis results.

[0074] In this way, the system based on the present invention provides a concrete means for managing the congestion status of each vehicle in real time and providing passengers with a comfortable commuter life.

[0075] The processing flow will be explained below.

[0076] Step 1: Collect location information

[0077] Terminal: The mobile terminal periodically (for example, every 5 seconds) collects location information obtained from GPS and beacon sensors and prepares to send it to the server.

[0078] Step 2: Send location information

[0079] Device: Sends location information to the server, which notifies the server of the user's current location in real time.

[0080] Step 3: Collecting data from monitoring devices

[0081] Server: Image data is collected in real time from monitoring devices installed inside and around stations. This image data is used to understand the congestion status of each train car.

[0082] Step 4: Analyzing the image data

[0083] Server: Runs algorithms to analyze the collected image data. Specifically, it uses image analysis technology to calculate the number of passengers and space occupancy in each vehicle.

[0084] Step 5: Calculate congestion

[0085] Server: Based on the results of image analysis, the congestion level of each vehicle is classified into categories such as "empty," "normal," or "crowded."

[0086] Step 6: Integrated analysis of location information

[0087] Server: Aggregates location information sent from mobile devices and analyzes each passenger's current location and destination, thereby estimating the number of passengers in each vehicle.

[0088] Step 7: Calculate the best pickup location

[0089] Server: Based on the analysis results, the server runs an algorithm that calculates the optimal boarding location (vehicle and boarding door) for each user.

[0090] Step 8: Generate notifications

[0091] Server: Based on the calculation results, a message is generated to notify each user of the specific boarding location.

[0092] Step 9: Sending notifications

[0093] Server: Sends the generated message to the user's mobile device. Example: "Car 3 is the least crowded. Please wait in front of car 3."

[0094] Step 10: Displaying notifications

[0095] Terminal: Displays the received message to the user, so that the user can check which car to board.

[0096] Step 11: Ride

[0097] User: Follow the notification and move to the designated vehicle and get in.

[0098] Step 12: Provide feedback

[0099] User: After the ride, enter information about the ride's congestion and other feedback through the app.

[0100] Step 13: Submit your feedback

[0101] Terminal: Sends the feedback entered by the user to the server.

[0102] Step 14: Analyze feedback

[0103] Server: Analyzes the received feedback and uses it as data to improve the accuracy of future predictions.

[0104] Through these steps, the system can grasp the congestion status of each vehicle in real time and suggest the optimal boarding location to the user.

[0105] Example 1

[0106] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0107] In modern urban areas, congestion during commutes has become a major social problem. For commuters, especially those using trains, crowded trains can cause significant stress and discomfort. Conventional methods struggle to grasp congestion levels in real time and provide users with optimal boarding locations, resulting in a lack of information for passengers to determine the appropriate boarding location. Therefore, an effective system is needed to alleviate congestion and ensure a comfortable ride.

[0108] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0109] In this invention, the server includes means for collecting location information from the mobile device, means for collecting image data from the monitoring device, means for analyzing the location information and the image data to calculate the congestion level of each vehicle, means for learning congestion patterns by time period and day of the week based on past data and predicting future congestion conditions, means for calculating the optimal boarding position based on the analysis results and prediction results and notifying the mobile device, means for receiving feedback from the user and improving the accuracy of the analysis and prediction, and means for notifying the mobile device of the optimal boarding position in text or map format. This makes it possible to grasp the congestion status of each vehicle in real time and provide the user with optimal boarding guidance.

[0110] "Mobile device" refers to a portable electronic device that is carried by a user and used by the user, such as a mobile phone, smartphone, or tablet.

[0111] "Location Information" refers to data indicating a specific geographic location obtained through GPS, beacons, Wi-Fi, cell towers, etc.

[0112] "Monitoring devices" refers to devices such as cameras, sensors, and beacons installed in or around stations that monitor the surrounding conditions and collect and provide data.

[0113] "Image data" refers to visual data in the form of still images or video captured or acquired by surveillance equipment.

[0114] "Crowding level" is an indicator of the density of people in a particular area or vehicle, and is calculated based on the number of passengers and the occupancy rate of space.

[0115] "Analysis" refers to the process of analyzing collected data in detail to extract meaning and patterns and derive results according to the intended purpose.

[0116] "Boarding location" refers to the specific car and its door location that the user selects to board the train.

[0117] "Notification" refers to the act of sending a message to a mobile device to notify the user of the results of a calculation or analysis.

[0118] "Past data" refers to data collected in the past, and primarily refers to past congestion conditions and user usage patterns.

[0119] "Feedback" is input data provided by users based on their experiences and opinions, which is used to reevaluate and improve the system.

[0120] "Forecasting" refers to the process of predicting and estimating future situations based on collected data and analysis results.

[0121] This invention is a system that provides optimal boarding guidance to passengers by utilizing location information from mobile devices and image data acquired from monitoring devices in and around stations. This system is implemented using the following means.

[0122] Data Collection Phase

[0123] Collection of mobile phone location information

[0124] Device: Each user's mobile device periodically transmits location information to the server. Location information is collected using GPS or beacon technology (e.g., Bluetooth beacons). Specifically, the device activates location services, measures its current location every 5 seconds, and transmits it to the server using a secure communication protocol (e.g., HTTPS).

[0125] User: Before arriving at the station, the user must enable location services on their mobile device and grant location sharing permission to the app.

[0126] Data collection from monitoring devices

[0127] Server: The server collects image data in real time from monitoring devices inside and around stations. The monitoring devices include cameras and sensors, and receives image data as a stream from these devices. The received data is temporarily stored in storage and used for subsequent analysis.

[0128] Data analysis phase

[0129] Understanding congestion levels through image analysis

[0130] Server: The server uses image analysis algorithms such as OpenCV and TensorFlow to analyze the collected image data. Specifically, it detects passengers, counts the number of people, and analyzes space occupancy for each frame, and classifies the congestion level of each car into categories of "empty," "normal," or "crowded."

[0131] Server: The server stores these analysis results and uses them for subsequent pickup location suggestions.

[0132] Integrated analysis of location information

[0133] Server: The server aggregates the location information collected from each user's mobile device and plots it on a map, allowing the server to determine the current number of passengers in each car and the number of passengers in a specific area of ​​the station.

[0134] Optimization Phase

[0135] Optimization calculation of boarding position

[0136] Server: The server calculates the optimal boarding location for each user based on the aggregated location information and analyzed image data. The algorithm references past congestion data and learns congestion patterns by time of day and day of the week to predict future congestion conditions.

[0137] Server: Based on the calculation results, the optimal boarding location is determined and saved.

[0138] Guide Phase

[0139] Notification of optimal pickup location

[0140] Server: Based on the calculation results, the server notifies each user of the optimal boarding location in real time. The notification content includes specific instructions, such as "Car 3 is the least crowded. Please wait in front of car 3."

[0141] Device: The device displays the received notification to the user in an intuitive format, such as a text message or a pin on a map.

[0142] Feedback Phase

[0143] Collecting post-ride feedback

[0144] User: After the ride, the user provides feedback through the app about the ride's congestion, with options such as "very crowded," "normal," and "empty."

[0145] Terminal: The terminal provides an interface for collecting feedback and accepts input from the user, which is then sent to the server.

[0146] Processing Feedback

[0147] Server: The server analyzes the received feedback and improves the accuracy of congestion predictions and boarding location suggestions for the next trip. The analysis integrates the feedback items and performs statistical evaluation.

[0148] Specific examples

[0149] Morning rush hour scenario

[0150] 1. Device: User A leaves home and activates the location information service on his mobile device on his way to the nearest station.

[0151] 2. Server: The server receives and stores the location information of User A. At the same time, it visually confirms User A's location using the augmented reality (AR) function.

[0152] 3. Server: The server acquires image data from the station monitoring devices and analyzes it in real time.

[0153] 4. Server: Based on the analysis results, it determines that car 3 is the least crowded and notifies user A, "Car 3 is the least crowded. Please wait in front of car 3."

[0154] 5. Terminal: The terminal displays a notification to User A, instructing him to move to the front of car 3. The notification is indicated by a text message and a pin on the map.

[0155] 6. User: User A follows the instructions to stand in front of car 3 and provides feedback on the congestion situation through the app after boarding.

[0156] 7. Server: The server receives User A's feedback and uses the analysis results to improve the accuracy of the next prediction.

[0157] Prompt Sentence Examples

[0158] Prompt: "Describe the steps for a system that uses station surveillance cameras and mobile device location information to provide optimal ride guidance."

[0159] This system is designed to grasp the congestion status of each vehicle in real time and provide users with a comfortable commute.

[0160] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0161] Step 1:

[0162] Collection of mobile phone location information

[0163] Input: Location information obtained from the mobile device's built-in sensors (GPS data, beacon signals, etc.)

[0164] Device behavior: The device activates location services, measures its current location every 5 seconds, and sends the measurements to a server using a secure communication protocol such as HTTPS.

[0165] Output: The user's current location is sent to the server.

[0166] Step 2:

[0167] Image data collection from monitoring devices

[0168] Input: Real-time image streams from surveillance devices (cameras, sensors) installed in and around the station

[0169] Server operation: The server establishes a secure connection with the monitoring device, receives image data in real time, and temporarily stores the received data in storage.

[0170] Output: Image data saved in storage

[0171] Step 3:

[0172] Understanding congestion levels through image analysis

[0173] Input: Image data stored in storage

[0174] Server operation: The server analyzes image data using image analysis algorithms such as OpenCV and TensorFlow. It detects passengers, counts the number of people, and analyzes space occupancy for each frame.

[0175] Output: Analyzed congestion data (e.g., empty, normal, crowded)

[0176] Step 4:

[0177] Congestion category classification for each vehicle

[0178] Input: Crowd data obtained by image analysis

[0179] Server operation: The server classifies each vehicle into a category such as "empty," "normal," or "crowded" based on the congestion data, using a set threshold.

[0180] Output: Congestion category information for each vehicle

[0181] Step 5:

[0182] Integrated analysis of location information

[0183] Input: Location data from each user's mobile device

[0184] Server operation: The server aggregates location information and plots it on a map. It analyzes overall trends and calculates the number of users concentrated in a particular car or in a particular area of ​​the station.

[0185] Output: Integrated analysis results based on user location information

[0186] Step 6:

[0187] Optimization calculation of boarding position

[0188] Input: Integrated analysis results of location information and congestion category information

[0189] Server operation: The server uses an algorithm to calculate the optimal boarding location for the user. It learns from past data and congestion patterns by time of day and day of the week, and runs an algorithm to predict future congestion conditions.

[0190] Output: Optimal pickup location information

[0191] Step 7:

[0192] Notification of optimal pickup location

[0193] Input: Best Pickup Location

[0194] Server operation: Based on the calculation results, the server notifies each user of the optimal boarding location in real time. The notification message includes specific instructions such as "Car 3 is the least crowded. Please wait in front of car 3."

[0195] Device behavior: The device displays notifications to the user in intuitive formats such as text messages or pins on a map.

[0196] Output: A notification message that is displayed to the user

[0197] Step 8:

[0198] Collecting post-ride feedback

[0199] Input: User feedback information

[0200] User Action: After the ride, the user provides feedback through the app about the ride's congestion, with options such as "very crowded," "normal," and "empty."

[0201] Device behavior: The device provides an interface for collecting feedback and accepts input from the user, which is then sent to the server.

[0202] Output: Feedback data sent to the server

[0203] Step 9:

[0204] Processing Feedback

[0205] Input: Feedback data

[0206] Server operation: The server analyzes the received feedback and uses it to improve the accuracy of future congestion predictions and boarding location suggestions. The analysis integrates the feedback items and performs statistical evaluation.

[0207] Output: Improved congestion prediction data and pickup location suggestions

[0208] (Application example 1)

[0209] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0210] Technologies that grasp the congestion situation in and around train stations and guide passengers to the optimal boarding location are widely used, but the application of these technologies to avoiding congestion in commercial facilities has not been sufficiently considered. While the importance of avoiding congestion in such commercial facilities is increasing, current systems lack the technology to analyze the congestion situation in real time and provide users with appropriate routes and areas. The objective of this invention is to provide a system that provides appropriate congestion avoidance guidance in order to provide a comfortable shopping experience in commercial facilities.

[0211] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0212] In this invention, the server includes means for collecting location information from mobile devices, means for collecting image data from monitoring devices, means for analyzing the location information and the image data to calculate the congestion level of each area, means for calculating the optimal route and area based on the analysis results and notifying the mobile devices, means for obtaining feedback from users and improving the accuracy of the analysis and prediction, and means for predicting future congestion conditions using past data. This makes it possible to grasp the congestion status in commercial facilities in real time and provide appropriate congestion avoidance guidance.

[0213] "Mobile device" refers to a device that a user can carry with them and that can transmit location information, such as a mobile phone, smartphone, or tablet.

[0214] "Location information" refers to data transmitted by a mobile device indicating its current geographical location, and is information obtained using GPS or beacons.

[0215] "Surveillance equipment" refers to devices such as cameras and sensors installed in commercial facilities or specific areas that monitor the surrounding situation in real time and collect image data.

[0216] "Image data" refers to video and photographic data collected by surveillance equipment, and is information that can be analyzed to determine the level of congestion in an area.

[0217] "Crowding level" is an index that indicates the density or occupancy rate of people in a specific area, and refers to information calculated from analyzed image data.

[0218] "Calculating routes and areas" refers to the process of deriving the optimal travel route and areas to visit for a user based on collected location information and image data.

[0219] "Means of notification" refers to the method of sending the calculated optimal route and area guidance information to the user's mobile device and displaying it.

[0220] "Feedback" refers to information provided by users about their experiences and opinions, which is collected and analyzed to help improve the accuracy of the system.

[0221] "Prediction means" refers to algorithms and methods for referencing past congestion data and estimating future congestion conditions.

[0222] "Commercial facilities" refers to facilities such as shopping malls, department stores, and shops that are visited by many people and may become crowded.

[0223] This invention relates to a system that provides guidance for avoiding crowds in commercial facilities. It integrates location information from mobile devices and image data collected from monitoring devices to guide users to the optimal route and area in real time. It is also a system that aims to further improve accuracy by collecting feedback from users.

[0224] Data Collection Phase

[0225] Mobile location information collection:

[0226] Device: Each user's mobile device periodically sends its location information to the server, using GPS or beacons.

[0227] Users: Must enable location services on their mobile device before arriving at the commercial establishment.

[0228] Data collection from monitoring devices:

[0229] Server: Collects image data in real time from monitoring devices (cameras and sensors) installed within commercial facilities. This data is used to analyze congestion levels.

[0230] Data Analysis Phase

[0231] Image analysis to understand congestion levels:

[0232] Server: Analyzes image data collected from the monitoring devices and runs an algorithm to calculate the congestion level of each area, specifically the number of users and the occupancy rate of the space.

[0233] Server: Based on the results of image analysis, the congestion level of each area is classified as "low," "normal," "high," etc.

[0234] Integrated location analysis:

[0235] Server: Aggregates location information from each user's mobile device and analyzes the location and destination information of all users to estimate the current congestion situation in each area.

[0236] Route and area optimization phase

[0237] Best route and area suggestions:

[0238] Server: Calculates the optimal route and area for the user based on the collected location information and image data. An algorithm is built in to prioritize suggesting areas with low congestion.

[0239] Server: Refers to past congestion data, learns congestion patterns by time of day and day of the week, and predicts future congestion conditions.

[0240] Guide Phase

[0241] Best route and area notifications:

[0242] Server: Based on the calculation results, the server notifies each user of the optimal route and area information. Notifications are sent to the device in real time.

[0243] Device: Displays received notification information to the user and provides specific instructions in the form of text and maps.

[0244] Feedback Phase

[0245] Collecting post-ride feedback:

[0246] User: After moving around the shopping mall, the user provides feedback about the congestion situation through the app, which is then sent to the server.

[0247] Terminal: Provides an interface for collecting feedback and accepts input from the user.

[0248] Feedback processing:

[0249] Server: Analyzes the received feedback and uses it to improve the accuracy of congestion predictions and route guidance for future trips.

[0250] Hardware and software used:

[0251] Hardware: smartphones, surveillance cameras, sensors.

[0252] Software: Python, image analysis algorithm, location analysis algorithm, external API (for acquiring congestion data).

[0253] Examples:

[0254] A user arrives at a shopping mall at 10:00 a.m. After arriving at shopping mall A, the user opens the app and is directed to the least crowded area. After enjoying shopping, the user provides feedback to the app.

[0255] Example prompt: "Current location: 10:00. Please guide me to the areas in Shopping Mall A. Please recommend the least crowded area, taking into account the congestion status of each area. After making the recommendation, please also display a prompt to provide feedback to the user. The feedback content should be an interface where the user can write about the congestion status of the shopping experience."

[0256] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0257] Step 1:

[0258] Collection of mobile phone location information

[0259] Input: The user enables location services on their mobile device.

[0260] Processing: The device periodically (for example, every 5 seconds) obtains its own location information using GPS or beacons and generates location information data.

[0261] Output: The acquired location data is sent to the server.

[0262] Specific operation: The GPS module in the smartphone measures the current location, and the terminal app collects that data and sends it to a specified server.

[0263] Step 2:

[0264] Data collection from monitoring devices

[0265] Input: Image data provided by surveillance equipment within a commercial facility.

[0266] Processing: The server collects image data from the monitoring devices in real time and prepares the image data for storage and processing.

[0267] Output: The collected image data is stored in the server.

[0268] Specific operation: The server receives the video captured by the surveillance camera and stores it in storage. The image data is also organized in a specific format.

[0269] Step 3:

[0270] Understanding congestion levels through image analysis

[0271] Input: Collected image data.

[0272] Processing: The server uses image analysis algorithms to calculate the congestion level of each area, which is the process of calculating the number of users and the occupancy rate of the space.

[0273] Output: Obtains congestion information for each area.

[0274] What it does: Server-supported image analysis software counts the number of people in the image and calculates the area occupancy rate.

[0275] Step 4:

[0276] Integrated analysis of location information

[0277] Input: Location data and congestion information from user devices.

[0278] Processing: The server combines the location data and congestion information to estimate the overall congestion situation for each area.

[0279] Output: Consolidated congestion data is obtained.

[0280] What it does: The server uses a geographic information system (GIS) to map users' locations and area congestion levels, and then aggregates the data.

[0281] Step 5:

[0282] Optimal route and area suggestions

[0283] Input: Consolidated crowding data.

[0284] Processing: The server uses an algorithm to calculate the best route and area for each user, prioritizing less congested areas.

[0285] Output: The calculated optimal route and area information is obtained.

[0286] Specific operation: The server uses specific rules and algorithms (e.g., shortest path algorithms or congestion-based optimization methods) to determine the most suitable route and area for the user.

[0287] Step 6:

[0288] Optimal route and area notifications

[0289] Input: Calculated optimal route and area information.

[0290] Processing: The server sends the optimal route and area information to the user's mobile device.

[0291] Output: A real-time notification is displayed on the user's mobile device.

[0292] What it does: Uses the notification system (push notifications and in-app notifications) to display text and map-based directions on the user's smartphone.

[0293] Step 7:

[0294] Collecting and Processing Feedback

[0295] Input: Feedback information provided by the user.

[0296] Processing: The device provides an interface for collecting feedback and accepts user input. The server analyzes the received feedback and uses it to improve the accuracy of future guidance.

[0297] Output: Parsed feedback data.

[0298] Specific operation: Collects information provided by users through feedback forms and survey functions within the app, and the server analyzes the data to perform feedback analysis.

[0299] Prompt Sentence Examples

[0300] "Current location: 10:00. Please guide the user to the areas in the shopping mall. Please recommend the least crowded area, taking into account the congestion level of each area. After making the recommendation, please also display a prompt to provide feedback to the user. The feedback should be an interface where the user can describe how crowded the shopping experience was."

[0301] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0302] The present invention combines an emotion engine with a system that uses location information from mobile devices and image data collected from monitoring devices in and around stations to provide appropriate guidance on which vehicle to board. Specific embodiments based on the system program are described below.

[0303] Data Collection Phase

[0304] Collection of mobile phone location information

[0305] Device: Each user's mobile device periodically (for example, every 5 seconds) transmits its location information to the server. This information is sent using GPS or beacons.

[0306] User: Before arriving at the station, you need to enable location services on your mobile device.

[0307] Data collection from monitoring devices

[0308] Server: Collects image data in real time from monitoring devices (cameras and sensors) installed inside and around stations. This image data is used to understand the congestion status of each train car.

[0309] Data analysis phase

[0310] Understanding congestion levels through image analysis

[0311] Server: Runs algorithms to analyze the collected image data. Specifically, it uses image analysis technology to calculate the number of passengers and space occupancy in each vehicle.

[0312] Server: Based on the results of image analysis, the congestion level of each vehicle is classified into categories such as "empty," "normal," or "crowded."

[0313] Integrated analysis of location information

[0314] Server: Aggregates location information from each user's mobile device and analyzes the location and destination information of all users, thereby estimating the number of passengers in each vehicle.

[0315] Optimization Phase

[0316] Optimization calculation of boarding position

[0317] Server: Based on the analysis results, the server runs an algorithm that calculates the optimal boarding location (vehicle and boarding door) for each user.

[0318] Server: Refers to past congestion data and learns congestion patterns by time of day and day of the week to predict future congestion conditions.

[0319] Emotion Recognition Phase

[0320] Using the Emotion Engine

[0321] Device: The emotion engine recognizes the user's emotions using facial expression analysis via a camera, voice analysis, and the user's operation patterns.

[0322] Server: Analyzes the emotional data obtained from the emotion engine and understands the user's current emotional state.

[0323] Emotion-Based Optimization

[0324] Server: Integrates the user's emotional data and analysis results, and adjusts the recommended ride locations based on the user's emotional state. For example, for users with high levels of stress or anxiety, it will prioritize suggesting less crowded vehicles.

[0325] Guide Phase

[0326] Notification of optimal pickup location

[0327] Server: Based on the calculation results, it generates a message to notify each user of the optimal boarding location.

[0328] Device: Displays received notification information to the user and provides specific instructions in the form of text and maps.

[0329] Feedback Phase

[0330] Collecting post-ride feedback

[0331] User: After the trip, provide feedback through the app about the ride's congestion and emotional state.

[0332] Terminal: Provides an interface for collecting feedback and accepts input from the user.

[0333] Processing Feedback

[0334] Server: Analyzes the received feedback and uses it as data to improve prediction accuracy and emotion recognition accuracy in future trials.

[0335] Specific examples

[0336] Morning rush hour scenario

[0337] 1. Device: User A leaves home and activates the location information on his mobile device on his way to the nearest station.

[0338] 2. Server: Obtains user A's location information and sends it to the server. At the same time, it uses augmented reality (AR) functions to confirm the user's location.

[0339] 3. Server: Acquires image data from station monitoring devices and analyzes it.

[0340] 4. Server: Based on the analysis results, it determines that car 3 is currently the vacant car, and notifies user A, "Car 3 is the vacant car. Please wait in front of car 3."

[0341] 5. Device: The emotion engine analyzes user A's emotion data and detects his / her stress state.

[0342] 6. Server: Consider user A’s emotional state and further adjust the pickup location suggestions.

[0343] 7. Terminal: Display a message to user A, prompting him to move in front of car 3.

[0344] 8. User: Follow the instructions to stand in front of car 3, get in, and provide feedback through the app.

[0345] 9. Server: Receives User A's feedback and improves the accuracy of the next prediction based on the analysis results.

[0346] In this way, the system based on the present invention provides a concrete means for providing users with a comfortable commuter life by managing the congestion status of each vehicle in real time and proposing the optimal boarding location taking into account the user's emotional state.

[0347] The processing flow will be explained below.

[0348] Step 1: Collect location information

[0349] Terminal: The mobile terminal periodically (for example, every 5 seconds) collects location information obtained from GPS and beacon sensors and prepares to send it to the server.

[0350] Step 2: Send location information

[0351] Device: Sends location information to the server, which notifies the server of the user's current location in real time.

[0352] Step 3: Collecting data from monitoring devices

[0353] Server: Obtains image data in real time from monitoring devices (cameras and sensors) installed inside and around stations. This image data is used to understand the congestion status of each train car.

[0354] Step 4: Analyzing the image data

[0355] Server: Runs algorithms to analyze the collected image data. Specifically, it uses image analysis technology to calculate the number of passengers and space occupancy in each vehicle.

[0356] Step 5: Calculate congestion

[0357] Server: Based on the results of image analysis, the congestion level of each vehicle is classified into categories such as "empty," "normal," or "crowded."

[0358] Step 6: Integrated analysis of location information

[0359] Server: Aggregates location information sent from mobile devices and analyzes each passenger's current location and destination, thereby estimating the number of passengers in each vehicle.

[0360] Step 7: Calculate the best pickup location

[0361] Server: Based on the analysis results, the server runs an algorithm that calculates the optimal boarding location (vehicle and boarding door) for each user.

[0362] Step 8: Use historical data

[0363] Server: Refers to past congestion data and learns congestion patterns by time of day and day of the week to predict future congestion conditions.

[0364] Step 9: Emotion Recognition with the Emotion Engine

[0365] Device: The emotion engine recognizes the user's emotions by analyzing facial expressions via a camera, voice analysis, and the user's operation patterns.

[0366] Step 10: Sending Emotion Data

[0367] Device: Sends the recognized emotion data to the server, which informs the server of the user's current emotional state.

[0368] Step 11: Adjusting riding position based on emotions

[0369] Server: Integrates the user's emotional data and analysis results to tailor the recommended ride locations based on their emotional state. For example, prioritize less crowded vehicles for users with high levels of stress.

[0370] Step 12: Generate and send notifications

[0371] Server: Based on the calculation results and emotion data, a message is generated to notify each user of their specific boarding location and sent to the device.

[0372] Step 13: Viewing Notifications

[0373] Terminal: Displays the received message to the user, so that the user can check which car they should board.

[0374] Step 14: Ride

[0375] User: Follow the notification and move to the designated vehicle and get in.

[0376] Step 15: Provide feedback

[0377] User: After the ride, users provide feedback through the app about the ride's congestion and emotional state.

[0378] Step 16: Submit your feedback

[0379] Terminal: Sends the feedback entered by the user to the server.

[0380] Step 17: Analyze feedback

[0381] Server: Analyzes the received feedback and uses it as data to improve prediction accuracy and emotion recognition accuracy in future trials.

[0382] Through these steps, the system can grasp the congestion status of each vehicle in real time and suggest the optimal boarding location taking into account the user's emotional state.

[0383] Example 2

[0384] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0385] Conventional ride guidance systems were able to grasp the congestion level of each vehicle using location information and image data, but did not provide ride guidance that took into account the user's emotional state. As a result, riding in a crowded vehicle could cause stress and discomfort to the user. It was particularly difficult to provide a comfortable ride experience for users during rush hour situations.

[0386] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for collecting location information from a mobile terminal, means for collecting image data from a monitoring device, means for analyzing the location information and the image data to calculate the congestion level of each vehicle, means for grasping the emotional state of the user using an emotion recognition engine, and means for calculating the optimal boarding position based on the analysis result and the emotional state and notifying the mobile terminal. This makes it possible to propose an optimal boarding position taking into account the user's emotional state in addition to the user's location information and congestion status.

[0387] A "mobile terminal" is a communication device carried by a user that provides location information and emotional state data.

[0388] "Location information" refers to geographic coordinate information collected from mobile devices and GPS sensors.

[0389] "Monitoring devices" are equipment such as cameras and sensors that are installed in or around stations to collect image data.

[0390] "Image data" refers to video and photo data acquired from a surveillance device.

[0391] "Analysis" involves processing the collected data and calculating the congestion level of each vehicle and the emotional state of the user.

[0392] "Crowding level" refers to the degree of congestion based on the number of passengers and space occupancy in each vehicle.

[0393] An "emotion recognition engine" is software for analyzing a user's emotional state, including facial expression analysis and voice analysis using a camera and microphone.

[0394] "Notification" refers to the act of sending the calculated optimal boarding location to the user's mobile device.

[0395] "Feedback" is information provided by users about the congestion situation and their emotional state after riding.

[0396] This invention combines an emotion engine with a system that uses location information from mobile devices and image data collected from monitoring devices in and around stations to provide appropriate guidance on which vehicle to board. Specific embodiments of the system are described below.

[0397] Data Collection Phase

[0398] Collection of mobile phone location information

[0399] Device: Each user's mobile device sends its location information to the server every 5 seconds. This information is obtained using GPS or beacons. Specifically, the device's built-in GPS sensor is used to obtain location information, which is then compressed and sent to the server.

[0400] User: Before arriving at the station, they will need to enable location services on their mobile device. A notification will appear on the device prompting them to enable the service.

[0401] Data collection from monitoring devices

[0402] Server: Collects image data in real time from monitoring devices (cameras and sensors) installed in and around the station. Specifically, it acquires video streams from cameras installed on platforms and at entrances and exits.

[0403] Server: Temporarily stores collected image data in storage for subsequent analysis.

[0404] Data analysis phase

[0405] Understanding congestion levels through image analysis

[0406] Server: Executes algorithms to analyze collected image data. Specifically, it applies image analysis technology using deep learning to calculate the number of passengers in each vehicle and the occupancy rate of space. For example, it identifies the positions of passengers' faces and bodies based on images acquired from surveillance cameras and calculates the degree of congestion.

[0407] Server: Based on the calculated congestion level, each vehicle is classified into a category such as "empty," "normal," or "crowded." This classification result is stored in a database and used in subsequent phases.

[0408] Integrated analysis of location information

[0409] Server: Aggregates location information from each user's mobile device and analyzes the location and destination information of all users, using a location clustering algorithm to identify areas where passengers are concentrated.

[0410] Server: Combines location and congestion data to calculate the expected number of passengers in each vehicle.

[0411] Emotion Recognition Phase

[0412] Using the Emotion Engine

[0413] On-device: The device uses an emotion recognition engine to analyze the user's emotional state. Specifically, the device uses the camera to analyze facial expressions and the microphone to analyze voices. This allows the device to identify emotional states such as stress or anxiety.

[0414] Server: Analyzes the emotional data sent from the emotion recognition engine and understands the user's emotional state. This makes it possible to suggest the optimal boarding location based on the user's emotional state.

[0415] Guide Phase

[0416] Notification of optimal pickup location

[0417] Server: Calculates the optimized boarding location (specific vehicle and boarding door) and generates customized instruction messages for each user.

[0418] Terminal: The generated notification message is displayed to the user. For example, specific instructions such as "Currently, car number 3 is the vacant car. Please wait in front of car number 3" are displayed in text or map format.

[0419] Feedback Phase

[0420] Collecting post-ride feedback

[0421] Users: After their ride, they provide feedback through the app about the occupancy and their emotional state, specifically in the form of a short survey about their ride experience.

[0422] Terminal: Provides a feedback input interface and accepts data from the user.

[0423] Processing Feedback

[0424] Server: Analyzes the received feedback data and performs batch processing to improve prediction accuracy and emotion recognition accuracy from the next time onwards. This is expected to improve the accuracy of the entire system.

[0425] Specific examples

[0426] Morning rush hour scenario

[0427] 1. Device: User A leaves home and activates the location information on his mobile device on his way to the nearest station.

[0428] 2. Server: Obtains user A's location information and sends it to the server. At the same time, the server confirms the user's location using AR functions, etc.

[0429] 3. Server: Acquires image data from station monitoring devices and analyzes it.

[0430] 4. Server: Based on the analysis results, it determines that car 3 is currently the vacant car, and notifies user A, "Car 3 is the vacant car. Please wait in front of car 3."

[0431] 5. Device: The emotion engine analyzes user A's emotion data and detects his / her stress state.

[0432] 6. Server: Consider user A’s emotional state and further adjust the pickup location suggestions.

[0433] 7. Terminal: Display a message to user A, prompting him to move in front of car 3.

[0434] 8. User: Follow the instructions to stand in front of car 3, get in, and provide feedback through the app.

[0435] 9. Server: Receives User A's feedback and improves the accuracy of the next prediction based on the analysis results.

[0436] In an embodiment based on this scenario, user location information, station monitoring data, and emotion data can be integrated in real time to provide users with a comfortable riding experience.

[0437] Prompt Sentence Examples

[0438] Below are some example prompts to input to the generative AI model:

[0439] "Please describe in detail the process of a system that uses mobile device location information and station surveillance camera data to calculate the optimal boarding location for passengers. This should include data collection, analysis phase, use of an emotion engine, optimization, notification phase, and user feedback processing."

[0440] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0441] Step 1:

[0442] Device: The user's mobile device enables location services. Specifically, the device's GPS module acquires its current location every 5 seconds and transmits the location data to the server. This provides the server with the user's real-time location information as input data.

[0443] Step 2:

[0444] Server: Collects image data from surveillance devices (cameras and sensors) installed in and around the station. Specifically, it acquires video streams from each surveillance camera in real time and temporarily stores them in storage. This image data is used in the next image analysis phase.

[0445] Step 3:

[0446] Server: Analyzes the collected image data. Specifically, it applies image analysis technology using deep learning to calculate the number of passengers in each vehicle and the occupancy rate of space. It uses an object detection algorithm to identify and count passengers from the input image data. The analysis results output the degree of congestion in each vehicle.

[0447] Step 4:

[0448] Server: Based on the congestion analysis results, classify each vehicle into a category such as "empty," "normal," or "crowded." The input is the analysis result from step 3, which is applied to a category classification algorithm to output the classification result. This classification result is stored in a database.

[0449] Step 5:

[0450] Server: Aggregates location information sent from each user's mobile device and analyzes the overall location and destination information. Specifically, it uses a location clustering algorithm to identify areas where users are concentrated. The input is real-time location data, and the output is passenger density information for each area.

[0451] Step 6:

[0452] Device: The device uses an emotion recognition engine to analyze the user's emotional state. Specifically, facial expression analysis is performed using the device's camera, and voice analysis is performed using the microphone. The input is the user's facial expression image and voice data, and the output is the evaluation result of the user's emotional state.

[0453] Step 7:

[0454] Server: Analyzes the emotion data sent from the emotion recognition engine and understands the user's emotional state. The input is the emotion engine's output data, and analysis is performed based on this to evaluate the user's emotional state. Based on this evaluation result, the boarding location suggestions are optimized.

[0455] Step 8:

[0456] Server: Calculates the optimal pickup location based on the user's emotional state and generates a personalized notification message. The inputs are congestion data, user location data, and the user's emotional state data, which are combined to run an algorithm to calculate the optimal pickup location. The output is a notification message.

[0457] Step 9:

[0458] Terminal: Displays notification messages received from the server to the user. Specifically, it displays boarding location instructions in text and map format. For example, it displays a message such as "Currently, car 3 is the least crowded. Please wait in front of car 3." The input is the notification message from the server, and the output is the notification to the user.

[0459] Step 10:

[0460] User: Follows the instructions to go to the designated pickup location. After boarding, the user provides feedback on the congestion situation and emotional state during the ride through the app. The input is the user's experience data, and the output is feedback information.

[0461] Step 11:

[0462] Server: Analyzes the feedback data received from users and improves the algorithms to improve prediction accuracy and emotion recognition accuracy in the future. The input is user feedback data, and the output is an improved analysis and prediction algorithm.

[0463] (Application example 2)

[0464] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0465] The issue of congestion in public transportation and the inability to properly guide passengers to a comfortable boarding location are issues. Furthermore, providing boarding guidance without taking into account the passenger's emotional state increases stress and discomfort, degrading the overall riding experience.

[0466] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for collecting location information from the mobile terminal, means for collecting image data from the monitoring device, means for analyzing the location information and the image data to calculate the congestion level of each vehicle, means for analyzing the emotional state of the user, and means for calculating the optimal boarding position based on the analysis result and the emotional state and notifying the mobile terminal. This makes it possible to suggest the optimal boarding position taking into consideration the congestion level of the vehicle and the emotional state of the user, thereby improving the overall riding experience.

[0467] A "mobile terminal" is an electronic device that can be carried by a user and has GPS and Internet connectivity capabilities.

[0468] "Location information" is information that indicates coordinate data at a specific location using a GPS, beacon, or the like.

[0469] A "surveillance device" is a device that includes a camera, sensor, etc., and collects image data and environmental data of a designated area in real time.

[0470] "Image data" is visual information, including still images and video, collected by surveillance equipment.

[0471] "Crowding" is an indicator that expresses the density of people and objects in a particular area or space using a number or category.

[0472] "Emotional state" is data that indicates an individual's psychological state, and is obtained by facial expression analysis and voice analysis via a camera.

[0473] "Analysis results" are information obtained as a result of calculations and evaluations performed on collected data.

[0474] "Forecasting" is the process of predicting future conditions or events based on past and current data.

[0475] "Optimal boarding position" refers to the recommended position or door for the user to board comfortably and efficiently.

[0476] A "notification" is information sent from the system to the user, and may be provided in text, audio, visual format, or the like.

[0477] "Feedback" refers to information such as opinions, evaluations, and impressions collected from users, and is used to improve the system and increase the accuracy of analysis.

[0478] The "smart ride guidance system" of the present invention is a system that collects location information from mobile devices and image data from monitoring devices, and guides users to the optimal ride location taking into account their emotional state. A specific method for implementing this system is described below.

[0479] Data Collection Phase

[0480] Collection of location information from mobile devices

[0481] The terminal is an electronic device (such as a smartphone) carried by the user, which periodically collects location information using its GPS function and transmits it to the server. Location information collection begins when the user is near a station or an autonomous vehicle, and the server receives this information in real time.

[0482] Collection of image data from monitoring devices

[0483] The server collects image data from cameras and other sensors installed in stations and on autonomous vehicles. These devices provide the real-time visual information needed to understand the occupancy level of each carriage.

[0484] Data analysis phase

[0485] Understanding congestion levels through image analysis

[0486] The server uses image analysis algorithms (e.g., OpenCV) to analyze the collected image data, which calculates the number of passengers and space occupancy in each vehicle and classifies the level of congestion into categories of "empty," "normal," or "crowded."

[0487] Emotional state analysis

[0488] The device uses an emotion engine (e.g., Microsoft's Azure Emotion API or Google Cloud's Vision API) to analyze the user's emotions. By analyzing facial expressions and voice via the device's camera and the user's operation patterns, the device detects emotional states such as stress, anxiety, and comfort.

[0489] Optimization Phase

[0490] Optimization calculation of boarding position

[0491] The server combines congestion data and the user's emotional state to calculate the optimal boarding location (car and boarding door) for each user. This calculation uses an algorithm that references past congestion data and learns congestion patterns by time of day and day of the week.

[0492] Guide Phase

[0493] Notification of optimal pickup location

[0494] The server generates a message informing each user of the optimal boarding location based on the calculation results, and the terminal displays this notification information to the user, providing specific instructions in the form of text and maps.

[0495] Feedback Phase

[0496] Collecting post-ride feedback

[0497] The terminal provides an interface to accept feedback from users after their ride. Users can provide feedback through the app regarding the congestion situation and emotional state during the ride.

[0498] The server analyzes the feedback it receives and uses it as data to improve prediction accuracy and emotion recognition accuracy in future trials.

[0499] Specific examples

[0500] For example, during the morning rush hour, User A sends location information from his mobile device on his way from his home to the nearest station. The device then uses an emotion engine to detect User A's stress level. The server then suggests the least crowded car based on the congestion data within the station and User A's emotional state, and notifies the device, "Car 3 is the least crowded. Please wait in front of Car 3." As a result, User A can commute comfortably without feeling stressed.

[0501] Prompt Sentence Examples

[0502] "Design an AI model to improve the accuracy of emotion recognition. The goal is to analyze the user's emotional state in real time and suggest the optimal boarding location based on the congestion situation. Specifically, identify the user's emotional state (stress, anxiety, comfort, etc.) from facial images and voice data, and use the results and in-car congestion data to provide optimal boarding guidance."

[0503] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0504] Step 1:

[0505] The device collects location information using the GPS function of the mobile device. When the user is near a station or an autonomous vehicle, the device periodically transmits the location information to the server.

[0506] Input: GPS data from your mobile device

[0507] Output: User's current location

[0508] Specific operation: The device acquires GPS data every 5 seconds and sends it to the server.

[0509] Step 2:

[0510] The server collects image data in real time from monitoring devices (cameras and sensors) installed within stations and autonomous vehicles.

[0511] Input: Real-time image data from a surveillance device

[0512] Output: Image information of station premises and train interiors

[0513] Specific operation: The server continuously receives the video stream transmitted from the monitoring device.

[0514] Step 3:

[0515] The server analyzes the collected image data using an image analysis algorithm (e.g., OpenCV) and calculates the congestion level of each vehicle.

[0516] Input: Collected image data

[0517] Output: Crowding level of each vehicle (empty, normal, crowded categories)

[0518] Specific operation: The server counts the number of people in the image, calculates the occupancy rate, and classifies the congestion level into categories.

[0519] Step 4:

[0520] The device uses an emotion engine (e.g., Microsoft's Azure Emotion API or Google Cloud's Vision API) to analyze the user's emotional state.

[0521] Input: Device camera images, audio data, user operation patterns

[0522] Output: User's emotional state (stressed, anxious, comfortable, etc.)

[0523] Specific operation: The device uses a camera to capture the user's facial expressions and analyzes them using an emotion analysis engine.

[0524] Step 5:

[0525] The server integrates the congestion data and the user's emotional state to calculate the optimal boarding location.

[0526] Input: crowding data, user emotional state

[0527] Output: Optimal boarding position (vehicle and boarding door)

[0528] Specific operation: The server uses an algorithm to calculate the optimal boarding location for each user, and also refers to past data to take into account future congestion conditions.

[0529] Step 6:

[0530] The server generates a message informing the user of the optimal boarding location based on the calculation results.

[0531] Input: Calculated optimal pickup location

[0532] Output: A message to inform the user

[0533] Specific operation: The server generates a text message and map-style directions including the best pickup location and sends them to the device.

[0534] Step 7:

[0535] The terminal displays the notification information received from the server to the user and provides specific instructions.

[0536] Input: A notification message to the user

[0537] Output: Instructions displayed to the user

[0538] Specific operation: The device uses the display and voice guidance functions to provide the user with information about the boarding location.

[0539] Step 8:

[0540] Users provide feedback through the app after their ride.

[0541] Input: User feedback information

[0542] Output: Feedback data sent to the server

[0543] What it does: Users use the app interface to input feedback about congestion and emotional state.

[0544] Step 9:

[0545] The server analyzes the feedback it receives and uses it to improve prediction accuracy and emotion recognition accuracy in future trials.

[0546] Input: User feedback data

[0547] Output: Improved parsing algorithm

[0548] Specific operation: The server stores the feedback data in a database and uses it to adjust and improve the analysis algorithm.

[0549] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0550] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0551] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0552] [Second embodiment]

[0553] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0554] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0555] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0556] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0557] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0558] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0559] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0560] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0561] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0562] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0563] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0564] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0565] This invention is a system that uses location information from a mobile device and image data collected from monitoring devices in and around stations to appropriately guide passengers to the vehicle they are boarding. Specific embodiments based on the system program are described below.

[0566] Data Collection Phase

[0567] Collection of mobile phone location information

[0568] Device: Each user's mobile device periodically (for example, every 5 seconds) transmits its location information to the server. This information is sent using GPS or beacons.

[0569] User: Before arriving at the station, you need to enable location services on your mobile device.

[0570] Data collection from monitoring devices

[0571] Server: Collects image data in real time from monitoring devices (cameras and sensors) inside and around stations. This data is later used to analyze congestion levels.

[0572] Data analysis phase

[0573] Understanding congestion levels through image analysis

[0574] Server: Analyzes image data collected from the monitoring devices and runs algorithms to calculate the congestion level of each vehicle, including the number of passengers and the space occupancy rate.

[0575] Server: Based on the results of image analysis, classify each vehicle into categories such as "empty," "normal," or "crowded."

[0576] Integrated analysis of location information

[0577] Server: Aggregates location information from each user's mobile device and analyzes the location and destination information of all users, thereby estimating the current number of passengers in each vehicle.

[0578] Optimization Phase

[0579] Optimization calculation of boarding position

[0580] Server: Based on the collected location information and image data, the server calculates the optimal boarding location for the user. For example, an algorithm is designed to prioritize suggesting vehicles with low congestion levels.

[0581] Server: Refers to past congestion data and learns congestion patterns by time of day and day of the week to predict future congestion conditions.

[0582] Guide Phase

[0583] Notification of optimal pickup location

[0584] Server: Based on the calculation results, the server notifies each user of the optimal boarding location (vehicle and door location). Notifications are sent to the device in real time.

[0585] Device: Displays received notification information to the user and provides specific instructions in the form of text and maps.

[0586] Feedback Phase

[0587] Collecting post-ride feedback

[0588] User: After the ride, the user provides feedback through the app about the congestion situation during the ride, which is then sent to the server.

[0589] Terminal: Provides an interface for collecting feedback and accepts input from the user.

[0590] Processing Feedback

[0591] Server: Analyzes the received feedback and uses it to improve the accuracy of future congestion predictions and boarding location suggestions.

[0592] Specific examples

[0593] Morning rush hour scenario

[0594] 1. Device: User A leaves home and activates the location information on his mobile device on his way to the nearest station.

[0595] 2. Server: Obtains user A's location information and sends it to the server. At the same time, it uses augmented reality (AR) functions to confirm the user's location.

[0596] 3. Server: Acquires image data from station monitoring devices and analyzes it.

[0597] 4. Server: Based on the analysis results, it determines that car 3 is currently the vacant car, and notifies user A, "Car 3 is the vacant car. Please wait in front of car 3."

[0598] 5. Terminal: Display a message to user A, prompting him to move in front of car 3.

[0599] 6. User: Follow the instructions to stand in front of car 3, get in, and provide feedback through the app.

[0600] 7. Server: Receives User A's feedback and improves the accuracy of the next prediction based on the analysis results.

[0601] In this way, the system based on the present invention provides a concrete means for managing the congestion status of each vehicle in real time and providing passengers with a comfortable commuter life.

[0602] The processing flow will be explained below.

[0603] Step 1: Collect location information

[0604] Terminal: The mobile terminal periodically (for example, every 5 seconds) collects location information obtained from GPS and beacon sensors and prepares to send it to the server.

[0605] Step 2: Send location information

[0606] Device: Sends location information to the server, which notifies the server of the user's current location in real time.

[0607] Step 3: Collecting data from monitoring devices

[0608] Server: Image data is collected in real time from monitoring devices installed inside and around stations. This image data is used to understand the congestion status of each train car.

[0609] Step 4: Analyzing the image data

[0610] Server: Runs algorithms to analyze the collected image data. Specifically, it uses image analysis technology to calculate the number of passengers and space occupancy in each vehicle.

[0611] Step 5: Calculate congestion

[0612] Server: Based on the results of image analysis, the congestion level of each vehicle is classified into categories such as "empty," "normal," or "crowded."

[0613] Step 6: Integrated analysis of location information

[0614] Server: Aggregates location information sent from mobile devices and analyzes each passenger's current location and destination, thereby estimating the number of passengers in each vehicle.

[0615] Step 7: Calculate the best pickup location

[0616] Server: Based on the analysis results, the server runs an algorithm that calculates the optimal boarding location (vehicle and boarding door) for each user.

[0617] Step 8: Generate notifications

[0618] Server: Based on the calculation results, a message is generated to notify each user of the specific boarding location.

[0619] Step 9: Sending notifications

[0620] Server: Sends the generated message to the user's mobile device. Example: "Car 3 is the least crowded. Please wait in front of car 3."

[0621] Step 10: Displaying notifications

[0622] Terminal: Displays the received message to the user, so that the user can check which car to board.

[0623] Step 11: Ride

[0624] User: Follow the notification and move to the designated vehicle and get in.

[0625] Step 12: Provide feedback

[0626] User: After the ride, enter information about the ride's congestion and other feedback through the app.

[0627] Step 13: Submit your feedback

[0628] Terminal: Sends the feedback entered by the user to the server.

[0629] Step 14: Analyze feedback

[0630] Server: Analyzes the received feedback and uses it as data to improve the accuracy of future predictions.

[0631] Through these steps, the system can grasp the congestion status of each vehicle in real time and suggest the optimal boarding location to the user.

[0632] Example 1

[0633] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0634] In modern urban areas, congestion during commutes has become a major social problem. For commuters, especially those using trains, crowded trains can cause significant stress and discomfort. Conventional methods struggle to grasp congestion levels in real time and provide users with optimal boarding locations, resulting in a lack of information for passengers to determine the appropriate boarding location. Therefore, an effective system is needed to alleviate congestion and ensure a comfortable ride.

[0635] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0636] In this invention, the server includes means for collecting location information from the mobile device, means for collecting image data from the monitoring device, means for analyzing the location information and the image data to calculate the congestion level of each vehicle, means for learning congestion patterns by time period and day of the week based on past data and predicting future congestion conditions, means for calculating the optimal boarding position based on the analysis results and prediction results and notifying the mobile device, means for receiving feedback from the user and improving the accuracy of the analysis and prediction, and means for notifying the mobile device of the optimal boarding position in text or map format. This makes it possible to grasp the congestion status of each vehicle in real time and provide the user with optimal boarding guidance.

[0637] "Mobile device" refers to a portable electronic device that is carried by a user and used by the user, such as a mobile phone, smartphone, or tablet.

[0638] "Location Information" refers to data indicating a specific geographic location obtained through GPS, beacons, Wi-Fi, cell towers, etc.

[0639] "Monitoring devices" refers to devices such as cameras, sensors, and beacons installed in or around stations that monitor the surrounding conditions and collect and provide data.

[0640] "Image data" refers to visual data in the form of still images or video captured or acquired by surveillance equipment.

[0641] "Crowding level" is an indicator of the density of people in a particular area or vehicle, and is calculated based on the number of passengers and the occupancy rate of space.

[0642] "Analysis" refers to the process of analyzing collected data in detail to extract meaning and patterns and derive results according to the intended purpose.

[0643] "Boarding location" refers to the specific car and its door location that the user selects to board the train.

[0644] "Notification" refers to the act of sending a message to a mobile device to notify the user of the results of a calculation or analysis.

[0645] "Past data" refers to data collected in the past, and primarily refers to past congestion conditions and user usage patterns.

[0646] "Feedback" is input data provided by users based on their experiences and opinions, which is used to reevaluate and improve the system.

[0647] "Forecasting" refers to the process of predicting and estimating future situations based on collected data and analysis results.

[0648] This invention is a system that provides optimal boarding guidance to passengers by utilizing location information from mobile devices and image data acquired from monitoring devices in and around stations. This system is implemented using the following means.

[0649] Data Collection Phase

[0650] Collection of mobile phone location information

[0651] Device: Each user's mobile device periodically transmits location information to the server. Location information is collected using GPS or beacon technology (e.g., Bluetooth beacons). Specifically, the device activates location services, measures its current location every 5 seconds, and transmits it to the server using a secure communication protocol (e.g., HTTPS).

[0652] User: Before arriving at the station, the user must enable location services on their mobile device and grant location sharing permission to the app.

[0653] Data collection from monitoring devices

[0654] Server: The server collects image data in real time from monitoring devices inside and around stations. The monitoring devices include cameras and sensors, and receives image data as a stream from these devices. The received data is temporarily stored in storage and used for subsequent analysis.

[0655] Data analysis phase

[0656] Understanding congestion levels through image analysis

[0657] Server: The server uses image analysis algorithms such as OpenCV and TensorFlow to analyze the collected image data. Specifically, it detects passengers, counts the number of people, and analyzes space occupancy for each frame, and classifies the congestion level of each car into categories of "empty," "normal," or "crowded."

[0658] Server: The server stores these analysis results and uses them for subsequent pickup location suggestions.

[0659] Integrated analysis of location information

[0660] Server: The server aggregates the location information collected from each user's mobile device and plots it on a map, allowing the server to determine the current number of passengers in each car and the number of passengers in a specific area of ​​the station.

[0661] Optimization Phase

[0662] Optimization calculation of boarding position

[0663] Server: The server calculates the optimal boarding location for each user based on the aggregated location information and analyzed image data. The algorithm references past congestion data and learns congestion patterns by time of day and day of the week to predict future congestion conditions.

[0664] Server: Based on the calculation results, the optimal boarding location is determined and saved.

[0665] Guide Phase

[0666] Notification of optimal pickup location

[0667] Server: Based on the calculation results, the server notifies each user of the optimal boarding location in real time. The notification content includes specific instructions, such as "Car 3 is the least crowded. Please wait in front of car 3."

[0668] Device: The device displays the received notification to the user in an intuitive format, such as a text message or a pin on a map.

[0669] Feedback Phase

[0670] Collecting post-ride feedback

[0671] User: After the ride, the user provides feedback through the app about the ride's congestion, with options such as "very crowded," "normal," and "empty."

[0672] Terminal: The terminal provides an interface for collecting feedback and accepts input from the user, which is then sent to the server.

[0673] Processing Feedback

[0674] Server: The server analyzes the received feedback and improves the accuracy of congestion predictions and boarding location suggestions for the next trip. The analysis integrates the feedback items and performs statistical evaluation.

[0675] Specific examples

[0676] Morning rush hour scenario

[0677] 1. Device: User A leaves home and activates the location information service on his mobile device on his way to the nearest station.

[0678] 2. Server: The server receives and stores the location information of User A. At the same time, it visually confirms User A's location using the augmented reality (AR) function.

[0679] 3. Server: The server acquires image data from the station monitoring devices and analyzes it in real time.

[0680] 4. Server: Based on the analysis results, it determines that car 3 is the least crowded and notifies user A, "Car 3 is the least crowded. Please wait in front of car 3."

[0681] 5. Terminal: The terminal displays a notification to User A, instructing him to move to the front of car 3. The notification is indicated by a text message and a pin on the map.

[0682] 6. User: User A follows the instructions to stand in front of car 3 and provides feedback on the congestion situation through the app after boarding.

[0683] 7. Server: The server receives User A's feedback and uses the analysis results to improve the accuracy of the next prediction.

[0684] Prompt Sentence Examples

[0685] Prompt: "Describe the steps for a system that uses station surveillance cameras and mobile device location information to provide optimal ride guidance."

[0686] This system is designed to grasp the congestion status of each vehicle in real time and provide users with a comfortable commute.

[0687] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0688] Step 1:

[0689] Collection of mobile phone location information

[0690] Input: Location information obtained from the mobile device's built-in sensors (GPS data, beacon signals, etc.)

[0691] Device behavior: The device activates location services, measures its current location every 5 seconds, and sends the measurements to a server using a secure communication protocol such as HTTPS.

[0692] Output: The user's current location is sent to the server.

[0693] Step 2:

[0694] Image data collection from monitoring devices

[0695] Input: Real-time image streams from surveillance devices (cameras, sensors) installed in and around the station

[0696] Server operation: The server establishes a secure connection with the monitoring device, receives image data in real time, and temporarily stores the received data in storage.

[0697] Output: Image data saved in storage

[0698] Step 3:

[0699] Understanding congestion levels through image analysis

[0700] Input: Image data stored in storage

[0701] Server operation: The server analyzes image data using image analysis algorithms such as OpenCV and TensorFlow. It detects passengers, counts the number of people, and analyzes space occupancy for each frame.

[0702] Output: Analyzed congestion data (e.g., empty, normal, crowded)

[0703] Step 4:

[0704] Congestion category classification for each vehicle

[0705] Input: Crowd data obtained by image analysis

[0706] Server operation: The server classifies each vehicle into a category such as "empty," "normal," or "crowded" based on the congestion data, using a set threshold.

[0707] Output: Congestion category information for each vehicle

[0708] Step 5:

[0709] Integrated analysis of location information

[0710] Input: Location data from each user's mobile device

[0711] Server operation: The server aggregates location information and plots it on a map. It analyzes overall trends and calculates the number of users concentrated in a particular car or in a particular area of ​​the station.

[0712] Output: Integrated analysis results based on user location information

[0713] Step 6:

[0714] Optimization calculation of boarding position

[0715] Input: Integrated analysis results of location information and congestion category information

[0716] Server operation: The server uses an algorithm to calculate the optimal boarding location for the user. It learns from past data and congestion patterns by time of day and day of the week, and runs an algorithm to predict future congestion conditions.

[0717] Output: Optimal pickup location information

[0718] Step 7:

[0719] Notification of optimal pickup location

[0720] Input: Best Pickup Location

[0721] Server operation: Based on the calculation results, the server notifies each user of the optimal boarding location in real time. The notification message includes specific instructions such as "Car 3 is the least crowded. Please wait in front of car 3."

[0722] Device behavior: The device displays notifications to the user in intuitive formats such as text messages or pins on a map.

[0723] Output: A notification message that is displayed to the user

[0724] Step 8:

[0725] Collecting post-ride feedback

[0726] Input: User feedback information

[0727] User Action: After the ride, the user provides feedback through the app about the ride's congestion, with options such as "very crowded," "normal," and "empty."

[0728] Device behavior: The device provides an interface for collecting feedback and accepts input from the user, which is then sent to the server.

[0729] Output: Feedback data sent to the server

[0730] Step 9:

[0731] Processing Feedback

[0732] Input: Feedback data

[0733] Server operation: The server analyzes the received feedback and uses it to improve the accuracy of future congestion predictions and boarding location suggestions. The analysis integrates the feedback items and performs statistical evaluation.

[0734] Output: Improved congestion prediction data and pickup location suggestions

[0735] (Application example 1)

[0736] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0737] Technologies that grasp the congestion situation in and around train stations and guide passengers to the optimal boarding location are widely used, but the application of these technologies to avoiding congestion in commercial facilities has not been sufficiently considered. While the importance of avoiding congestion in such commercial facilities is increasing, current systems lack the technology to analyze the congestion situation in real time and provide users with appropriate routes and areas. The objective of this invention is to provide a system that provides appropriate congestion avoidance guidance in order to provide a comfortable shopping experience in commercial facilities.

[0738] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0739] In this invention, the server includes means for collecting location information from mobile devices, means for collecting image data from monitoring devices, means for analyzing the location information and the image data to calculate the congestion level of each area, means for calculating the optimal route and area based on the analysis results and notifying the mobile devices, means for obtaining feedback from users and improving the accuracy of the analysis and prediction, and means for predicting future congestion conditions using past data. This makes it possible to grasp the congestion status in commercial facilities in real time and provide appropriate congestion avoidance guidance.

[0740] "Mobile device" refers to a device that a user can carry with them and that can transmit location information, such as a mobile phone, smartphone, or tablet.

[0741] "Location information" refers to data transmitted by a mobile device indicating its current geographical location, and is information obtained using GPS or beacons.

[0742] "Surveillance equipment" refers to devices such as cameras and sensors installed in commercial facilities or specific areas that monitor the surrounding situation in real time and collect image data.

[0743] "Image data" refers to video and photographic data collected by surveillance equipment, and is information that can be analyzed to determine the level of congestion in an area.

[0744] "Crowding level" is an index that indicates the density or occupancy rate of people in a specific area, and refers to information calculated from analyzed image data.

[0745] "Calculating routes and areas" refers to the process of deriving the optimal travel route and areas to visit for a user based on collected location information and image data.

[0746] "Means of notification" refers to the method of sending the calculated optimal route and area guidance information to the user's mobile device and displaying it.

[0747] "Feedback" refers to information provided by users about their experiences and opinions, which is collected and analyzed to help improve the accuracy of the system.

[0748] "Prediction means" refers to algorithms and methods for referencing past congestion data and estimating future congestion conditions.

[0749] "Commercial facilities" refers to facilities such as shopping malls, department stores, and shops that are visited by many people and may become crowded.

[0750] This invention relates to a system that provides guidance for avoiding crowds in commercial facilities. It integrates location information from mobile devices and image data collected from monitoring devices to guide users to the optimal route and area in real time. It is also a system that aims to further improve accuracy by collecting feedback from users.

[0751] Data Collection Phase

[0752] Mobile location information collection:

[0753] Device: Each user's mobile device periodically sends its location information to the server, using GPS or beacons.

[0754] Users: Must enable location services on their mobile device before arriving at the commercial establishment.

[0755] Data collection from monitoring devices:

[0756] Server: Collects image data in real time from monitoring devices (cameras and sensors) installed within commercial facilities. This data is used to analyze congestion levels.

[0757] Data Analysis Phase

[0758] Image analysis to understand congestion levels:

[0759] Server: Analyzes image data collected from the monitoring devices and runs an algorithm to calculate the congestion level of each area, specifically the number of users and the occupancy rate of the space.

[0760] Server: Based on the results of image analysis, the congestion level of each area is classified as "low," "normal," "high," etc.

[0761] Integrated location analysis:

[0762] Server: Aggregates location information from each user's mobile device and analyzes the location and destination information of all users to estimate the current congestion situation in each area.

[0763] Route and area optimization phase

[0764] Best route and area suggestions:

[0765] Server: Calculates the optimal route and area for the user based on the collected location information and image data. An algorithm is built in to prioritize suggesting areas with low congestion.

[0766] Server: Refers to past congestion data, learns congestion patterns by time of day and day of the week, and predicts future congestion conditions.

[0767] Guide Phase

[0768] Best route and area notifications:

[0769] Server: Based on the calculation results, the server notifies each user of the optimal route and area information. Notifications are sent to the device in real time.

[0770] Device: Displays received notification information to the user and provides specific instructions in the form of text and maps.

[0771] Feedback Phase

[0772] Collecting post-ride feedback:

[0773] User: After moving around the shopping mall, the user provides feedback about the congestion situation through the app, which is then sent to the server.

[0774] Terminal: Provides an interface for collecting feedback and accepts input from the user.

[0775] Feedback processing:

[0776] Server: Analyzes the received feedback and uses it to improve the accuracy of congestion predictions and route guidance for future trips.

[0777] Hardware and software used:

[0778] Hardware: smartphones, surveillance cameras, sensors.

[0779] Software: Python, image analysis algorithm, location analysis algorithm, external API (for acquiring congestion data).

[0780] Examples:

[0781] A user arrives at a shopping mall at 10:00 a.m. After arriving at shopping mall A, the user opens the app and is directed to the least crowded area. After enjoying shopping, the user provides feedback to the app.

[0782] Example prompt: "Current location: 10:00. Please guide me to the areas in Shopping Mall A. Please recommend the least crowded area, taking into account the congestion status of each area. After making the recommendation, please also display a prompt to provide feedback to the user. The feedback content should be an interface where the user can write about the congestion status of the shopping experience."

[0783] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0784] Step 1:

[0785] Collection of mobile phone location information

[0786] Input: The user enables location services on their mobile device.

[0787] Processing: The device periodically (for example, every 5 seconds) obtains its own location information using GPS or beacons and generates location information data.

[0788] Output: The acquired location data is sent to the server.

[0789] Specific operation: The GPS module in the smartphone measures the current location, and the terminal app collects that data and sends it to a specified server.

[0790] Step 2:

[0791] Data collection from monitoring devices

[0792] Input: Image data provided by surveillance equipment within a commercial facility.

[0793] Processing: The server collects image data from the monitoring devices in real time and prepares the image data for storage and processing.

[0794] Output: The collected image data is stored in the server.

[0795] Specific operation: The server receives the video captured by the surveillance camera and stores it in storage. The image data is also organized in a specific format.

[0796] Step 3:

[0797] Understanding congestion levels through image analysis

[0798] Input: Collected image data.

[0799] Processing: The server uses image analysis algorithms to calculate the congestion level of each area, which is the process of calculating the number of users and the occupancy rate of the space.

[0800] Output: Obtains congestion information for each area.

[0801] What it does: Server-supported image analysis software counts the number of people in the image and calculates the area occupancy rate.

[0802] Step 4:

[0803] Integrated analysis of location information

[0804] Input: Location data and congestion information from user devices.

[0805] Processing: The server combines the location data and congestion information to estimate the overall congestion situation for each area.

[0806] Output: Consolidated congestion data is obtained.

[0807] What it does: The server uses a geographic information system (GIS) to map users' locations and area congestion levels, and then aggregates the data.

[0808] Step 5:

[0809] Optimal route and area suggestions

[0810] Input: Consolidated crowding data.

[0811] Processing: The server uses an algorithm to calculate the best route and area for each user, prioritizing less congested areas.

[0812] Output: The calculated optimal route and area information is obtained.

[0813] Specific operation: The server uses specific rules and algorithms (e.g., shortest path algorithms or congestion-based optimization methods) to determine the most suitable route and area for the user.

[0814] Step 6:

[0815] Optimal route and area notifications

[0816] Input: Calculated optimal route and area information.

[0817] Processing: The server sends the optimal route and area information to the user's mobile device.

[0818] Output: A real-time notification is displayed on the user's mobile device.

[0819] What it does: Uses the notification system (push notifications and in-app notifications) to display text and map-based directions on the user's smartphone.

[0820] Step 7:

[0821] Collecting and Processing Feedback

[0822] Input: Feedback information provided by the user.

[0823] Processing: The device provides an interface for collecting feedback and accepts user input. The server analyzes the received feedback and uses it to improve the accuracy of future guidance.

[0824] Output: Parsed feedback data.

[0825] Specific operation: Collects information provided by users through feedback forms and survey functions within the app, and the server analyzes the data to perform feedback analysis.

[0826] Prompt Sentence Examples

[0827] "Current location: 10:00. Please guide the user to the areas in the shopping mall. Please recommend the least crowded area, taking into account the congestion level of each area. After making the recommendation, please also display a prompt to provide feedback to the user. The feedback should be an interface where the user can describe how crowded the shopping experience was."

[0828] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0829] The present invention combines an emotion engine with a system that uses location information from mobile devices and image data collected from monitoring devices in and around stations to provide appropriate guidance on which vehicle to board. Specific embodiments based on the system program are described below.

[0830] Data Collection Phase

[0831] Collection of mobile phone location information

[0832] Device: Each user's mobile device periodically (for example, every 5 seconds) transmits its location information to the server. This information is sent using GPS or beacons.

[0833] User: Before arriving at the station, you need to enable location services on your mobile device.

[0834] Data collection from monitoring devices

[0835] Server: Collects image data in real time from monitoring devices (cameras and sensors) installed inside and around stations. This image data is used to understand the congestion status of each train car.

[0836] Data analysis phase

[0837] Understanding congestion levels through image analysis

[0838] Server: Runs algorithms to analyze the collected image data. Specifically, it uses image analysis technology to calculate the number of passengers and space occupancy in each vehicle.

[0839] Server: Based on the results of image analysis, the congestion level of each vehicle is classified into categories such as "empty," "normal," or "crowded."

[0840] Integrated analysis of location information

[0841] Server: Aggregates location information from each user's mobile device and analyzes the location and destination information of all users, thereby estimating the number of passengers in each vehicle.

[0842] Optimization Phase

[0843] Optimization calculation of boarding position

[0844] Server: Based on the analysis results, the server runs an algorithm that calculates the optimal boarding location (vehicle and boarding door) for each user.

[0845] Server: Refers to past congestion data and learns congestion patterns by time of day and day of the week to predict future congestion conditions.

[0846] Emotion Recognition Phase

[0847] Using the Emotion Engine

[0848] Device: The emotion engine recognizes the user's emotions using facial expression analysis via a camera, voice analysis, and the user's operation patterns.

[0849] Server: Analyzes the emotional data obtained from the emotion engine and understands the user's current emotional state.

[0850] Emotion-Based Optimization

[0851] Server: Integrates the user's emotional data and analysis results, and adjusts the recommended ride locations based on the user's emotional state. For example, for users with high levels of stress or anxiety, it will prioritize suggesting less crowded vehicles.

[0852] Guide Phase

[0853] Notification of optimal pickup location

[0854] Server: Based on the calculation results, it generates a message to notify each user of the optimal boarding location.

[0855] Device: Displays received notification information to the user and provides specific instructions in the form of text and maps.

[0856] Feedback Phase

[0857] Collecting post-ride feedback

[0858] User: After the trip, provide feedback through the app about the ride's congestion and emotional state.

[0859] Terminal: Provides an interface for collecting feedback and accepts input from the user.

[0860] Processing Feedback

[0861] Server: Analyzes the received feedback and uses it as data to improve prediction accuracy and emotion recognition accuracy in future trials.

[0862] Specific examples

[0863] Morning rush hour scenario

[0864] 1. Device: User A leaves home and activates the location information on his mobile device on his way to the nearest station.

[0865] 2. Server: Obtains user A's location information and sends it to the server. At the same time, it uses augmented reality (AR) functions to confirm the user's location.

[0866] 3. Server: Acquires image data from station monitoring devices and analyzes it.

[0867] 4. Server: Based on the analysis results, it determines that car 3 is currently the vacant car, and notifies user A, "Car 3 is the vacant car. Please wait in front of car 3."

[0868] 5. Device: The emotion engine analyzes user A's emotion data and detects his / her stress state.

[0869] 6. Server: Consider user A’s emotional state and further adjust the pickup location suggestions.

[0870] 7. Terminal: Display a message to user A, prompting him to move in front of car 3.

[0871] 8. User: Follow the instructions to stand in front of car 3, get in, and provide feedback through the app.

[0872] 9. Server: Receives User A's feedback and improves the accuracy of the next prediction based on the analysis results.

[0873] In this way, the system based on the present invention provides a concrete means for providing users with a comfortable commuter life by managing the congestion status of each vehicle in real time and proposing the optimal boarding location taking into account the user's emotional state.

[0874] The processing flow will be explained below.

[0875] Step 1: Collect location information

[0876] Terminal: The mobile terminal periodically (for example, every 5 seconds) collects location information obtained from GPS and beacon sensors and prepares to send it to the server.

[0877] Step 2: Send location information

[0878] Device: Sends location information to the server, which notifies the server of the user's current location in real time.

[0879] Step 3: Collecting data from monitoring devices

[0880] Server: Obtains image data in real time from monitoring devices (cameras and sensors) installed inside and around stations. This image data is used to understand the congestion status of each train car.

[0881] Step 4: Analyzing the image data

[0882] Server: Runs algorithms to analyze the collected image data. Specifically, it uses image analysis technology to calculate the number of passengers and space occupancy in each vehicle.

[0883] Step 5: Calculate congestion

[0884] Server: Based on the results of image analysis, the congestion level of each vehicle is classified into categories such as "empty," "normal," or "crowded."

[0885] Step 6: Integrated analysis of location information

[0886] Server: Aggregates location information sent from mobile devices and analyzes each passenger's current location and destination, thereby estimating the number of passengers in each vehicle.

[0887] Step 7: Calculate the best pickup location

[0888] Server: Based on the analysis results, the server runs an algorithm that calculates the optimal boarding location (vehicle and boarding door) for each user.

[0889] Step 8: Use historical data

[0890] Server: Refers to past congestion data and learns congestion patterns by time of day and day of the week to predict future congestion conditions.

[0891] Step 9: Emotion Recognition with the Emotion Engine

[0892] Device: The emotion engine recognizes the user's emotions by analyzing facial expressions via a camera, voice analysis, and the user's operation patterns.

[0893] Step 10: Sending Emotion Data

[0894] Device: Sends the recognized emotion data to the server, which informs the server of the user's current emotional state.

[0895] Step 11: Adjusting riding position based on emotions

[0896] Server: Integrates the user's emotional data and analysis results to tailor the recommended ride locations based on their emotional state. For example, prioritize less crowded vehicles for users with high levels of stress.

[0897] Step 12: Generate and send notifications

[0898] Server: Based on the calculation results and emotion data, a message is generated to notify each user of their specific boarding location and sent to the device.

[0899] Step 13: Viewing Notifications

[0900] Terminal: Displays the received message to the user, so that the user can check which car they should board.

[0901] Step 14: Ride

[0902] User: Follow the notification and move to the designated vehicle and get in.

[0903] Step 15: Provide feedback

[0904] User: After the ride, users provide feedback through the app about the ride's congestion and emotional state.

[0905] Step 16: Submit your feedback

[0906] Terminal: Sends the feedback entered by the user to the server.

[0907] Step 17: Analyze feedback

[0908] Server: Analyzes the received feedback and uses it as data to improve prediction accuracy and emotion recognition accuracy in future trials.

[0909] Through these steps, the system can grasp the congestion status of each vehicle in real time and suggest the optimal boarding location taking into account the user's emotional state.

[0910] Example 2

[0911] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0912] Conventional ride guidance systems were able to grasp the congestion level of each vehicle using location information and image data, but did not provide ride guidance that took into account the user's emotional state. As a result, riding in a crowded vehicle could cause stress and discomfort to the user. It was particularly difficult to provide a comfortable ride experience for users during rush hour situations.

[0913] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for collecting location information from a mobile terminal, means for collecting image data from a monitoring device, means for analyzing the location information and the image data to calculate the congestion level of each vehicle, means for grasping the emotional state of the user using an emotion recognition engine, and means for calculating the optimal boarding position based on the analysis result and the emotional state and notifying the mobile terminal. This makes it possible to propose an optimal boarding position taking into account the user's emotional state in addition to the user's location information and congestion status.

[0914] A "mobile terminal" is a communication device carried by a user that provides location information and emotional state data.

[0915] "Location information" refers to geographic coordinate information collected from mobile devices and GPS sensors.

[0916] "Monitoring devices" are equipment such as cameras and sensors that are installed in or around stations to collect image data.

[0917] "Image data" refers to video and photo data acquired from a surveillance device.

[0918] "Analysis" involves processing the collected data and calculating the congestion level of each vehicle and the emotional state of the user.

[0919] "Crowding level" refers to the degree of congestion based on the number of passengers and space occupancy in each vehicle.

[0920] An "emotion recognition engine" is software for analyzing a user's emotional state, including facial expression analysis and voice analysis using a camera and microphone.

[0921] "Notification" refers to the act of sending the calculated optimal boarding location to the user's mobile device.

[0922] "Feedback" is information provided by users about the congestion situation and their emotional state after riding.

[0923] This invention combines an emotion engine with a system that uses location information from mobile devices and image data collected from monitoring devices in and around stations to provide appropriate guidance on which vehicle to board. Specific embodiments of the system are described below.

[0924] Data Collection Phase

[0925] Collection of mobile phone location information

[0926] Device: Each user's mobile device sends its location information to the server every 5 seconds. This information is obtained using GPS or beacons. Specifically, the device's built-in GPS sensor is used to obtain location information, which is then compressed and sent to the server.

[0927] User: Before arriving at the station, they will need to enable location services on their mobile device. A notification will appear on the device prompting them to enable the service.

[0928] Data collection from monitoring devices

[0929] Server: Collects image data in real time from monitoring devices (cameras and sensors) installed in and around the station. Specifically, it acquires video streams from cameras installed on platforms and at entrances and exits.

[0930] Server: Temporarily stores collected image data in storage for subsequent analysis.

[0931] Data analysis phase

[0932] Understanding congestion levels through image analysis

[0933] Server: Executes algorithms to analyze collected image data. Specifically, it applies image analysis technology using deep learning to calculate the number of passengers in each vehicle and the occupancy rate of space. For example, it identifies the positions of passengers' faces and bodies based on images acquired from surveillance cameras and calculates the degree of congestion.

[0934] Server: Based on the calculated congestion level, each vehicle is classified into a category such as "empty," "normal," or "crowded." This classification result is stored in a database and used in subsequent phases.

[0935] Integrated analysis of location information

[0936] Server: Aggregates location information from each user's mobile device and analyzes the location and destination information of all users, using a location clustering algorithm to identify areas where passengers are concentrated.

[0937] Server: Combines location and congestion data to calculate the expected number of passengers in each vehicle.

[0938] Emotion Recognition Phase

[0939] Using the Emotion Engine

[0940] On-device: The device uses an emotion recognition engine to analyze the user's emotional state. Specifically, the device uses the camera to analyze facial expressions and the microphone to analyze voices. This allows the device to identify emotional states such as stress or anxiety.

[0941] Server: Analyzes the emotional data sent from the emotion recognition engine and understands the user's emotional state. This makes it possible to suggest the optimal boarding location based on the user's emotional state.

[0942] Guide Phase

[0943] Notification of optimal pickup location

[0944] Server: Calculates the optimized boarding location (specific vehicle and boarding door) and generates customized instruction messages for each user.

[0945] Terminal: The generated notification message is displayed to the user. For example, specific instructions such as "Currently, car number 3 is the vacant car. Please wait in front of car number 3" are displayed in text or map format.

[0946] Feedback Phase

[0947] Collecting post-ride feedback

[0948] Users: After their ride, they provide feedback through the app about the occupancy and their emotional state, specifically in the form of a short survey about their ride experience.

[0949] Terminal: Provides a feedback input interface and accepts data from the user.

[0950] Processing Feedback

[0951] Server: Analyzes the received feedback data and performs batch processing to improve prediction accuracy and emotion recognition accuracy from the next time onwards. This is expected to improve the accuracy of the entire system.

[0952] Specific examples

[0953] Morning rush hour scenario

[0954] 1. Device: User A leaves home and activates the location information on his mobile device on his way to the nearest station.

[0955] 2. Server: Obtains user A's location information and sends it to the server. At the same time, the server confirms the user's location using AR functions, etc.

[0956] 3. Server: Acquires image data from station monitoring devices and analyzes it.

[0957] 4. Server: Based on the analysis results, it determines that car 3 is currently the vacant car, and notifies user A, "Car 3 is the vacant car. Please wait in front of car 3."

[0958] 5. Device: The emotion engine analyzes user A's emotion data and detects his / her stress state.

[0959] 6. Server: Consider user A’s emotional state and further adjust the pickup location suggestions.

[0960] 7. Terminal: Display a message to user A, prompting him to move in front of car 3.

[0961] 8. User: Follow the instructions to stand in front of car 3, get in, and provide feedback through the app.

[0962] 9. Server: Receives User A's feedback and improves the accuracy of the next prediction based on the analysis results.

[0963] In an embodiment based on this scenario, user location information, station monitoring data, and emotion data can be integrated in real time to provide users with a comfortable riding experience.

[0964] Prompt Sentence Examples

[0965] Below are some example prompts to input to the generative AI model:

[0966] "Please describe in detail the process of a system that uses mobile device location information and station surveillance camera data to calculate the optimal boarding location for passengers. This should include data collection, analysis phase, use of an emotion engine, optimization, notification phase, and user feedback processing."

[0967] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0968] Step 1:

[0969] Device: The user's mobile device enables location services. Specifically, the device's GPS module acquires its current location every 5 seconds and transmits the location data to the server. This provides the server with the user's real-time location information as input data.

[0970] Step 2:

[0971] Server: Collects image data from surveillance devices (cameras and sensors) installed in and around the station. Specifically, it acquires video streams from each surveillance camera in real time and temporarily stores them in storage. This image data is used in the next image analysis phase.

[0972] Step 3:

[0973] Server: Analyzes the collected image data. Specifically, it applies image analysis technology using deep learning to calculate the number of passengers in each vehicle and the occupancy rate of space. It uses an object detection algorithm to identify and count passengers from the input image data. The analysis results output the degree of congestion in each vehicle.

[0974] Step 4:

[0975] Server: Based on the congestion analysis results, classify each vehicle into a category such as "empty," "normal," or "crowded." The input is the analysis result from step 3, which is applied to a category classification algorithm to output the classification result. This classification result is stored in a database.

[0976] Step 5:

[0977] Server: Aggregates location information sent from each user's mobile device and analyzes the overall location and destination information. Specifically, it uses a location clustering algorithm to identify areas where users are concentrated. The input is real-time location data, and the output is passenger density information for each area.

[0978] Step 6:

[0979] Device: The device uses an emotion recognition engine to analyze the user's emotional state. Specifically, facial expression analysis is performed using the device's camera, and voice analysis is performed using the microphone. The input is the user's facial expression image and voice data, and the output is the evaluation result of the user's emotional state.

[0980] Step 7:

[0981] Server: Analyzes the emotion data sent from the emotion recognition engine and understands the user's emotional state. The input is the emotion engine's output data, and analysis is performed based on this to evaluate the user's emotional state. Based on this evaluation result, the boarding location suggestions are optimized.

[0982] Step 8:

[0983] Server: Calculates the optimal pickup location based on the user's emotional state and generates a personalized notification message. The inputs are congestion data, user location data, and the user's emotional state data, which are combined to run an algorithm to calculate the optimal pickup location. The output is a notification message.

[0984] Step 9:

[0985] Terminal: Displays notification messages received from the server to the user. Specifically, it displays boarding location instructions in text and map format. For example, it displays a message such as "Currently, car 3 is the least crowded. Please wait in front of car 3." The input is the notification message from the server, and the output is the notification to the user.

[0986] Step 10:

[0987] User: Follows the instructions to go to the designated pickup location. After boarding, the user provides feedback on the congestion situation and emotional state during the ride through the app. The input is the user's experience data, and the output is feedback information.

[0988] Step 11:

[0989] Server: Analyzes the feedback data received from users and improves the algorithms to improve prediction accuracy and emotion recognition accuracy in the future. The input is user feedback data, and the output is an improved analysis and prediction algorithm.

[0990] (Application example 2)

[0991] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0992] The issue of congestion in public transportation and the inability to properly guide passengers to a comfortable boarding location are issues. Furthermore, providing boarding guidance without taking into account the passenger's emotional state increases stress and discomfort, degrading the overall riding experience.

[0993] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for collecting location information from the mobile terminal, means for collecting image data from the monitoring device, means for analyzing the location information and the image data to calculate the congestion level of each vehicle, means for analyzing the emotional state of the user, and means for calculating the optimal boarding position based on the analysis result and the emotional state and notifying the mobile terminal. This makes it possible to suggest the optimal boarding position taking into consideration the congestion level of the vehicle and the emotional state of the user, thereby improving the overall riding experience.

[0994] A "mobile terminal" is an electronic device that can be carried by a user and has GPS and Internet connectivity capabilities.

[0995] "Location information" is information that indicates coordinate data at a specific location using a GPS, beacon, or the like.

[0996] A "surveillance device" is a device that includes a camera, sensor, etc., and collects image data and environmental data of a designated area in real time.

[0997] "Image data" is visual information, including still images and video, collected by surveillance equipment.

[0998] "Crowding" is an indicator that expresses the density of people and objects in a particular area or space using a number or category.

[0999] "Emotional state" is data that indicates an individual's psychological state, and is obtained by facial expression analysis and voice analysis via a camera.

[1000] "Analysis results" are information obtained as a result of calculations and evaluations performed on collected data.

[1001] "Forecasting" is the process of predicting future conditions or events based on past and current data.

[1002] "Optimal boarding position" refers to the recommended position or door for the user to board comfortably and efficiently.

[1003] A "notification" is information sent from the system to the user, and may be provided in text, audio, visual format, or the like.

[1004] "Feedback" refers to information such as opinions, evaluations, and impressions collected from users, and is used to improve the system and increase the accuracy of analysis.

[1005] The "smart ride guidance system" of the present invention is a system that collects location information from mobile devices and image data from monitoring devices, and guides users to the optimal ride location taking into account their emotional state. A specific method for implementing this system is described below.

[1006] Data Collection Phase

[1007] Collection of location information from mobile devices

[1008] The terminal is an electronic device (such as a smartphone) carried by the user, which periodically collects location information using its GPS function and transmits it to the server. Location information collection begins when the user is near a station or an autonomous vehicle, and the server receives this information in real time.

[1009] Collection of image data from monitoring devices

[1010] The server collects image data from cameras and other sensors installed in stations and on autonomous vehicles. These devices provide the real-time visual information needed to understand the occupancy level of each carriage.

[1011] Data analysis phase

[1012] Understanding congestion levels through image analysis

[1013] The server uses image analysis algorithms (e.g., OpenCV) to analyze the collected image data, which calculates the number of passengers and space occupancy in each vehicle and classifies the level of congestion into categories of "empty," "normal," or "crowded."

[1014] Emotional state analysis

[1015] The device uses an emotion engine (e.g., Microsoft's Azure Emotion API or Google Cloud's Vision API) to analyze the user's emotions. By analyzing facial expressions and voice via the device's camera and the user's operation patterns, the device detects emotional states such as stress, anxiety, and comfort.

[1016] Optimization Phase

[1017] Optimization calculation of boarding position

[1018] The server combines congestion data and the user's emotional state to calculate the optimal boarding location (car and boarding door) for each user. This calculation uses an algorithm that references past congestion data and learns congestion patterns by time of day and day of the week.

[1019] Guide Phase

[1020] Notification of optimal pickup location

[1021] The server generates a message informing each user of the optimal boarding location based on the calculation results, and the terminal displays this notification information to the user, providing specific instructions in the form of text and maps.

[1022] Feedback Phase

[1023] Collecting post-ride feedback

[1024] The terminal provides an interface to accept feedback from users after their ride. Users can provide feedback through the app regarding the congestion situation and emotional state during the ride.

[1025] The server analyzes the feedback it receives and uses it as data to improve prediction accuracy and emotion recognition accuracy in future trials.

[1026] Specific examples

[1027] For example, during the morning rush hour, User A sends location information from his mobile device on his way from his home to the nearest station. The device then uses an emotion engine to detect User A's stress level. The server then suggests the least crowded car based on the congestion data within the station and User A's emotional state, and notifies the device, "Car 3 is the least crowded. Please wait in front of Car 3." As a result, User A can commute comfortably without feeling stressed.

[1028] Prompt Sentence Examples

[1029] "Design an AI model to improve the accuracy of emotion recognition. The goal is to analyze the user's emotional state in real time and suggest the optimal boarding location based on the congestion situation. Specifically, identify the user's emotional state (stress, anxiety, comfort, etc.) from facial images and voice data, and use the results and in-car congestion data to provide optimal boarding guidance."

[1030] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1031] Step 1:

[1032] The device collects location information using the GPS function of the mobile device. When the user is near a station or an autonomous vehicle, the device periodically transmits the location information to the server.

[1033] Input: GPS data from your mobile device

[1034] Output: User's current location

[1035] Specific operation: The device acquires GPS data every 5 seconds and sends it to the server.

[1036] Step 2:

[1037] The server collects image data in real time from monitoring devices (cameras and sensors) installed within stations and autonomous vehicles.

[1038] Input: Real-time image data from a surveillance device

[1039] Output: Image information of station premises and train interiors

[1040] Specific operation: The server continuously receives the video stream transmitted from the monitoring device.

[1041] Step 3:

[1042] The server analyzes the collected image data using an image analysis algorithm (e.g., OpenCV) and calculates the congestion level of each vehicle.

[1043] Input: Collected image data

[1044] Output: Crowding level of each vehicle (empty, normal, crowded categories)

[1045] Specific operation: The server counts the number of people in the image, calculates the occupancy rate, and classifies the congestion level into categories.

[1046] Step 4:

[1047] The device uses an emotion engine (e.g., Microsoft's Azure Emotion API or Google Cloud's Vision API) to analyze the user's emotional state.

[1048] Input: Device camera images, audio data, user operation patterns

[1049] Output: User's emotional state (stressed, anxious, comfortable, etc.)

[1050] Specific operation: The device uses a camera to capture the user's facial expressions and analyzes them using an emotion analysis engine.

[1051] Step 5:

[1052] The server integrates the congestion data and the user's emotional state to calculate the optimal boarding location.

[1053] Input: crowding data, user emotional state

[1054] Output: Optimal boarding position (vehicle and boarding door)

[1055] Specific operation: The server uses an algorithm to calculate the optimal boarding location for each user, and also refers to past data to take into account future congestion conditions.

[1056] Step 6:

[1057] The server generates a message informing the user of the optimal boarding location based on the calculation results.

[1058] Input: Calculated optimal pickup location

[1059] Output: A message to inform the user

[1060] Specific operation: The server generates a text message and map-style directions including the best pickup location and sends them to the device.

[1061] Step 7:

[1062] The terminal displays the notification information received from the server to the user and provides specific instructions.

[1063] Input: A notification message to the user

[1064] Output: Instructions displayed to the user

[1065] Specific operation: The device uses the display and voice guidance functions to provide the user with information about the boarding location.

[1066] Step 8:

[1067] Users provide feedback through the app after their ride.

[1068] Input: User feedback information

[1069] Output: Feedback data sent to the server

[1070] What it does: Users use the app interface to input feedback about congestion and emotional state.

[1071] Step 9:

[1072] The server analyzes the feedback it receives and uses it to improve prediction accuracy and emotion recognition accuracy in future trials.

[1073] Input: User feedback data

[1074] Output: Improved parsing algorithm

[1075] Specific operation: The server stores the feedback data in a database and uses it to adjust and improve the analysis algorithm.

[1076] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1077] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1078] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1079] [Third embodiment]

[1080] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1081] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1082] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1083] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1084] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1085] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1086] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1087] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1088] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1089] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1090] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1091] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1092] This invention is a system that uses location information from a mobile device and image data collected from monitoring devices in and around stations to appropriately guide passengers to the vehicle they are boarding. Specific embodiments based on the system program are described below.

[1093] Data Collection Phase

[1094] Collection of mobile phone location information

[1095] Device: Each user's mobile device periodically (for example, every 5 seconds) transmits its location information to the server. This information is sent using GPS or beacons.

[1096] User: Before arriving at the station, you need to enable location services on your mobile device.

[1097] Data collection from monitoring devices

[1098] Server: Collects image data in real time from monitoring devices (cameras and sensors) inside and around stations. This data is later used to analyze congestion levels.

[1099] Data analysis phase

[1100] Understanding congestion levels through image analysis

[1101] Server: Analyzes image data collected from the monitoring devices and runs algorithms to calculate the congestion level of each vehicle, including the number of passengers and the space occupancy rate.

[1102] Server: Based on the results of image analysis, classify each vehicle into categories such as "empty," "normal," or "crowded."

[1103] Integrated analysis of location information

[1104] Server: Aggregates location information from each user's mobile device and analyzes the location and destination information of all users, thereby estimating the current number of passengers in each vehicle.

[1105] Optimization Phase

[1106] Optimization calculation of boarding position

[1107] Server: Based on the collected location information and image data, the server calculates the optimal boarding location for the user. For example, an algorithm is designed to prioritize suggesting vehicles with low congestion levels.

[1108] Server: Refers to past congestion data and learns congestion patterns by time of day and day of the week to predict future congestion conditions.

[1109] Guide Phase

[1110] Notification of optimal pickup location

[1111] Server: Based on the calculation results, the server notifies each user of the optimal boarding location (vehicle and door location). Notifications are sent to the device in real time.

[1112] Device: Displays received notification information to the user and provides specific instructions in the form of text and maps.

[1113] Feedback Phase

[1114] Collecting post-ride feedback

[1115] User: After the ride, the user provides feedback through the app about the congestion situation during the ride, which is then sent to the server.

[1116] Terminal: Provides an interface for collecting feedback and accepts input from the user.

[1117] Processing Feedback

[1118] Server: Analyzes the received feedback and uses it to improve the accuracy of future congestion predictions and boarding location suggestions.

[1119] Specific examples

[1120] Morning rush hour scenario

[1121] 1. Device: User A leaves home and activates the location information on his mobile device on his way to the nearest station.

[1122] 2. Server: Obtains user A's location information and sends it to the server. At the same time, it uses augmented reality (AR) functions to confirm the user's location.

[1123] 3. Server: Acquires image data from station monitoring devices and analyzes it.

[1124] 4. Server: Based on the analysis results, it determines that car 3 is currently the vacant car, and notifies user A, "Car 3 is the vacant car. Please wait in front of car 3."

[1125] 5. Terminal: Display a message to user A, prompting him to move in front of car 3.

[1126] 6. User: Follow the instructions to stand in front of car 3, get in, and provide feedback through the app.

[1127] 7. Server: Receives User A's feedback and improves the accuracy of the next prediction based on the analysis results.

[1128] In this way, the system based on the present invention provides a concrete means for managing the congestion status of each vehicle in real time and providing passengers with a comfortable commuter life.

[1129] The processing flow will be explained below.

[1130] Step 1: Collect location information

[1131] Terminal: The mobile terminal periodically (for example, every 5 seconds) collects location information obtained from GPS and beacon sensors and prepares to send it to the server.

[1132] Step 2: Send location information

[1133] Device: Sends location information to the server, which notifies the server of the user's current location in real time.

[1134] Step 3: Collecting data from monitoring devices

[1135] Server: Image data is collected in real time from monitoring devices installed inside and around stations. This image data is used to understand the congestion status of each train car.

[1136] Step 4: Analyzing the image data

[1137] Server: Runs algorithms to analyze the collected image data. Specifically, it uses image analysis technology to calculate the number of passengers and space occupancy in each vehicle.

[1138] Step 5: Calculate congestion

[1139] Server: Based on the results of image analysis, the congestion level of each vehicle is classified into categories such as "empty," "normal," or "crowded."

[1140] Step 6: Integrated analysis of location information

[1141] Server: Aggregates location information sent from mobile devices and analyzes each passenger's current location and destination, thereby estimating the number of passengers in each vehicle.

[1142] Step 7: Calculate the best pickup location

[1143] Server: Based on the analysis results, the server runs an algorithm that calculates the optimal boarding location (vehicle and boarding door) for each user.

[1144] Step 8: Generate notifications

[1145] Server: Based on the calculation results, a message is generated to notify each user of the specific boarding location.

[1146] Step 9: Sending notifications

[1147] Server: Sends the generated message to the user's mobile device. Example: "Car 3 is the least crowded. Please wait in front of car 3."

[1148] Step 10: Displaying notifications

[1149] Terminal: Displays the received message to the user, so that the user can check which car to board.

[1150] Step 11: Ride

[1151] User: Follow the notification and move to the designated vehicle and get in.

[1152] Step 12: Provide feedback

[1153] User: After the ride, enter information about the ride's congestion and other feedback through the app.

[1154] Step 13: Submit your feedback

[1155] Terminal: Sends the feedback entered by the user to the server.

[1156] Step 14: Analyze feedback

[1157] Server: Analyzes the received feedback and uses it as data to improve the accuracy of future predictions.

[1158] Through these steps, the system can grasp the congestion status of each vehicle in real time and suggest the optimal boarding location to the user.

[1159] Example 1

[1160] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1161] In modern urban areas, congestion during commutes has become a major social problem. For commuters, especially those using trains, crowded trains can cause significant stress and discomfort. Conventional methods struggle to grasp congestion levels in real time and provide users with optimal boarding locations, resulting in a lack of information for passengers to determine the appropriate boarding location. Therefore, an effective system is needed to alleviate congestion and ensure a comfortable ride.

[1162] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1163] In this invention, the server includes means for collecting location information from the mobile device, means for collecting image data from the monitoring device, means for analyzing the location information and the image data to calculate the congestion level of each vehicle, means for learning congestion patterns by time period and day of the week based on past data and predicting future congestion conditions, means for calculating the optimal boarding position based on the analysis results and prediction results and notifying the mobile device, means for receiving feedback from the user and improving the accuracy of the analysis and prediction, and means for notifying the mobile device of the optimal boarding position in text or map format. This makes it possible to grasp the congestion status of each vehicle in real time and provide the user with optimal boarding guidance.

[1164] "Mobile device" refers to a portable electronic device that is carried by a user and used by the user, such as a mobile phone, smartphone, or tablet.

[1165] "Location Information" refers to data indicating a specific geographic location obtained through GPS, beacons, Wi-Fi, cell towers, etc.

[1166] "Monitoring devices" refers to devices such as cameras, sensors, and beacons installed in or around stations that monitor the surrounding conditions and collect and provide data.

[1167] "Image data" refers to visual data in the form of still images or video captured or acquired by surveillance equipment.

[1168] "Crowding level" is an indicator of the density of people in a particular area or vehicle, and is calculated based on the number of passengers and the occupancy rate of space.

[1169] "Analysis" refers to the process of analyzing collected data in detail to extract meaning and patterns and derive results according to the intended purpose.

[1170] "Boarding location" refers to the specific car and its door location that the user selects to board the train.

[1171] "Notification" refers to the act of sending a message to a mobile device to notify the user of the results of a calculation or analysis.

[1172] "Past data" refers to data collected in the past, and primarily refers to past congestion conditions and user usage patterns.

[1173] "Feedback" is input data provided by users based on their experiences and opinions, which is used to reevaluate and improve the system.

[1174] "Forecasting" refers to the process of predicting and estimating future situations based on collected data and analysis results.

[1175] This invention is a system that provides optimal boarding guidance to passengers by utilizing location information from mobile devices and image data acquired from monitoring devices in and around stations. This system is implemented using the following means.

[1176] Data Collection Phase

[1177] Collection of mobile phone location information

[1178] Device: Each user's mobile device periodically transmits location information to the server. Location information is collected using GPS or beacon technology (e.g., Bluetooth beacons). Specifically, the device activates location services, measures its current location every 5 seconds, and transmits it to the server using a secure communication protocol (e.g., HTTPS).

[1179] User: Before arriving at the station, the user must enable location services on their mobile device and grant location sharing permission to the app.

[1180] Data collection from monitoring devices

[1181] Server: The server collects image data in real time from monitoring devices inside and around stations. The monitoring devices include cameras and sensors, and receives image data as a stream from these devices. The received data is temporarily stored in storage and used for subsequent analysis.

[1182] Data analysis phase

[1183] Understanding congestion levels through image analysis

[1184] Server: The server uses image analysis algorithms such as OpenCV and TensorFlow to analyze the collected image data. Specifically, it detects passengers, counts the number of people, and analyzes space occupancy for each frame, and classifies the congestion level of each car into categories of "empty," "normal," or "crowded."

[1185] Server: The server stores these analysis results and uses them for subsequent pickup location suggestions.

[1186] Integrated analysis of location information

[1187] Server: The server aggregates the location information collected from each user's mobile device and plots it on a map, allowing the server to determine the current number of passengers in each car and the number of passengers in a specific area of ​​the station.

[1188] Optimization Phase

[1189] Optimization calculation of boarding position

[1190] Server: The server calculates the optimal boarding location for each user based on the aggregated location information and analyzed image data. The algorithm references past congestion data and learns congestion patterns by time of day and day of the week to predict future congestion conditions.

[1191] Server: Based on the calculation results, the optimal boarding location is determined and saved.

[1192] Guide Phase

[1193] Notification of optimal pickup location

[1194] Server: Based on the calculation results, the server notifies each user of the optimal boarding location in real time. The notification content includes specific instructions, such as "Car 3 is the least crowded. Please wait in front of car 3."

[1195] Device: The device displays the received notification to the user in an intuitive format, such as a text message or a pin on a map.

[1196] Feedback Phase

[1197] Collecting post-ride feedback

[1198] User: After the ride, the user provides feedback through the app about the ride's congestion, with options such as "very crowded," "normal," and "empty."

[1199] Terminal: The terminal provides an interface for collecting feedback and accepts input from the user, which is then sent to the server.

[1200] Processing Feedback

[1201] Server: The server analyzes the received feedback and improves the accuracy of congestion predictions and boarding location suggestions for the next trip. The analysis integrates the feedback items and performs statistical evaluation.

[1202] Specific examples

[1203] Morning rush hour scenario

[1204] 1. Device: User A leaves home and activates the location information service on his mobile device on his way to the nearest station.

[1205] 2. Server: The server receives and stores the location information of User A. At the same time, it visually confirms User A's location using the augmented reality (AR) function.

[1206] 3. Server: The server acquires image data from the station monitoring devices and analyzes it in real time.

[1207] 4. Server: Based on the analysis results, it determines that car 3 is the least crowded and notifies user A, "Car 3 is the least crowded. Please wait in front of car 3."

[1208] 5. Terminal: The terminal displays a notification to User A, instructing him to move to the front of car 3. The notification is indicated by a text message and a pin on the map.

[1209] 6. User: User A follows the instructions to stand in front of car 3 and provides feedback on the congestion situation through the app after boarding.

[1210] 7. Server: The server receives User A's feedback and uses the analysis results to improve the accuracy of the next prediction.

[1211] Prompt Sentence Examples

[1212] Prompt: "Describe the steps for a system that uses station surveillance cameras and mobile device location information to provide optimal ride guidance."

[1213] This system is designed to grasp the congestion status of each vehicle in real time and provide users with a comfortable commute.

[1214] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1215] Step 1:

[1216] Collection of mobile phone location information

[1217] Input: Location information obtained from the mobile device's built-in sensors (GPS data, beacon signals, etc.)

[1218] Device behavior: The device activates location services, measures its current location every 5 seconds, and sends the measurements to a server using a secure communication protocol such as HTTPS.

[1219] Output: The user's current location is sent to the server.

[1220] Step 2:

[1221] Image data collection from monitoring devices

[1222] Input: Real-time image streams from surveillance devices (cameras, sensors) installed in and around the station

[1223] Server operation: The server establishes a secure connection with the monitoring device, receives image data in real time, and temporarily stores the received data in storage.

[1224] Output: Image data saved in storage

[1225] Step 3:

[1226] Understanding congestion levels through image analysis

[1227] Input: Image data stored in storage

[1228] Server operation: The server analyzes image data using image analysis algorithms such as OpenCV and TensorFlow. It detects passengers, counts the number of people, and analyzes space occupancy for each frame.

[1229] Output: Analyzed congestion data (e.g., empty, normal, crowded)

[1230] Step 4:

[1231] Congestion category classification for each vehicle

[1232] Input: Crowd data obtained by image analysis

[1233] Server operation: The server classifies each vehicle into a category such as "empty," "normal," or "crowded" based on the congestion data, using a set threshold.

[1234] Output: Congestion category information for each vehicle

[1235] Step 5:

[1236] Integrated analysis of location information

[1237] Input: Location data from each user's mobile device

[1238] Server operation: The server aggregates location information and plots it on a map. It analyzes overall trends and calculates the number of users concentrated in a particular car or in a particular area of ​​the station.

[1239] Output: Integrated analysis results based on user location information

[1240] Step 6:

[1241] Optimization calculation of boarding position

[1242] Input: Integrated analysis results of location information and congestion category information

[1243] Server operation: The server uses an algorithm to calculate the optimal boarding location for the user. It learns from past data and congestion patterns by time of day and day of the week, and runs an algorithm to predict future congestion conditions.

[1244] Output: Optimal pickup location information

[1245] Step 7:

[1246] Notification of optimal pickup location

[1247] Input: Best Pickup Location

[1248] Server operation: Based on the calculation results, the server notifies each user of the optimal boarding location in real time. The notification message includes specific instructions such as "Car 3 is the least crowded. Please wait in front of car 3."

[1249] Device behavior: The device displays notifications to the user in intuitive formats such as text messages or pins on a map.

[1250] Output: A notification message that is displayed to the user

[1251] Step 8:

[1252] Collecting post-ride feedback

[1253] Input: User feedback information

[1254] User Action: After the ride, the user provides feedback through the app about the ride's congestion, with options such as "very crowded," "normal," and "empty."

[1255] Device behavior: The device provides an interface for collecting feedback and accepts input from the user, which is then sent to the server.

[1256] Output: Feedback data sent to the server

[1257] Step 9:

[1258] Processing Feedback

[1259] Input: Feedback data

[1260] Server operation: The server analyzes the received feedback and uses it to improve the accuracy of future congestion predictions and boarding location suggestions. The analysis integrates the feedback items and performs statistical evaluation.

[1261] Output: Improved congestion prediction data and pickup location suggestions

[1262] (Application example 1)

[1263] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1264] Technologies that grasp the congestion situation in and around train stations and guide passengers to the optimal boarding location are widely used, but the application of these technologies to avoiding congestion in commercial facilities has not been sufficiently considered. While the importance of avoiding congestion in such commercial facilities is increasing, current systems lack the technology to analyze the congestion situation in real time and provide users with appropriate routes and areas. The objective of this invention is to provide a system that provides appropriate congestion avoidance guidance in order to provide a comfortable shopping experience in commercial facilities.

[1265] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1266] In this invention, the server includes means for collecting location information from mobile devices, means for collecting image data from monitoring devices, means for analyzing the location information and the image data to calculate the congestion level of each area, means for calculating the optimal route and area based on the analysis results and notifying the mobile devices, means for obtaining feedback from users and improving the accuracy of the analysis and prediction, and means for predicting future congestion conditions using past data. This makes it possible to grasp the congestion status in commercial facilities in real time and provide appropriate congestion avoidance guidance.

[1267] "Mobile device" refers to a device that a user can carry with them and that can transmit location information, such as a mobile phone, smartphone, or tablet.

[1268] "Location information" refers to data transmitted by a mobile device indicating its current geographical location, and is information obtained using GPS or beacons.

[1269] "Surveillance equipment" refers to devices such as cameras and sensors installed in commercial facilities or specific areas that monitor the surrounding situation in real time and collect image data.

[1270] "Image data" refers to video and photographic data collected by surveillance equipment, and is information that can be analyzed to determine the level of congestion in an area.

[1271] "Crowding level" is an index that indicates the density or occupancy rate of people in a specific area, and refers to information calculated from analyzed image data.

[1272] "Calculating routes and areas" refers to the process of deriving the optimal travel route and areas to visit for a user based on collected location information and image data.

[1273] "Means of notification" refers to the method of sending the calculated optimal route and area guidance information to the user's mobile device and displaying it.

[1274] "Feedback" refers to information provided by users about their experiences and opinions, which is collected and analyzed to help improve the accuracy of the system.

[1275] "Prediction means" refers to algorithms and methods for referencing past congestion data and estimating future congestion conditions.

[1276] "Commercial facilities" refers to facilities such as shopping malls, department stores, and shops that are visited by many people and may become crowded.

[1277] This invention relates to a system that provides guidance for avoiding crowds in commercial facilities. It integrates location information from mobile devices and image data collected from monitoring devices to guide users to the optimal route and area in real time. It is also a system that aims to further improve accuracy by collecting feedback from users.

[1278] Data Collection Phase

[1279] Mobile location information collection:

[1280] Device: Each user's mobile device periodically sends its location information to the server, using GPS or beacons.

[1281] Users: Must enable location services on their mobile device before arriving at the commercial establishment.

[1282] Data collection from monitoring devices:

[1283] Server: Collects image data in real time from monitoring devices (cameras and sensors) installed within commercial facilities. This data is used to analyze congestion levels.

[1284] Data Analysis Phase

[1285] Image analysis to understand congestion levels:

[1286] Server: Analyzes image data collected from the monitoring devices and runs an algorithm to calculate the congestion level of each area, specifically the number of users and the occupancy rate of the space.

[1287] Server: Based on the results of image analysis, the congestion level of each area is classified as "low," "normal," "high," etc.

[1288] Integrated location analysis:

[1289] Server: Aggregates location information from each user's mobile device and analyzes the location and destination information of all users to estimate the current congestion situation in each area.

[1290] Route and area optimization phase

[1291] Best route and area suggestions:

[1292] Server: Calculates the optimal route and area for the user based on the collected location information and image data. An algorithm is built in to prioritize suggesting areas with low congestion.

[1293] Server: Refers to past congestion data, learns congestion patterns by time of day and day of the week, and predicts future congestion conditions.

[1294] Guide Phase

[1295] Best route and area notifications:

[1296] Server: Based on the calculation results, the server notifies each user of the optimal route and area information. Notifications are sent to the device in real time.

[1297] Device: Displays received notification information to the user and provides specific instructions in the form of text and maps.

[1298] Feedback Phase

[1299] Collecting post-ride feedback:

[1300] User: After moving around the shopping mall, the user provides feedback about the congestion situation through the app, which is then sent to the server.

[1301] Terminal: Provides an interface for collecting feedback and accepts input from the user.

[1302] Feedback processing:

[1303] Server: Analyzes the received feedback and uses it to improve the accuracy of congestion predictions and route guidance for future trips.

[1304] Hardware and software used:

[1305] Hardware: smartphones, surveillance cameras, sensors.

[1306] Software: Python, image analysis algorithm, location analysis algorithm, external API (for acquiring congestion data).

[1307] Examples:

[1308] A user arrives at a shopping mall at 10:00 a.m. After arriving at shopping mall A, the user opens the app and is directed to the least crowded area. After enjoying shopping, the user provides feedback to the app.

[1309] Example prompt: "Current location: 10:00. Please guide me to the areas in Shopping Mall A. Please recommend the least crowded area, taking into account the congestion status of each area. After making the recommendation, please also display a prompt to provide feedback to the user. The feedback content should be an interface where the user can write about the congestion status of the shopping experience."

[1310] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1311] Step 1:

[1312] Collection of mobile phone location information

[1313] Input: The user enables location services on their mobile device.

[1314] Processing: The device periodically (for example, every 5 seconds) obtains its own location information using GPS or beacons and generates location information data.

[1315] Output: The acquired location data is sent to the server.

[1316] Specific operation: The GPS module in the smartphone measures the current location, and the terminal app collects that data and sends it to a specified server.

[1317] Step 2:

[1318] Data collection from monitoring devices

[1319] Input: Image data provided by surveillance equipment within a commercial facility.

[1320] Processing: The server collects image data from the monitoring devices in real time and prepares the image data for storage and processing.

[1321] Output: The collected image data is stored in the server.

[1322] Specific operation: The server receives the video captured by the surveillance camera and stores it in storage. The image data is also organized in a specific format.

[1323] Step 3:

[1324] Understanding congestion levels through image analysis

[1325] Input: Collected image data.

[1326] Processing: The server uses image analysis algorithms to calculate the congestion level of each area, which is the process of calculating the number of users and the occupancy rate of the space.

[1327] Output: Obtains congestion information for each area.

[1328] What it does: Server-supported image analysis software counts the number of people in the image and calculates the area occupancy rate.

[1329] Step 4:

[1330] Integrated analysis of location information

[1331] Input: Location data and congestion information from user devices.

[1332] Processing: The server combines the location data and congestion information to estimate the overall congestion situation for each area.

[1333] Output: Consolidated congestion data is obtained.

[1334] What it does: The server uses a geographic information system (GIS) to map users' locations and area congestion levels, and then aggregates the data.

[1335] Step 5:

[1336] Optimal route and area suggestions

[1337] Input: Consolidated crowding data.

[1338] Processing: The server uses an algorithm to calculate the best route and area for each user, prioritizing less congested areas.

[1339] Output: The calculated optimal route and area information is obtained.

[1340] Specific operation: The server uses specific rules and algorithms (e.g., shortest path algorithms or congestion-based optimization methods) to determine the most suitable route and area for the user.

[1341] Step 6:

[1342] Optimal route and area notifications

[1343] Input: Calculated optimal route and area information.

[1344] Processing: The server sends the optimal route and area information to the user's mobile device.

[1345] Output: A real-time notification is displayed on the user's mobile device.

[1346] What it does: Uses the notification system (push notifications and in-app notifications) to display text and map-based directions on the user's smartphone.

[1347] Step 7:

[1348] Collecting and Processing Feedback

[1349] Input: Feedback information provided by the user.

[1350] Processing: The device provides an interface for collecting feedback and accepts user input. The server analyzes the received feedback and uses it to improve the accuracy of future guidance.

[1351] Output: Parsed feedback data.

[1352] Specific operation: Collects information provided by users through feedback forms and survey functions within the app, and the server analyzes the data to perform feedback analysis.

[1353] Prompt Sentence Examples

[1354] "Current location: 10:00. Please guide the user to the areas in the shopping mall. Please recommend the least crowded area, taking into account the congestion level of each area. After making the recommendation, please also display a prompt to provide feedback to the user. The feedback should be an interface where the user can describe how crowded the shopping experience was."

[1355] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1356] The present invention combines an emotion engine with a system that uses location information from mobile devices and image data collected from monitoring devices in and around stations to provide appropriate guidance on which vehicle to board. Specific embodiments based on the system program are described below.

[1357] Data Collection Phase

[1358] Collection of mobile phone location information

[1359] Device: Each user's mobile device periodically (for example, every 5 seconds) transmits its location information to the server. This information is sent using GPS or beacons.

[1360] User: Before arriving at the station, you need to enable location services on your mobile device.

[1361] Data collection from monitoring devices

[1362] Server: Collects image data in real time from monitoring devices (cameras and sensors) installed inside and around stations. This image data is used to understand the congestion status of each train car.

[1363] Data analysis phase

[1364] Understanding congestion levels through image analysis

[1365] Server: Runs algorithms to analyze the collected image data. Specifically, it uses image analysis technology to calculate the number of passengers and space occupancy in each vehicle.

[1366] Server: Based on the results of image analysis, the congestion level of each vehicle is classified into categories such as "empty," "normal," or "crowded."

[1367] Integrated analysis of location information

[1368] Server: Aggregates location information from each user's mobile device and analyzes the location and destination information of all users, thereby estimating the number of passengers in each vehicle.

[1369] Optimization Phase

[1370] Optimization calculation of boarding position

[1371] Server: Based on the analysis results, the server runs an algorithm that calculates the optimal boarding location (vehicle and boarding door) for each user.

[1372] Server: Refers to past congestion data and learns congestion patterns by time of day and day of the week to predict future congestion conditions.

[1373] Emotion Recognition Phase

[1374] Using the Emotion Engine

[1375] Device: The emotion engine recognizes the user's emotions using facial expression analysis via a camera, voice analysis, and the user's operation patterns.

[1376] Server: Analyzes the emotional data obtained from the emotion engine and understands the user's current emotional state.

[1377] Emotion-Based Optimization

[1378] Server: Integrates the user's emotional data and analysis results, and adjusts the recommended ride locations based on the user's emotional state. For example, for users with high levels of stress or anxiety, it will prioritize suggesting less crowded vehicles.

[1379] Guide Phase

[1380] Notification of optimal pickup location

[1381] Server: Based on the calculation results, it generates a message to notify each user of the optimal boarding location.

[1382] Device: Displays received notification information to the user and provides specific instructions in the form of text and maps.

[1383] Feedback Phase

[1384] Collecting post-ride feedback

[1385] User: After the trip, provide feedback through the app about the ride's congestion and emotional state.

[1386] Terminal: Provides an interface for collecting feedback and accepts input from the user.

[1387] Processing Feedback

[1388] Server: Analyzes the received feedback and uses it as data to improve prediction accuracy and emotion recognition accuracy in future trials.

[1389] Specific examples

[1390] Morning rush hour scenario

[1391] 1. Device: User A leaves home and activates the location information on his mobile device on his way to the nearest station.

[1392] 2. Server: Obtains user A's location information and sends it to the server. At the same time, it uses augmented reality (AR) functions to confirm the user's location.

[1393] 3. Server: Acquires image data from station monitoring devices and analyzes it.

[1394] 4. Server: Based on the analysis results, it determines that car 3 is currently the vacant car, and notifies user A, "Car 3 is the vacant car. Please wait in front of car 3."

[1395] 5. Device: The emotion engine analyzes user A's emotion data and detects his / her stress state.

[1396] 6. Server: Consider user A’s emotional state and further adjust the pickup location suggestions.

[1397] 7. Terminal: Display a message to user A, prompting him to move in front of car 3.

[1398] 8. User: Follow the instructions to stand in front of car 3, get in, and provide feedback through the app.

[1399] 9. Server: Receives User A's feedback and improves the accuracy of the next prediction based on the analysis results.

[1400] In this way, the system based on the present invention provides a concrete means for providing users with a comfortable commuter life by managing the congestion status of each vehicle in real time and proposing the optimal boarding location taking into account the user's emotional state.

[1401] The processing flow will be explained below.

[1402] Step 1: Collect location information

[1403] Terminal: The mobile terminal periodically (for example, every 5 seconds) collects location information obtained from GPS and beacon sensors and prepares to send it to the server.

[1404] Step 2: Send location information

[1405] Device: Sends location information to the server, which notifies the server of the user's current location in real time.

[1406] Step 3: Collecting data from monitoring devices

[1407] Server: Obtains image data in real time from monitoring devices (cameras and sensors) installed inside and around stations. This image data is used to understand the congestion status of each train car.

[1408] Step 4: Analyzing the image data

[1409] Server: Runs algorithms to analyze the collected image data. Specifically, it uses image analysis technology to calculate the number of passengers and space occupancy in each vehicle.

[1410] Step 5: Calculate congestion

[1411] Server: Based on the results of image analysis, the congestion level of each vehicle is classified into categories such as "empty," "normal," or "crowded."

[1412] Step 6: Integrated analysis of location information

[1413] Server: Aggregates location information sent from mobile devices and analyzes each passenger's current location and destination, thereby estimating the number of passengers in each vehicle.

[1414] Step 7: Calculate the best pickup location

[1415] Server: Based on the analysis results, the server runs an algorithm that calculates the optimal boarding location (vehicle and boarding door) for each user.

[1416] Step 8: Use historical data

[1417] Server: Refers to past congestion data and learns congestion patterns by time of day and day of the week to predict future congestion conditions.

[1418] Step 9: Emotion Recognition with the Emotion Engine

[1419] Device: The emotion engine recognizes the user's emotions by analyzing facial expressions via a camera, voice analysis, and the user's operation patterns.

[1420] Step 10: Sending Emotion Data

[1421] Device: Sends the recognized emotion data to the server, which informs the server of the user's current emotional state.

[1422] Step 11: Adjusting riding position based on emotions

[1423] Server: Integrates the user's emotional data and analysis results to tailor the recommended ride locations based on their emotional state. For example, prioritize less crowded vehicles for users with high levels of stress.

[1424] Step 12: Generate and send notifications

[1425] Server: Based on the calculation results and emotion data, a message is generated to notify each user of their specific boarding location and sent to the device.

[1426] Step 13: Viewing Notifications

[1427] Terminal: Displays the received message to the user, so that the user can check which car they should board.

[1428] Step 14: Ride

[1429] User: Follow the notification and move to the designated vehicle and get in.

[1430] Step 15: Provide feedback

[1431] User: After the ride, users provide feedback through the app about the ride's congestion and emotional state.

[1432] Step 16: Submit your feedback

[1433] Terminal: Sends the feedback entered by the user to the server.

[1434] Step 17: Analyze feedback

[1435] Server: Analyzes the received feedback and uses it as data to improve prediction accuracy and emotion recognition accuracy in future trials.

[1436] Through these steps, the system can grasp the congestion status of each vehicle in real time and suggest the optimal boarding location taking into account the user's emotional state.

[1437] Example 2

[1438] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1439] Conventional ride guidance systems were able to grasp the congestion level of each vehicle using location information and image data, but did not provide ride guidance that took into account the user's emotional state. As a result, riding in a crowded vehicle could cause stress and discomfort to the user. It was particularly difficult to provide a comfortable ride experience for users during rush hour situations.

[1440] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for collecting location information from a mobile terminal, means for collecting image data from a monitoring device, means for analyzing the location information and the image data to calculate the congestion level of each vehicle, means for grasping the emotional state of the user using an emotion recognition engine, and means for calculating the optimal boarding position based on the analysis result and the emotional state and notifying the mobile terminal. This makes it possible to propose an optimal boarding position taking into account the user's emotional state in addition to the user's location information and congestion status.

[1441] A "mobile terminal" is a communication device carried by a user that provides location information and emotional state data.

[1442] "Location information" refers to geographic coordinate information collected from mobile devices and GPS sensors.

[1443] "Monitoring devices" are equipment such as cameras and sensors that are installed in or around stations to collect image data.

[1444] "Image data" refers to video and photo data acquired from a surveillance device.

[1445] "Analysis" involves processing the collected data and calculating the congestion level of each vehicle and the emotional state of the user.

[1446] "Crowding level" refers to the degree of congestion based on the number of passengers and space occupancy in each vehicle.

[1447] An "emotion recognition engine" is software for analyzing a user's emotional state, including facial expression analysis and voice analysis using a camera and microphone.

[1448] "Notification" refers to the act of sending the calculated optimal boarding location to the user's mobile device.

[1449] "Feedback" is information provided by users about the congestion situation and their emotional state after riding.

[1450] This invention combines an emotion engine with a system that uses location information from mobile devices and image data collected from monitoring devices in and around stations to provide appropriate guidance on which vehicle to board. Specific embodiments of the system are described below.

[1451] Data Collection Phase

[1452] Collection of mobile phone location information

[1453] Device: Each user's mobile device sends its location information to the server every 5 seconds. This information is obtained using GPS or beacons. Specifically, the device's built-in GPS sensor is used to obtain location information, which is then compressed and sent to the server.

[1454] User: Before arriving at the station, they will need to enable location services on their mobile device. A notification will appear on the device prompting them to enable the service.

[1455] Data collection from monitoring devices

[1456] Server: Collects image data in real time from monitoring devices (cameras and sensors) installed in and around the station. Specifically, it acquires video streams from cameras installed on platforms and at entrances and exits.

[1457] Server: Temporarily stores collected image data in storage for subsequent analysis.

[1458] Data analysis phase

[1459] Understanding congestion levels through image analysis

[1460] Server: Executes algorithms to analyze collected image data. Specifically, it applies image analysis technology using deep learning to calculate the number of passengers in each vehicle and the occupancy rate of space. For example, it identifies the positions of passengers' faces and bodies based on images acquired from surveillance cameras and calculates the degree of congestion.

[1461] Server: Based on the calculated congestion level, each vehicle is classified into a category such as "empty," "normal," or "crowded." This classification result is stored in a database and used in subsequent phases.

[1462] Integrated analysis of location information

[1463] Server: Aggregates location information from each user's mobile device and analyzes the location and destination information of all users, using a location clustering algorithm to identify areas where passengers are concentrated.

[1464] Server: Combines location and congestion data to calculate the expected number of passengers in each vehicle.

[1465] Emotion Recognition Phase

[1466] Using the Emotion Engine

[1467] On-device: The device uses an emotion recognition engine to analyze the user's emotional state. Specifically, the device uses the camera to analyze facial expressions and the microphone to analyze voices. This allows the device to identify emotional states such as stress or anxiety.

[1468] Server: Analyzes the emotional data sent from the emotion recognition engine and understands the user's emotional state. This makes it possible to suggest the optimal boarding location based on the user's emotional state.

[1469] Guide Phase

[1470] Notification of optimal pickup location

[1471] Server: Calculates the optimized boarding location (specific vehicle and boarding door) and generates customized instruction messages for each user.

[1472] Terminal: The generated notification message is displayed to the user. For example, specific instructions such as "Currently, car number 3 is the vacant car. Please wait in front of car number 3" are displayed in text or map format.

[1473] Feedback Phase

[1474] Collecting post-ride feedback

[1475] Users: After their ride, they provide feedback through the app about the occupancy and their emotional state, specifically in the form of a short survey about their ride experience.

[1476] Terminal: Provides a feedback input interface and accepts data from the user.

[1477] Processing Feedback

[1478] Server: Analyzes the received feedback data and performs batch processing to improve prediction accuracy and emotion recognition accuracy from the next time onwards. This is expected to improve the accuracy of the entire system.

[1479] Specific examples

[1480] Morning rush hour scenario

[1481] 1. Device: User A leaves home and activates the location information on his mobile device on his way to the nearest station.

[1482] 2. Server: Obtains user A's location information and sends it to the server. At the same time, the server confirms the user's location using AR functions, etc.

[1483] 3. Server: Acquires image data from station monitoring devices and analyzes it.

[1484] 4. Server: Based on the analysis results, it determines that car 3 is currently the vacant car, and notifies user A, "Car 3 is the vacant car. Please wait in front of car 3."

[1485] 5. Device: The emotion engine analyzes user A's emotion data and detects his / her stress state.

[1486] 6. Server: Consider user A’s emotional state and further adjust the pickup location suggestions.

[1487] 7. Terminal: Display a message to user A, prompting him to move in front of car 3.

[1488] 8. User: Follow the instructions to stand in front of car 3, get in, and provide feedback through the app.

[1489] 9. Server: Receives User A's feedback and improves the accuracy of the next prediction based on the analysis results.

[1490] In an embodiment based on this scenario, user location information, station monitoring data, and emotion data can be integrated in real time to provide users with a comfortable riding experience.

[1491] Prompt Sentence Examples

[1492] Below are some example prompts to input to the generative AI model:

[1493] "Please describe in detail the process of a system that uses mobile device location information and station surveillance camera data to calculate the optimal boarding location for passengers. This should include data collection, analysis phase, use of an emotion engine, optimization, notification phase, and user feedback processing."

[1494] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1495] Step 1:

[1496] Device: The user's mobile device enables location services. Specifically, the device's GPS module acquires its current location every 5 seconds and transmits the location data to the server. This provides the server with the user's real-time location information as input data.

[1497] Step 2:

[1498] Server: Collects image data from surveillance devices (cameras and sensors) installed in and around the station. Specifically, it acquires video streams from each surveillance camera in real time and temporarily stores them in storage. This image data is used in the next image analysis phase.

[1499] Step 3:

[1500] Server: Analyzes the collected image data. Specifically, it applies image analysis technology using deep learning to calculate the number of passengers in each vehicle and the occupancy rate of space. It uses an object detection algorithm to identify and count passengers from the input image data. The analysis results output the degree of congestion in each vehicle.

[1501] Step 4:

[1502] Server: Based on the congestion analysis results, classify each vehicle into a category such as "empty," "normal," or "crowded." The input is the analysis result from step 3, which is applied to a category classification algorithm to output the classification result. This classification result is stored in a database.

[1503] Step 5:

[1504] Server: Aggregates location information sent from each user's mobile device and analyzes the overall location and destination information. Specifically, it uses a location clustering algorithm to identify areas where users are concentrated. The input is real-time location data, and the output is passenger density information for each area.

[1505] Step 6:

[1506] Device: The device uses an emotion recognition engine to analyze the user's emotional state. Specifically, facial expression analysis is performed using the device's camera, and voice analysis is performed using the microphone. The input is the user's facial expression image and voice data, and the output is the evaluation result of the user's emotional state.

[1507] Step 7:

[1508] Server: Analyzes the emotion data sent from the emotion recognition engine and understands the user's emotional state. The input is the emotion engine's output data, and analysis is performed based on this to evaluate the user's emotional state. Based on this evaluation result, the boarding location suggestions are optimized.

[1509] Step 8:

[1510] Server: Calculates the optimal pickup location based on the user's emotional state and generates a personalized notification message. The inputs are congestion data, user location data, and the user's emotional state data, which are combined to run an algorithm to calculate the optimal pickup location. The output is a notification message.

[1511] Step 9:

[1512] Terminal: Displays notification messages received from the server to the user. Specifically, it displays boarding location instructions in text and map format. For example, it displays a message such as "Currently, car 3 is the least crowded. Please wait in front of car 3." The input is the notification message from the server, and the output is the notification to the user.

[1513] Step 10:

[1514] User: Follows the instructions to go to the designated pickup location. After boarding, the user provides feedback on the congestion situation and emotional state during the ride through the app. The input is the user's experience data, and the output is feedback information.

[1515] Step 11:

[1516] Server: Analyzes the feedback data received from users and improves the algorithms to improve prediction accuracy and emotion recognition accuracy in the future. The input is user feedback data, and the output is an improved analysis and prediction algorithm.

[1517] (Application example 2)

[1518] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1519] The issue of congestion in public transportation and the inability to properly guide passengers to a comfortable boarding location are issues. Furthermore, providing boarding guidance without taking into account the passenger's emotional state increases stress and discomfort, degrading the overall riding experience.

[1520] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for collecting location information from the mobile terminal, means for collecting image data from the monitoring device, means for analyzing the location information and the image data to calculate the congestion level of each vehicle, means for analyzing the emotional state of the user, and means for calculating the optimal boarding position based on the analysis result and the emotional state and notifying the mobile terminal. This makes it possible to suggest the optimal boarding position taking into consideration the congestion level of the vehicle and the emotional state of the user, thereby improving the overall riding experience.

[1521] A "mobile terminal" is an electronic device that can be carried by a user and has GPS and Internet connectivity capabilities.

[1522] "Location information" is information that indicates coordinate data at a specific location using a GPS, beacon, or the like.

[1523] A "surveillance device" is a device that includes a camera, sensor, etc., and collects image data and environmental data of a designated area in real time.

[1524] "Image data" is visual information, including still images and video, collected by surveillance equipment.

[1525] "Crowding" is an indicator that expresses the density of people and objects in a particular area or space using a number or category.

[1526] "Emotional state" is data that indicates an individual's psychological state, and is obtained by facial expression analysis and voice analysis via a camera.

[1527] "Analysis results" are information obtained as a result of calculations and evaluations performed on collected data.

[1528] "Forecasting" is the process of predicting future conditions or events based on past and current data.

[1529] "Optimal boarding position" refers to the recommended position or door for the user to board comfortably and efficiently.

[1530] A "notification" is information sent from the system to the user, and may be provided in text, audio, visual format, or the like.

[1531] "Feedback" refers to information such as opinions, evaluations, and impressions collected from users, and is used to improve the system and increase the accuracy of analysis.

[1532] The "smart ride guidance system" of the present invention is a system that collects location information from mobile devices and image data from monitoring devices, and guides users to the optimal ride location taking into account their emotional state. A specific method for implementing this system is described below.

[1533] Data Collection Phase

[1534] Collection of location information from mobile devices

[1535] The terminal is an electronic device (such as a smartphone) carried by the user, which periodically collects location information using its GPS function and transmits it to the server. Location information collection begins when the user is near a station or an autonomous vehicle, and the server receives this information in real time.

[1536] Collection of image data from monitoring devices

[1537] The server collects image data from cameras and other sensors installed in stations and on autonomous vehicles. These devices provide the real-time visual information needed to understand the occupancy level of each carriage.

[1538] Data analysis phase

[1539] Understanding congestion levels through image analysis

[1540] The server uses image analysis algorithms (e.g., OpenCV) to analyze the collected image data, which calculates the number of passengers and space occupancy in each vehicle and classifies the level of congestion into categories of "empty," "normal," or "crowded."

[1541] Emotional state analysis

[1542] The device uses an emotion engine (e.g., Microsoft's Azure Emotion API or Google Cloud's Vision API) to analyze the user's emotions. By analyzing facial expressions and voice via the device's camera and the user's operation patterns, the device detects emotional states such as stress, anxiety, and comfort.

[1543] Optimization Phase

[1544] Optimization calculation of boarding position

[1545] The server combines congestion data and the user's emotional state to calculate the optimal boarding location (car and boarding door) for each user. This calculation uses an algorithm that references past congestion data and learns congestion patterns by time of day and day of the week.

[1546] Guide Phase

[1547] Notification of optimal pickup location

[1548] The server generates a message informing each user of the optimal boarding location based on the calculation results, and the terminal displays this notification information to the user, providing specific instructions in the form of text and maps.

[1549] Feedback Phase

[1550] Collecting post-ride feedback

[1551] The terminal provides an interface to accept feedback from users after their ride. Users can provide feedback through the app regarding the congestion situation and emotional state during the ride.

[1552] The server analyzes the feedback it receives and uses it as data to improve prediction accuracy and emotion recognition accuracy in future trials.

[1553] Specific examples

[1554] For example, during the morning rush hour, User A sends location information from his mobile device on his way from his home to the nearest station. The device then uses an emotion engine to detect User A's stress level. The server then suggests the least crowded car based on the congestion data within the station and User A's emotional state, and notifies the device, "Car 3 is the least crowded. Please wait in front of Car 3." As a result, User A can commute comfortably without feeling stressed.

[1555] Prompt Sentence Examples

[1556] "Design an AI model to improve the accuracy of emotion recognition. The goal is to analyze the user's emotional state in real time and suggest the optimal boarding location based on the congestion situation. Specifically, identify the user's emotional state (stress, anxiety, comfort, etc.) from facial images and voice data, and use the results and in-car congestion data to provide optimal boarding guidance."

[1557] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1558] Step 1:

[1559] The device collects location information using the GPS function of the mobile device. When the user is near a station or an autonomous vehicle, the device periodically transmits the location information to the server.

[1560] Input: GPS data from your mobile device

[1561] Output: User's current location

[1562] Specific operation: The device acquires GPS data every 5 seconds and sends it to the server.

[1563] Step 2:

[1564] The server collects image data in real time from monitoring devices (cameras and sensors) installed within stations and autonomous vehicles.

[1565] Input: Real-time image data from a surveillance device

[1566] Output: Image information of station premises and train interiors

[1567] Specific operation: The server continuously receives the video stream transmitted from the monitoring device.

[1568] Step 3:

[1569] The server analyzes the collected image data using an image analysis algorithm (e.g., OpenCV) and calculates the congestion level of each vehicle.

[1570] Input: Collected image data

[1571] Output: Crowding level of each vehicle (empty, normal, crowded categories)

[1572] Specific operation: The server counts the number of people in the image, calculates the occupancy rate, and classifies the congestion level into categories.

[1573] Step 4:

[1574] The device uses an emotion engine (e.g., Microsoft's Azure Emotion API or Google Cloud's Vision API) to analyze the user's emotional state.

[1575] Input: Device camera images, audio data, user operation patterns

[1576] Output: User's emotional state (stressed, anxious, comfortable, etc.)

[1577] Specific operation: The device uses a camera to capture the user's facial expressions and analyzes them using an emotion analysis engine.

[1578] Step 5:

[1579] The server integrates the congestion data and the user's emotional state to calculate the optimal boarding location.

[1580] Input: crowding data, user emotional state

[1581] Output: Optimal boarding position (vehicle and boarding door)

[1582] Specific operation: The server uses an algorithm to calculate the optimal boarding location for each user, and also refers to past data to take into account future congestion conditions.

[1583] Step 6:

[1584] The server generates a message informing the user of the optimal boarding location based on the calculation results.

[1585] Input: Calculated optimal pickup location

[1586] Output: A message to inform the user

[1587] Specific operation: The server generates a text message and map-style directions including the best pickup location and sends them to the device.

[1588] Step 7:

[1589] The terminal displays the notification information received from the server to the user and provides specific instructions.

[1590] Input: A notification message to the user

[1591] Output: Instructions displayed to the user

[1592] Specific operation: The device uses the display and voice guidance functions to provide the user with information about the boarding location.

[1593] Step 8:

[1594] Users provide feedback through the app after their ride.

[1595] Input: User feedback information

[1596] Output: Feedback data sent to the server

[1597] What it does: Users use the app interface to input feedback about congestion and emotional state.

[1598] Step 9:

[1599] The server analyzes the feedback it receives and uses it to improve prediction accuracy and emotion recognition accuracy in future trials.

[1600] Input: User feedback data

[1601] Output: Improved parsing algorithm

[1602] Specific operation: The server stores the feedback data in a database and uses it to adjust and improve the analysis algorithm.

[1603] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1604] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1605] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1606] [Fourth embodiment]

[1607] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1608] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1609] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1610] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1611] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1612] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1613] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1614] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1615] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1616] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1617] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1618] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1619] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1620] This invention is a system that uses location information from a mobile device and image data collected from monitoring devices in and around stations to appropriately guide passengers to the vehicle they are boarding. Specific embodiments based on the system program are described below.

[1621] Data Collection Phase

[1622] Collection of mobile phone location information

[1623] Device: Each user's mobile device periodically (for example, every 5 seconds) transmits its location information to the server. This information is sent using GPS or beacons.

[1624] User: Before arriving at the station, you need to enable location services on your mobile device.

[1625] Data collection from monitoring devices

[1626] Server: Collects image data in real time from monitoring devices (cameras and sensors) inside and around stations. This data is later used to analyze congestion levels.

[1627] Data analysis phase

[1628] Understanding congestion levels through image analysis

[1629] Server: Analyzes image data collected from the monitoring devices and runs algorithms to calculate the congestion level of each vehicle, including the number of passengers and the space occupancy rate.

[1630] Server: Based on the results of image analysis, classify each vehicle into categories such as "empty," "normal," or "crowded."

[1631] Integrated analysis of location information

[1632] Server: Aggregates location information from each user's mobile device and analyzes the location and destination information of all users, thereby estimating the current number of passengers in each vehicle.

[1633] Optimization Phase

[1634] Optimization calculation of boarding position

[1635] Server: Based on the collected location information and image data, the server calculates the optimal boarding location for the user. For example, an algorithm is designed to prioritize suggesting vehicles with low congestion levels.

[1636] Server: Refers to past congestion data and learns congestion patterns by time of day and day of the week to predict future congestion conditions.

[1637] Guide Phase

[1638] Notification of optimal pickup location

[1639] Server: Based on the calculation results, the server notifies each user of the optimal boarding location (vehicle and door location). Notifications are sent to the device in real time.

[1640] Device: Displays received notification information to the user and provides specific instructions in the form of text and maps.

[1641] Feedback Phase

[1642] Collecting post-ride feedback

[1643] User: After the ride, the user provides feedback through the app about the congestion situation during the ride, which is then sent to the server.

[1644] Terminal: Provides an interface for collecting feedback and accepts input from the user.

[1645] Processing Feedback

[1646] Server: Analyzes the received feedback and uses it to improve the accuracy of future congestion predictions and boarding location suggestions.

[1647] Specific examples

[1648] Morning rush hour scenario

[1649] 1. Device: User A leaves home and activates the location information on his mobile device on his way to the nearest station.

[1650] 2. Server: Obtains user A's location information and sends it to the server. At the same time, it uses augmented reality (AR) functions to confirm the user's location.

[1651] 3. Server: Acquires image data from station monitoring devices and analyzes it.

[1652] 4. Server: Based on the analysis results, it determines that car 3 is currently the vacant car, and notifies user A, "Car 3 is the vacant car. Please wait in front of car 3."

[1653] 5. Terminal: Display a message to user A, prompting him to move in front of car 3.

[1654] 6. User: Follow the instructions to stand in front of car 3, get in, and provide feedback through the app.

[1655] 7. Server: Receives User A's feedback and improves the accuracy of the next prediction based on the analysis results.

[1656] In this way, the system based on the present invention provides a concrete means for managing the congestion status of each vehicle in real time and providing passengers with a comfortable commuter life.

[1657] The processing flow will be explained below.

[1658] Step 1: Collect location information

[1659] Terminal: The mobile terminal periodically (for example, every 5 seconds) collects location information obtained from GPS and beacon sensors and prepares to send it to the server.

[1660] Step 2: Send location information

[1661] Device: Sends location information to the server, which notifies the server of the user's current location in real time.

[1662] Step 3: Collecting data from monitoring devices

[1663] Server: Image data is collected in real time from monitoring devices installed inside and around stations. This image data is used to understand the congestion status of each train car.

[1664] Step 4: Analyzing the image data

[1665] Server: Runs algorithms to analyze the collected image data. Specifically, it uses image analysis technology to calculate the number of passengers and space occupancy in each vehicle.

[1666] Step 5: Calculate congestion

[1667] Server: Based on the results of image analysis, the congestion level of each vehicle is classified into categories such as "empty," "normal," or "crowded."

[1668] Step 6: Integrated analysis of location information

[1669] Server: Aggregates location information sent from mobile devices and analyzes each passenger's current location and destination, thereby estimating the number of passengers in each vehicle.

[1670] Step 7: Calculate the best pickup location

[1671] Server: Based on the analysis results, the server runs an algorithm that calculates the optimal boarding location (vehicle and boarding door) for each user.

[1672] Step 8: Generate notifications

[1673] Server: Based on the calculation results, a message is generated to notify each user of the specific boarding location.

[1674] Step 9: Sending notifications

[1675] Server: Sends the generated message to the user's mobile device. Example: "Car 3 is the least crowded. Please wait in front of car 3."

[1676] Step 10: Displaying notifications

[1677] Terminal: Displays the received message to the user, so that the user can check which car to board.

[1678] Step 11: Ride

[1679] User: Follow the notification and move to the designated vehicle and get in.

[1680] Step 12: Provide feedback

[1681] User: After the ride, enter information about the ride's congestion and other feedback through the app.

[1682] Step 13: Submit your feedback

[1683] Terminal: Sends the feedback entered by the user to the server.

[1684] Step 14: Analyze feedback

[1685] Server: Analyzes the received feedback and uses it as data to improve the accuracy of future predictions.

[1686] Through these steps, the system can grasp the congestion status of each vehicle in real time and suggest the optimal boarding location to the user.

[1687] Example 1

[1688] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1689] In modern urban areas, congestion during commutes has become a major social problem. For commuters, especially those using trains, crowded trains can cause significant stress and discomfort. Conventional methods struggle to grasp congestion levels in real time and provide users with optimal boarding locations, resulting in a lack of information for passengers to determine the appropriate boarding location. Therefore, an effective system is needed to alleviate congestion and ensure a comfortable ride.

[1690] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1691] In this invention, the server includes means for collecting location information from the mobile device, means for collecting image data from the monitoring device, means for analyzing the location information and the image data to calculate the congestion level of each vehicle, means for learning congestion patterns by time period and day of the week based on past data and predicting future congestion conditions, means for calculating the optimal boarding position based on the analysis results and prediction results and notifying the mobile device, means for receiving feedback from the user and improving the accuracy of the analysis and prediction, and means for notifying the mobile device of the optimal boarding position in text or map format. This makes it possible to grasp the congestion status of each vehicle in real time and provide the user with optimal boarding guidance.

[1692] "Mobile device" refers to a portable electronic device that is carried by a user and used by the user, such as a mobile phone, smartphone, or tablet.

[1693] "Location Information" refers to data indicating a specific geographic location obtained through GPS, beacons, Wi-Fi, cell towers, etc.

[1694] "Monitoring devices" refers to devices such as cameras, sensors, and beacons installed in or around stations that monitor the surrounding conditions and collect and provide data.

[1695] "Image data" refers to visual data in the form of still images or video captured or acquired by surveillance equipment.

[1696] "Crowding level" is an indicator of the density of people in a particular area or vehicle, and is calculated based on the number of passengers and the occupancy rate of space.

[1697] "Analysis" refers to the process of analyzing collected data in detail to extract meaning and patterns and derive results according to the intended purpose.

[1698] "Boarding location" refers to the specific car and its door location that the user selects to board the train.

[1699] "Notification" refers to the act of sending a message to a mobile device to notify the user of the results of a calculation or analysis.

[1700] "Past data" refers to data collected in the past, and primarily refers to past congestion conditions and user usage patterns.

[1701] "Feedback" is input data provided by users based on their experiences and opinions, which is used to reevaluate and improve the system.

[1702] "Forecasting" refers to the process of predicting and estimating future situations based on collected data and analysis results.

[1703] This invention is a system that provides optimal boarding guidance to passengers by utilizing location information from mobile devices and image data acquired from monitoring devices in and around stations. This system is implemented using the following means.

[1704] Data Collection Phase

[1705] Collection of mobile phone location information

[1706] Device: Each user's mobile device periodically transmits location information to the server. Location information is collected using GPS or beacon technology (e.g., Bluetooth beacons). Specifically, the device activates location services, measures its current location every 5 seconds, and transmits it to the server using a secure communication protocol (e.g., HTTPS).

[1707] User: Before arriving at the station, the user must enable location services on their mobile device and grant location sharing permission to the app.

[1708] Data collection from monitoring devices

[1709] Server: The server collects image data in real time from monitoring devices inside and around stations. The monitoring devices include cameras and sensors, and receives image data as a stream from these devices. The received data is temporarily stored in storage and used for subsequent analysis.

[1710] Data analysis phase

[1711] Understanding congestion levels through image analysis

[1712] Server: The server uses image analysis algorithms such as OpenCV and TensorFlow to analyze the collected image data. Specifically, it detects passengers, counts the number of people, and analyzes space occupancy for each frame, and classifies the congestion level of each car into categories of "empty," "normal," or "crowded."

[1713] Server: The server stores these analysis results and uses them for subsequent pickup location suggestions.

[1714] Integrated analysis of location information

[1715] Server: The server aggregates the location information collected from each user's mobile device and plots it on a map, allowing the server to determine the current number of passengers in each car and the number of passengers in a specific area of ​​the station.

[1716] Optimization Phase

[1717] Optimization calculation of boarding position

[1718] Server: The server calculates the optimal boarding location for each user based on the aggregated location information and analyzed image data. The algorithm references past congestion data and learns congestion patterns by time of day and day of the week to predict future congestion conditions.

[1719] Server: Based on the calculation results, the optimal boarding location is determined and saved.

[1720] Guide Phase

[1721] Notification of optimal pickup location

[1722] Server: Based on the calculation results, the server notifies each user of the optimal boarding location in real time. The notification content includes specific instructions, such as "Car 3 is the least crowded. Please wait in front of car 3."

[1723] Device: The device displays the received notification to the user in an intuitive format, such as a text message or a pin on a map.

[1724] Feedback Phase

[1725] Collecting post-ride feedback

[1726] User: After the ride, the user provides feedback through the app about the ride's congestion, with options such as "very crowded," "normal," and "empty."

[1727] Terminal: The terminal provides an interface for collecting feedback and accepts input from the user, which is then sent to the server.

[1728] Processing Feedback

[1729] Server: The server analyzes the received feedback and improves the accuracy of congestion predictions and boarding location suggestions for the next trip. The analysis integrates the feedback items and performs statistical evaluation.

[1730] Specific examples

[1731] Morning rush hour scenario

[1732] 1. Device: User A leaves home and activates the location information service on his mobile device on his way to the nearest station.

[1733] 2. Server: The server receives and stores the location information of User A. At the same time, it visually confirms User A's location using the augmented reality (AR) function.

[1734] 3. Server: The server acquires image data from the station monitoring devices and analyzes it in real time.

[1735] 4. Server: Based on the analysis results, it determines that car 3 is the least crowded and notifies user A, "Car 3 is the least crowded. Please wait in front of car 3."

[1736] 5. Terminal: The terminal displays a notification to User A, instructing him to move to the front of car 3. The notification is indicated by a text message and a pin on the map.

[1737] 6. User: User A follows the instructions to stand in front of car 3 and provides feedback on the congestion situation through the app after boarding.

[1738] 7. Server: The server receives User A's feedback and uses the analysis results to improve the accuracy of the next prediction.

[1739] Prompt Sentence Examples

[1740] Prompt: "Describe the steps for a system that uses station surveillance cameras and mobile device location information to provide optimal ride guidance."

[1741] This system is designed to grasp the congestion status of each vehicle in real time and provide users with a comfortable commute.

[1742] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1743] Step 1:

[1744] Collection of mobile phone location information

[1745] Input: Location information obtained from the mobile device's built-in sensors (GPS data, beacon signals, etc.)

[1746] Device behavior: The device activates location services, measures its current location every 5 seconds, and sends the measurements to a server using a secure communication protocol such as HTTPS.

[1747] Output: The user's current location is sent to the server.

[1748] Step 2:

[1749] Image data collection from monitoring devices

[1750] Input: Real-time image streams from surveillance devices (cameras, sensors) installed in and around the station

[1751] Server operation: The server establishes a secure connection with the monitoring device, receives image data in real time, and temporarily stores the received data in storage.

[1752] Output: Image data saved in storage

[1753] Step 3:

[1754] Understanding congestion levels through image analysis

[1755] Input: Image data stored in storage

[1756] Server operation: The server analyzes image data using image analysis algorithms such as OpenCV and TensorFlow. It detects passengers, counts the number of people, and analyzes space occupancy for each frame.

[1757] Output: Analyzed congestion data (e.g., empty, normal, crowded)

[1758] Step 4:

[1759] Congestion category classification for each vehicle

[1760] Input: Crowd data obtained by image analysis

[1761] Server operation: The server classifies each vehicle into a category such as "empty," "normal," or "crowded" based on the congestion data, using a set threshold.

[1762] Output: Congestion category information for each vehicle

[1763] Step 5:

[1764] Integrated analysis of location information

[1765] Input: Location data from each user's mobile device

[1766] Server operation: The server aggregates location information and plots it on a map. It analyzes overall trends and calculates the number of users concentrated in a particular car or in a particular area of ​​the station.

[1767] Output: Integrated analysis results based on user location information

[1768] Step 6:

[1769] Optimization calculation of boarding position

[1770] Input: Integrated analysis results of location information and congestion category information

[1771] Server operation: The server uses an algorithm to calculate the optimal boarding location for the user. It learns from past data and congestion patterns by time of day and day of the week, and runs an algorithm to predict future congestion conditions.

[1772] Output: Optimal pickup location information

[1773] Step 7:

[1774] Notification of optimal pickup location

[1775] Input: Best Pickup Location

[1776] Server operation: Based on the calculation results, the server notifies each user of the optimal boarding location in real time. The notification message includes specific instructions such as "Car 3 is the least crowded. Please wait in front of car 3."

[1777] Device behavior: The device displays notifications to the user in intuitive formats such as text messages or pins on a map.

[1778] Output: A notification message that is displayed to the user

[1779] Step 8:

[1780] Collecting post-ride feedback

[1781] Input: User feedback information

[1782] User Action: After the ride, the user provides feedback through the app about the ride's congestion, with options such as "very crowded," "normal," and "empty."

[1783] Device behavior: The device provides an interface for collecting feedback and accepts input from the user, which is then sent to the server.

[1784] Output: Feedback data sent to the server

[1785] Step 9:

[1786] Processing Feedback

[1787] Input: Feedback data

[1788] Server operation: The server analyzes the received feedback and uses it to improve the accuracy of future congestion predictions and boarding location suggestions. The analysis integrates the feedback items and performs statistical evaluation.

[1789] Output: Improved congestion prediction data and pickup location suggestions

[1790] (Application example 1)

[1791] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1792] Technologies that grasp the congestion situation in and around train stations and guide passengers to the optimal boarding location are widely used, but the application of these technologies to avoiding congestion in commercial facilities has not been sufficiently considered. While the importance of avoiding congestion in such commercial facilities is increasing, current systems lack the technology to analyze the congestion situation in real time and provide users with appropriate routes and areas. The objective of this invention is to provide a system that provides appropriate congestion avoidance guidance in order to provide a comfortable shopping experience in commercial facilities.

[1793] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1794] In this invention, the server includes means for collecting location information from mobile devices, means for collecting image data from monitoring devices, means for analyzing the location information and the image data to calculate the congestion level of each area, means for calculating the optimal route and area based on the analysis results and notifying the mobile devices, means for obtaining feedback from users and improving the accuracy of the analysis and prediction, and means for predicting future congestion conditions using past data. This makes it possible to grasp the congestion status in commercial facilities in real time and provide appropriate congestion avoidance guidance.

[1795] "Mobile device" refers to a device that a user can carry with them and that can transmit location information, such as a mobile phone, smartphone, or tablet.

[1796] "Location information" refers to data transmitted by a mobile device indicating its current geographical location, and is information obtained using GPS or beacons.

[1797] "Surveillance equipment" refers to devices such as cameras and sensors installed in commercial facilities or specific areas that monitor the surrounding situation in real time and collect image data.

[1798] "Image data" refers to video and photographic data collected by surveillance equipment, and is information that can be analyzed to determine the level of congestion in an area.

[1799] "Crowding level" is an index that indicates the density or occupancy rate of people in a specific area, and refers to information calculated from analyzed image data.

[1800] "Calculating routes and areas" refers to the process of deriving the optimal travel route and areas to visit for a user based on collected location information and image data.

[1801] "Means of notification" refers to the method of sending the calculated optimal route and area guidance information to the user's mobile device and displaying it.

[1802] "Feedback" refers to information provided by users about their experiences and opinions, which is collected and analyzed to help improve the accuracy of the system.

[1803] "Prediction means" refers to algorithms and methods for referencing past congestion data and estimating future congestion conditions.

[1804] "Commercial facilities" refers to facilities such as shopping malls, department stores, and shops that are visited by many people and may become crowded.

[1805] This invention relates to a system that provides guidance for avoiding crowds in commercial facilities. It integrates location information from mobile devices and image data collected from monitoring devices to guide users to the optimal route and area in real time. It is also a system that aims to further improve accuracy by collecting feedback from users.

[1806] Data Collection Phase

[1807] Mobile location information collection:

[1808] Device: Each user's mobile device periodically sends its location information to the server, using GPS or beacons.

[1809] Users: Must enable location services on their mobile device before arriving at the commercial establishment.

[1810] Data collection from monitoring devices:

[1811] Server: Collects image data in real time from monitoring devices (cameras and sensors) installed within commercial facilities. This data is used to analyze congestion levels.

[1812] Data Analysis Phase

[1813] Image analysis to understand congestion levels:

[1814] Server: Analyzes image data collected from the monitoring devices and runs an algorithm to calculate the congestion level of each area, specifically the number of users and the occupancy rate of the space.

[1815] Server: Based on the results of image analysis, the congestion level of each area is classified as "low," "normal," "high," etc.

[1816] Integrated location analysis:

[1817] Server: Aggregates location information from each user's mobile device and analyzes the location and destination information of all users to estimate the current congestion situation in each area.

[1818] Route and area optimization phase

[1819] Best route and area suggestions:

[1820] Server: Calculates the optimal route and area for the user based on the collected location information and image data. An algorithm is built in to prioritize suggesting areas with low congestion.

[1821] Server: Refers to past congestion data, learns congestion patterns by time of day and day of the week, and predicts future congestion conditions.

[1822] Guide Phase

[1823] Best route and area notifications:

[1824] Server: Based on the calculation results, the server notifies each user of the optimal route and area information. Notifications are sent to the device in real time.

[1825] Device: Displays received notification information to the user and provides specific instructions in the form of text and maps.

[1826] Feedback Phase

[1827] Collecting post-ride feedback:

[1828] User: After moving around the shopping mall, the user provides feedback about the congestion situation through the app, which is then sent to the server.

[1829] Terminal: Provides an interface for collecting feedback and accepts input from the user.

[1830] Feedback processing:

[1831] Server: Analyzes the received feedback and uses it to improve the accuracy of congestion predictions and route guidance for future trips.

[1832] Hardware and software used:

[1833] Hardware: smartphones, surveillance cameras, sensors.

[1834] Software: Python, image analysis algorithm, location analysis algorithm, external API (for acquiring congestion data).

[1835] Examples:

[1836] A user arrives at a shopping mall at 10:00 a.m. After arriving at shopping mall A, the user opens the app and is directed to the least crowded area. After enjoying shopping, the user provides feedback to the app.

[1837] Example prompt: "Current location: 10:00. Please guide me to the areas in Shopping Mall A. Please recommend the least crowded area, taking into account the congestion status of each area. After making the recommendation, please also display a prompt to provide feedback to the user. The feedback content should be an interface where the user can write about the congestion status of the shopping experience."

[1838] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1839] Step 1:

[1840] Collection of mobile phone location information

[1841] Input: The user enables location services on their mobile device.

[1842] Processing: The device periodically (for example, every 5 seconds) obtains its own location information using GPS or beacons and generates location information data.

[1843] Output: The acquired location data is sent to the server.

[1844] Specific operation: The GPS module in the smartphone measures the current location, and the terminal app collects that data and sends it to a specified server.

[1845] Step 2:

[1846] Data collection from monitoring devices

[1847] Input: Image data provided by surveillance equipment within a commercial facility.

[1848] Processing: The server collects image data from the monitoring devices in real time and prepares the image data for storage and processing.

[1849] Output: The collected image data is stored in the server.

[1850] Specific operation: The server receives the video captured by the surveillance camera and stores it in storage. The image data is also organized in a specific format.

[1851] Step 3:

[1852] Understanding congestion levels through image analysis

[1853] Input: Collected image data.

[1854] Processing: The server uses image analysis algorithms to calculate the congestion level of each area, which is the process of calculating the number of users and the occupancy rate of the space.

[1855] Output: Obtains congestion information for each area.

[1856] What it does: Server-supported image analysis software counts the number of people in the image and calculates the area occupancy rate.

[1857] Step 4:

[1858] Integrated analysis of location information

[1859] Input: Location data and congestion information from user devices.

[1860] Processing: The server combines the location data and congestion information to estimate the overall congestion situation for each area.

[1861] Output: Consolidated congestion data is obtained.

[1862] What it does: The server uses a geographic information system (GIS) to map users' locations and area congestion levels, and then aggregates the data.

[1863] Step 5:

[1864] Optimal route and area suggestions

[1865] Input: Consolidated crowding data.

[1866] Processing: The server uses an algorithm to calculate the best route and area for each user, prioritizing less congested areas.

[1867] Output: The calculated optimal route and area information is obtained.

[1868] Specific operation: The server uses specific rules and algorithms (e.g., shortest path algorithms or congestion-based optimization methods) to determine the most suitable route and area for the user.

[1869] Step 6:

[1870] Optimal route and area notifications

[1871] Input: Calculated optimal route and area information.

[1872] Processing: The server sends the optimal route and area information to the user's mobile device.

[1873] Output: A real-time notification is displayed on the user's mobile device.

[1874] What it does: Uses the notification system (push notifications and in-app notifications) to display text and map-based directions on the user's smartphone.

[1875] Step 7:

[1876] Collecting and Processing Feedback

[1877] Input: Feedback information provided by the user.

[1878] Processing: The device provides an interface for collecting feedback and accepts user input. The server analyzes the received feedback and uses it to improve the accuracy of future guidance.

[1879] Output: Parsed feedback data.

[1880] Specific operation: Collects information provided by users through feedback forms and survey functions within the app, and the server analyzes the data to perform feedback analysis.

[1881] Prompt Sentence Examples

[1882] "Current location: 10:00. Please guide the user to the areas in the shopping mall. Please recommend the least crowded area, taking into account the congestion level of each area. After making the recommendation, please also display a prompt to provide feedback to the user. The feedback should be an interface where the user can describe how crowded the shopping experience was."

[1883] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1884] The present invention combines an emotion engine with a system that uses location information from mobile devices and image data collected from monitoring devices in and around stations to provide appropriate guidance on which vehicle to board. Specific embodiments based on the system program are described below.

[1885] Data Collection Phase

[1886] Collection of mobile phone location information

[1887] Device: Each user's mobile device periodically (for example, every 5 seconds) transmits its location information to the server. This information is sent using GPS or beacons.

[1888] User: Before arriving at the station, you need to enable location services on your mobile device.

[1889] Data collection from monitoring devices

[1890] Server: Collects image data in real time from monitoring devices (cameras and sensors) installed inside and around stations. This image data is used to understand the congestion status of each train car.

[1891] Data analysis phase

[1892] Understanding congestion levels through image analysis

[1893] Server: Runs algorithms to analyze the collected image data. Specifically, it uses image analysis technology to calculate the number of passengers and space occupancy in each vehicle.

[1894] Server: Based on the results of image analysis, the congestion level of each vehicle is classified into categories such as "empty," "normal," or "crowded."

[1895] Integrated analysis of location information

[1896] Server: Aggregates location information from each user's mobile device and analyzes the location and destination information of all users, thereby estimating the number of passengers in each vehicle.

[1897] Optimization Phase

[1898] Optimization calculation of boarding position

[1899] Server: Based on the analysis results, the server runs an algorithm that calculates the optimal boarding location (vehicle and boarding door) for each user.

[1900] Server: Refers to past congestion data and learns congestion patterns by time of day and day of the week to predict future congestion conditions.

[1901] Emotion Recognition Phase

[1902] Using the Emotion Engine

[1903] Device: The emotion engine recognizes the user's emotions using facial expression analysis via a camera, voice analysis, and the user's operation patterns.

[1904] Server: Analyzes the emotional data obtained from the emotion engine and understands the user's current emotional state.

[1905] Emotion-Based Optimization

[1906] Server: Integrates the user's emotional data and analysis results, and adjusts the recommended ride locations based on the user's emotional state. For example, for users with high levels of stress or anxiety, it will prioritize suggesting less crowded vehicles.

[1907] Guide Phase

[1908] Notification of optimal pickup location

[1909] Server: Based on the calculation results, it generates a message to notify each user of the optimal boarding location.

[1910] Device: Displays received notification information to the user and provides specific instructions in the form of text and maps.

[1911] Feedback Phase

[1912] Collecting post-ride feedback

[1913] User: After the trip, provide feedback through the app about the ride's congestion and emotional state.

[1914] Terminal: Provides an interface for collecting feedback and accepts input from the user.

[1915] Processing Feedback

[1916] Server: Analyzes the received feedback and uses it as data to improve prediction accuracy and emotion recognition accuracy in future trials.

[1917] Specific examples

[1918] Morning rush hour scenario

[1919] 1. Device: User A leaves home and activates the location information on his mobile device on his way to the nearest station.

[1920] 2. Server: Obtains user A's location information and sends it to the server. At the same time, it uses augmented reality (AR) functions to confirm the user's location.

[1921] 3. Server: Acquires image data from station monitoring devices and analyzes it.

[1922] 4. Server: Based on the analysis results, it determines that car 3 is currently the vacant car, and notifies user A, "Car 3 is the vacant car. Please wait in front of car 3."

[1923] 5. Device: The emotion engine analyzes user A's emotion data and detects his / her stress state.

[1924] 6. Server: Consider user A’s emotional state and further adjust the pickup location suggestions.

[1925] 7. Terminal: Display a message to user A, prompting him to move in front of car 3.

[1926] 8. User: Follow the instructions to stand in front of car 3, get in, and provide feedback through the app.

[1927] 9. Server: Receives User A's feedback and improves the accuracy of the next prediction based on the analysis results.

[1928] In this way, the system based on the present invention provides a concrete means for providing users with a comfortable commuter life by managing the congestion status of each vehicle in real time and proposing the optimal boarding location taking into account the user's emotional state.

[1929] The processing flow will be explained below.

[1930] Step 1: Collect location information

[1931] Terminal: The mobile terminal periodically (for example, every 5 seconds) collects location information obtained from GPS and beacon sensors and prepares to send it to the server.

[1932] Step 2: Send location information

[1933] Device: Sends location information to the server, which notifies the server of the user's current location in real time.

[1934] Step 3: Collecting data from monitoring devices

[1935] Server: Obtains image data in real time from monitoring devices (cameras and sensors) installed inside and around stations. This image data is used to understand the congestion status of each train car.

[1936] Step 4: Analyzing the image data

[1937] Server: Runs algorithms to analyze the collected image data. Specifically, it uses image analysis technology to calculate the number of passengers and space occupancy in each vehicle.

[1938] Step 5: Calculate congestion

[1939] Server: Based on the results of image analysis, the congestion level of each vehicle is classified into categories such as "empty," "normal," or "crowded."

[1940] Step 6: Integrated analysis of location information

[1941] Server: Aggregates location information sent from mobile devices and analyzes each passenger's current location and destination, thereby estimating the number of passengers in each vehicle.

[1942] Step 7: Calculate the best pickup location

[1943] Server: Based on the analysis results, the server runs an algorithm that calculates the optimal boarding location (vehicle and boarding door) for each user.

[1944] Step 8: Use historical data

[1945] Server: Refers to past congestion data and learns congestion patterns by time of day and day of the week to predict future congestion conditions.

[1946] Step 9: Emotion Recognition with the Emotion Engine

[1947] Device: The emotion engine recognizes the user's emotions by analyzing facial expressions via a camera, voice analysis, and the user's operation patterns.

[1948] Step 10: Sending Emotion Data

[1949] Device: Sends the recognized emotion data to the server, which informs the server of the user's current emotional state.

[1950] Step 11: Adjusting riding position based on emotions

[1951] Server: Integrates the user's emotional data and analysis results to tailor the recommended ride locations based on their emotional state. For example, prioritize less crowded vehicles for users with high levels of stress.

[1952] Step 12: Generate and send notifications

[1953] Server: Based on the calculation results and emotion data, a message is generated to notify each user of their specific boarding location and sent to the device.

[1954] Step 13: Viewing Notifications

[1955] Terminal: Displays the received message to the user, so that the user can check which car they should board.

[1956] Step 14: Ride

[1957] User: Follow the notification and move to the designated vehicle and get in.

[1958] Step 15: Provide feedback

[1959] User: After the ride, users provide feedback through the app about the ride's congestion and emotional state.

[1960] Step 16: Submit your feedback

[1961] Terminal: Sends the feedback entered by the user to the server.

[1962] Step 17: Analyze feedback

[1963] Server: Analyzes the received feedback and uses it as data to improve prediction accuracy and emotion recognition accuracy in future trials.

[1964] Through these steps, the system can grasp the congestion status of each vehicle in real time and suggest the optimal boarding location taking into account the user's emotional state.

[1965] Example 2

[1966] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1967] Conventional ride guidance systems were able to grasp the congestion level of each vehicle using location information and image data, but did not provide ride guidance that took into account the user's emotional state. As a result, riding in a crowded vehicle could cause stress and discomfort to the user. It was particularly difficult to provide a comfortable ride experience for users during rush hour situations.

[1968] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for collecting location information from a mobile terminal, means for collecting image data from a monitoring device, means for analyzing the location information and the image data to calculate the congestion level of each vehicle, means for grasping the emotional state of the user using an emotion recognition engine, and means for calculating the optimal boarding position based on the analysis result and the emotional state and notifying the mobile terminal. This makes it possible to propose an optimal boarding position taking into account the user's emotional state in addition to the user's location information and congestion status.

[1969] A "mobile terminal" is a communication device carried by a user that provides location information and emotional state data.

[1970] "Location information" refers to geographic coordinate information collected from mobile devices and GPS sensors.

[1971] "Monitoring devices" are equipment such as cameras and sensors that are installed in or around stations to collect image data.

[1972] "Image data" refers to video and photo data acquired from a surveillance device.

[1973] "Analysis" involves processing the collected data and calculating the congestion level of each vehicle and the emotional state of the user.

[1974] "Crowding level" refers to the degree of congestion based on the number of passengers and space occupancy in each vehicle.

[1975] An "emotion recognition engine" is software for analyzing a user's emotional state, including facial expression analysis and voice analysis using a camera and microphone.

[1976] "Notification" refers to the act of sending the calculated optimal boarding location to the user's mobile device.

[1977] "Feedback" is information provided by users about the congestion situation and their emotional state after riding.

[1978] This invention combines an emotion engine with a system that uses location information from mobile devices and image data collected from monitoring devices in and around stations to provide appropriate guidance on which vehicle to board. Specific embodiments of the system are described below.

[1979] Data Collection Phase

[1980] Collection of mobile phone location information

[1981] Device: Each user's mobile device sends its location information to the server every 5 seconds. This information is obtained using GPS or beacons. Specifically, the device's built-in GPS sensor is used to obtain location information, which is then compressed and sent to the server.

[1982] User: Before arriving at the station, they will need to enable location services on their mobile device. A notification will appear on the device prompting them to enable the service.

[1983] Data collection from monitoring devices

[1984] Server: Collects image data in real time from monitoring devices (cameras and sensors) installed in and around the station. Specifically, it acquires video streams from cameras installed on platforms and at entrances and exits.

[1985] Server: Temporarily stores collected image data in storage for subsequent analysis.

[1986] Data analysis phase

[1987] Understanding congestion levels through image analysis

[1988] Server: Executes algorithms to analyze collected image data. Specifically, it applies image analysis technology using deep learning to calculate the number of passengers in each vehicle and the occupancy rate of space. For example, it identifies the positions of passengers' faces and bodies based on images acquired from surveillance cameras and calculates the degree of congestion.

[1989] Server: Based on the calculated congestion level, each vehicle is classified into a category such as "empty," "normal," or "crowded." This classification result is stored in a database and used in subsequent phases.

[1990] Integrated analysis of location information

[1991] Server: Aggregates location information from each user's mobile device and analyzes the location and destination information of all users, using a location clustering algorithm to identify areas where passengers are concentrated.

[1992] Server: Combines location and congestion data to calculate the expected number of passengers in each vehicle.

[1993] Emotion Recognition Phase

[1994] Using the Emotion Engine

[1995] On-device: The device uses an emotion recognition engine to analyze the user's emotional state. Specifically, the device uses the camera to analyze facial expressions and the microphone to analyze voices. This allows the device to identify emotional states such as stress or anxiety.

[1996] Server: Analyzes the emotional data sent from the emotion recognition engine and understands the user's emotional state. This makes it possible to suggest the optimal boarding location based on the user's emotional state.

[1997] Guide Phase

[1998] Notification of optimal pickup location

[1999] Server: Calculates the optimized boarding location (specific vehicle and boarding door) and generates customized instruction messages for each user.

[2000] Terminal: The generated notification message is displayed to the user. For example, specific instructions such as "Currently, car number 3 is the vacant car. Please wait in front of car number 3" are displayed in text or map format.

[2001] Feedback Phase

[2002] Collecting post-ride feedback

[2003] Users: After their ride, they provide feedback through the app about the occupancy and their emotional state, specifically in the form of a short survey about their ride experience.

[2004] Terminal: Provides a feedback input interface and accepts data from the user.

[2005] Processing Feedback

[2006] Server: Analyzes the received feedback data and performs batch processing to improve prediction accuracy and emotion recognition accuracy from the next time onwards. This is expected to improve the accuracy of the entire system.

[2007] Specific examples

[2008] Morning rush hour scenario

[2009] 1. Device: User A leaves home and activates the location information on his mobile device on his way to the nearest station.

[2010] 2. Server: Obtains user A's location information and sends it to the server. At the same time, the server confirms the user's location using AR functions, etc.

[2011] 3. Server: Acquires image data from station monitoring devices and analyzes it.

[2012] 4. Server: Based on the analysis results, it determines that car 3 is currently the vacant car, and notifies user A, "Car 3 is the vacant car. Please wait in front of car 3."

[2013] 5. Device: The emotion engine analyzes user A's emotion data and detects his / her stress state.

[2014] 6. Server: Consider user A’s emotional state and further adjust the pickup location suggestions.

[2015] 7. Terminal: Display a message to user A, prompting him to move in front of car 3.

[2016] 8. User: Follow the instructions to stand in front of car 3, get in, and provide feedback through the app.

[2017] 9. Server: Receives User A's feedback and improves the accuracy of the next prediction based on the analysis results.

[2018] In an embodiment based on this scenario, user location information, station monitoring data, and emotion data can be integrated in real time to provide users with a comfortable riding experience.

[2019] Prompt Sentence Examples

[2020] Below are some example prompts to input to the generative AI model:

[2021] "Please describe in detail the process of a system that uses mobile device location information and station surveillance camera data to calculate the optimal boarding location for passengers. This should include data collection, analysis phase, use of an emotion engine, optimization, notification phase, and user feedback processing."

[2022] The flow of the identification process in the second embodiment will be described with reference to FIG.

[2023] Step 1:

[2024] Device: The user's mobile device enables location services. Specifically, the device's GPS module acquires its current location every 5 seconds and transmits the location data to the server. This provides the server with the user's real-time location information as input data.

[2025] Step 2:

[2026] Server: Collects image data from surveillance devices (cameras and sensors) installed in and around the station. Specifically, it acquires video streams from each surveillance camera in real time and temporarily stores them in storage. This image data is used in the next image analysis phase.

[2027] Step 3:

[2028] Server: Analyzes the collected image data. Specifically, it applies image analysis technology using deep learning to calculate the number of passengers in each vehicle and the occupancy rate of space. It uses an object detection algorithm to identify and count passengers from the input image data. The analysis results output the degree of congestion in each vehicle.

[2029] Step 4:

[2030] Server: Based on the congestion analysis results, classify each vehicle into a category such as "empty," "normal," or "crowded." The input is the analysis result from step 3, which is applied to a category classification algorithm to output the classification result. This classification result is stored in a database.

[2031] Step 5:

[2032] Server: Aggregates location information sent from each user's mobile device and analyzes the overall location and destination information. Specifically, it uses a location clustering algorithm to identify areas where users are concentrated. The input is real-time location data, and the output is passenger density information for each area.

[2033] Step 6:

[2034] Device: The device uses an emotion recognition engine to analyze the user's emotional state. Specifically, facial expression analysis is performed using the device's camera, and voice analysis is performed using the microphone. The input is the user's facial expression image and voice data, and the output is the evaluation result of the user's emotional state.

[2035] Step 7:

[2036] Server: Analyzes the emotion data sent from the emotion recognition engine and understands the user's emotional state. The input is the emotion engine's output data, and analysis is performed based on this to evaluate the user's emotional state. Based on this evaluation result, the boarding location suggestions are optimized.

[2037] Step 8:

[2038] Server: Calculates the optimal pickup location based on the user's emotional state and generates a personalized notification message. The inputs are congestion data, user location data, and the user's emotional state data, which are combined to run an algorithm to calculate the optimal pickup location. The output is a notification message.

[2039] Step 9:

[2040] Terminal: Displays notification messages received from the server to the user. Specifically, it displays boarding location instructions in text and map format. For example, it displays a message such as "Currently, car 3 is the least crowded. Please wait in front of car 3." The input is the notification message from the server, and the output is the notification to the user.

[2041] Step 10:

[2042] User: Follows the instructions to go to the designated pickup location. After boarding, the user provides feedback on the congestion situation and emotional state during the ride through the app. The input is the user's experience data, and the output is feedback information.

[2043] Step 11:

[2044] Server: Analyzes the feedback data received from users and improves the algorithms to improve prediction accuracy and emotion recognition accuracy in the future. The input is user feedback data, and the output is an improved analysis and prediction algorithm.

[2045] (Application example 2)

[2046] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2047] The issue of congestion in public transportation and the inability to properly guide passengers to a comfortable boarding location are issues. Furthermore, providing boarding guidance without taking into account the passenger's emotional state increases stress and discomfort, degrading the overall riding experience.

[2048] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for collecting location information from the mobile terminal, means for collecting image data from the monitoring device, means for analyzing the location information and the image data to calculate the congestion level of each vehicle, means for analyzing the emotional state of the user, and means for calculating the optimal boarding position based on the analysis result and the emotional state and notifying the mobile terminal. This makes it possible to suggest the optimal boarding position taking into consideration the congestion level of the vehicle and the emotional state of the user, thereby improving the overall riding experience.

[2049] A "mobile terminal" is an electronic device that can be carried by a user and has GPS and Internet connectivity capabilities.

[2050] "Location information" is information that indicates coordinate data at a specific location using a GPS, beacon, or the like.

[2051] A "surveillance device" is a device that includes a camera, sensor, etc., and collects image data and environmental data of a designated area in real time.

[2052] "Image data" is visual information, including still images and video, collected by surveillance equipment.

[2053] "Crowding" is an indicator that expresses the density of people and objects in a particular area or space using a number or category.

[2054] "Emotional state" is data that indicates an individual's psychological state, and is obtained by facial expression analysis and voice analysis via a camera.

[2055] "Analysis results" are information obtained as a result of calculations and evaluations performed on collected data.

[2056] "Forecasting" is the process of predicting future conditions or events based on past and current data.

[2057] "Optimal boarding position" refers to the recommended position or door for the user to board comfortably and efficiently.

[2058] A "notification" is information sent from the system to the user, and may be provided in text, audio, visual format, or the like.

[2059] "Feedback" refers to information such as opinions, evaluations, and impressions collected from users, and is used to improve the system and increase the accuracy of analysis.

[2060] The "smart ride guidance system" of the present invention is a system that collects location information from mobile devices and image data from monitoring devices, and guides users to the optimal ride location taking into account their emotional state. A specific method for implementing this system is described below.

[2061] Data Collection Phase

[2062] Collection of location information from mobile devices

[2063] The terminal is an electronic device (such as a smartphone) carried by the user, which periodically collects location information using its GPS function and transmits it to the server. Location information collection begins when the user is near a station or an autonomous vehicle, and the server receives this information in real time.

[2064] Collection of image data from monitoring devices

[2065] The server collects image data from cameras and other sensors installed in stations and on autonomous vehicles. These devices provide the real-time visual information needed to understand the occupancy level of each carriage.

[2066] Data analysis phase

[2067] Understanding congestion levels through image analysis

[2068] The server uses image analysis algorithms (e.g., OpenCV) to analyze the collected image data, which calculates the number of passengers and space occupancy in each vehicle and classifies the level of congestion into categories of "empty," "normal," or "crowded."

[2069] Emotional state analysis

[2070] The device uses an emotion engine (e.g., Microsoft's Azure Emotion API or Google Cloud's Vision API) to analyze the user's emotions. By analyzing facial expressions and voice via the device's camera and the user's operation patterns, the device detects emotional states such as stress, anxiety, and comfort.

[2071] Optimization Phase

[2072] Optimization calculation of boarding position

[2073] The server combines congestion data and the user's emotional state to calculate the optimal boarding location (car and boarding door) for each user. This calculation uses an algorithm that references past congestion data and learns congestion patterns by time of day and day of the week.

[2074] Guide Phase

[2075] Notification of optimal pickup location

[2076] The server generates a message informing each user of the optimal boarding location based on the calculation results, and the terminal displays this notification information to the user, providing specific instructions in the form of text and maps.

[2077] Feedback Phase

[2078] Collecting post-ride feedback

[2079] The terminal provides an interface to accept feedback from users after their ride. Users can provide feedback through the app regarding the congestion situation and emotional state during the ride.

[2080] The server analyzes the feedback it receives and uses it as data to improve prediction accuracy and emotion recognition accuracy in future trials.

[2081] Specific examples

[2082] For example, during the morning rush hour, User A sends location information from his mobile device on his way from his home to the nearest station. The device then uses an emotion engine to detect User A's stress level. The server then suggests the least crowded car based on the congestion data within the station and User A's emotional state, and notifies the device, "Car 3 is the least crowded. Please wait in front of Car 3." As a result, User A can commute comfortably without feeling stressed.

[2083] Prompt Sentence Examples

[2084] "Design an AI model to improve the accuracy of emotion recognition. The goal is to analyze the user's emotional state in real time and suggest the optimal boarding location based on the congestion situation. Specifically, identify the user's emotional state (stress, anxiety, comfort, etc.) from facial images and voice data, and use the results and in-car congestion data to provide optimal boarding guidance."

[2085] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[2086] Step 1:

[2087] The device collects location information using the GPS function of the mobile device. When the user is near a station or an autonomous vehicle, the device periodically transmits the location information to the server.

[2088] Input: GPS data from your mobile device

[2089] Output: User's current location

[2090] Specific operation: The device acquires GPS data every 5 seconds and sends it to the server.

[2091] Step 2:

[2092] The server collects image data in real time from monitoring devices (cameras and sensors) installed within stations and autonomous vehicles.

[2093] Input: Real-time image data from a surveillance device

[2094] Output: Image information of station premises and train interiors

[2095] Specific operation: The server continuously receives the video stream transmitted from the monitoring device.

[2096] Step 3:

[2097] The server analyzes the collected image data using an image analysis algorithm (e.g., OpenCV) and calculates the congestion level of each vehicle.

[2098] Input: Collected image data

[2099] Output: Crowding level of each vehicle (empty, normal, crowded categories)

[2100] Specific operation: The server counts the number of people in the image, calculates the occupancy rate, and classifies the congestion level into categories.

[2101] Step 4:

[2102] The device uses an emotion engine (e.g., Microsoft's Azure Emotion API or Google Cloud's Vision API) to analyze the user's emotional state.

[2103] Input: Device camera images, audio data, user operation patterns

[2104] Output: User's emotional state (stressed, anxious, comfortable, etc.)

[2105] Specific operation: The device uses a camera to capture the user's facial expressions and analyzes them using an emotion analysis engine.

[2106] Step 5:

[2107] The server integrates the congestion data and the user's emotional state to calculate the optimal boarding location.

[2108] Input: crowding data, user emotional state

[2109] Output: Optimal boarding position (vehicle and boarding door)

[2110] Specific operation: The server uses an algorithm to calculate the optimal boarding location for each user, and also refers to past data to take into account future congestion conditions.

[2111] Step 6:

[2112] The server generates a message informing the user of the optimal boarding location based on the calculation results.

[2113] Input: Calculated optimal pickup location

[2114] Output: A message to inform the user

[2115] Specific operation: The server generates a text message and map-style directions including the best pickup location and sends them to the device.

[2116] Step 7:

[2117] The terminal displays the notification information received from the server to the user and provides specific instructions.

[2118] Input: A notification message to the user

[2119] Output: Instructions displayed to the user

[2120] Specific operation: The device uses the display and voice guidance functions to provide the user with information about the boarding location.

[2121] Step 8:

[2122] Users provide feedback through the app after their ride.

[2123] Input: User feedback information

[2124] Output: Feedback data sent to the server

[2125] What it does: Users use the app interface to input feedback about congestion and emotional state.

[2126] Step 9:

[2127] The server analyzes the feedback it receives and uses it to improve prediction accuracy and emotion recognition accuracy in future trials.

[2128] Input: User feedback data

[2129] Output: Improved parsing algorithm

[2130] Specific operation: The server stores the feedback data in a database and uses it to adjust and improve the analysis algorithm.

[2131] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[2132] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[2133] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[2134] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[2135] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are ...

Claims

1. means for collecting location information from a mobile device; means for collecting image data from a monitoring device; means for analyzing the position information and the image data to calculate the congestion level of each vehicle; A system including a means for calculating the optimal boarding location based on the analysis results and notifying the mobile terminal.

2. 2. The system according to claim 1, further comprising means for predicting future congestion conditions using past data.

3. The system of claim 1 further comprising means for obtaining feedback from a user to improve the accuracy of said analysis and prediction.

Images